this post was submitted on 17 Jul 2023
44 points (90.7% liked)

Linux

47934 readers
1721 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
 

I've been using Linux for the better part of 4 years so I'm not new to it, but I've always learned stuff on an as-needed basis. Today I ran into an issue that I want to prevent in the future since I had a mini heart attack thinking about how my last backup on this system was... Never since I'm an idiot who forgot to set it up like I have on my laptop. Here are my steps:

  • Ran sudo pacman -Syu; sudo pacman -Syy like I do every few days
  • packages updated
  • restarted computer
  • can only boot into emergency mode

The journal was really long so I moved past it and went to the pacman logs, linux had updated from 6.4.3.1-1 to 6.4.3.1-2. Nothing else was important enough to cause the system to only boot into emergency (gcc, vbox, some libs) so I did a quick pacman -U to the cached 6.4.3.1-1 version for both Linux and Linux headers and rebooted - hurrah it was fixed! But I have no idea why it happened, or how to prevent it.

Has anyone else ran into this issue when updating? Any advice for preventing future crashes or issues like this so I don't fear updating?

Edit: Thanks to everyone for your advice! I ended up following multiple bits of advice. I reinstalled arch to get btrfs as the filesystem (didn't have anything important other than some docked-compose files and books yet) and grabbed the linux-lts kernal as a backup as well. I haven't configured snapper yet, but it's on my list of things to do.

top 20 comments
sorted by: hot top controversial new old
[–] [email protected] 18 points 1 year ago (1 children)

Ran sudo pacman -Syu; sudo pacman -Syy like I do every few days

Syy forces the package database to be updated even if no updates are available.

In my opinion, this makes no sense, especially after you have already run pacman -Syu before. Basically, you only generate additional, unnecessary traffic on the mirror you are using. Pacman -Syu is normally always sufficient.

The journal was really long so I moved past it

The display of the systemd journal can be easily filtered. For example, with journalctl -p err -b -1, all entries of the last boot process that are marked as error, critical, alarm or emergency are displayed.

Has anyone else ran into this issue when updating?

Not me. But other users do. Some of them also use a distribution other than Arch (or a distribution based on it). When I look at the problems, the current kernel is probably quite a minefield as far as problems are concerned.

Any advice for preventing future crashes or issues like this so I don’t fear updating?

As other users have already recommended, you could additionally install the LTS kernel. And if you use BTRFS as a file system, create snapshots before an update (https://wiki.archlinux.org/title/snapper#Wrapping_pacman_transactions_in_snapshots).

And it should be obvious that important data should be backed up on a regular basis.

[–] [email protected] 2 points 1 year ago (1 children)

What if i run

pacman -Syyu

?

[–] [email protected] 3 points 1 year ago* (last edited 1 year ago) (1 children)

I guess that's a bit better than the original command in question. But from what I understand it's still unnecessary and there is simply no need to force the refresh. A regular pacman -Syu is all you need and will refresh all databases that need it.

[–] [email protected] 3 points 1 year ago

Thanks! I willf then do that moving forwards :)

[–] [email protected] 18 points 1 year ago* (last edited 1 year ago) (1 children)

You could install the linux-lts kernel alongside the one you have already installed to have the option to just boot into that one when a kernel update seems to be the problem.

Another thing would be to look into backup solutions that execute automatically when updating your system. Personally I have my system on BTRFS subvolumes and a package called snapper to manage the snapshots (backups). Alternatively the package timeshift gets mentioned a lot when discussing backup solutions.

Otherwise you did exactly what I have done to fix almost every issue I ever had. Downgrading the likely culprit and updating again a bit later.

[–] [email protected] 10 points 1 year ago* (last edited 1 year ago) (1 children)

I second btrfs with snapper. With snapper, you can set it up so that it automatically makes snapshots at a timed interval and/or when you run your package manager. You can restore any of your saved snapshots from the snapper app or even from GRUB.

It's a bit hard to set up, but some distros come with it set up by default. You could install one if you don't want to figure btrfs setup out and are open to OS hopping. OP, you mentioned you're using arch, Garuda OS is an Arch based distro that comes with btrfs and the grub snapper configurations set up by default.

[–] [email protected] 5 points 1 year ago

Thanks for the info! I've tried garuda and didn't like it, but I'll try snapper!

[–] [email protected] 13 points 1 year ago* (last edited 1 year ago) (1 children)

Sadly, I can’t help you there, but I must applaud to your attitude of figuring out what happend and why.

When reading forum posts about Windows, Android or iOS stuff, it’s infuriatingly common to find a list of potential fixes without any explanations. Many people don’t know what went wrong, or why, but they do have some ideas what might fix it. Unfortunately, they just can’t tell you why a particular action is supposed to fix anything, because they don’t understand the root causes.

[–] [email protected] 4 points 1 year ago (1 children)

The amount of times ive seen "well if x and y don't work, you might have to reinstall Windows" keeps me up at night

[–] [email protected] 2 points 1 year ago

If turning it on and off again didn’t help, it’s time to reinstall. 🙈

[–] [email protected] 10 points 1 year ago

Maybe rebuilding the ramdisk failed during the original upgrade?
One of the post-install stages after upgrading the kernel is rebuilding the initramfs - a tiny environment for bootstrapping the main OS.
If you trigger it manually with mkinitcpio --allpresets you'll notice it has fancy colorful output, with clearly visible warnings/errors.
However when invoked as part of an upgrade this coloring is removed, making errors difficult to spot.
I had this stage randomly fail a few times, resulting in an unbootable system like you described - solution was to just trigger a manual rebuild or reinstall the kernel with pacman -U.
It's possible that this is what actually fixed things when you downgraded the kernel.

[–] [email protected] 6 points 1 year ago* (last edited 1 year ago)

The only difference between those two versions of linux is that the new one was built with a newer version of gcc. That doesn't really narrow the problem down, though. As far as I'm aware, emergency mode is caused by either a kernel panic or a failure to mount a needed filesystem. I'm leaning towards a corrupted kernel, since it doesn't sound like you changed your fstab or had any problem mounting /. I would run fsck -f on your boot partition, then try to re-download and reinstall the new package.

If that doesn't work, then you can add IgnorePkg = linux linux-headers to pacman.conf so you can update without installing the broken package, until you resolve the underlying issue. Or your can install a different kernel altogether.

As for preventing problems in the future, there's only so much you can do. Check archlinux.org before updating to see if anything requires manual intervention, and pay close attention while running pacman in case something goes wrong. You already seem to know the most important part, which is to keep a set of packages that are certain to work, so you can easily downgrade if a crash does happen.

[–] [email protected] 5 points 1 year ago

Arch never breaks.

[–] [email protected] 3 points 1 year ago

btrfs and snapshots is one solution

Also, it looks like you were able to solve the problem, even if you aren't 100% sure why. Maybe the difference in package versions lead to some problem with the bootloader, causing it to go to emergency mode, and updating it fixed the problem?

Another is using a more stable distro....

[–] [email protected] 3 points 1 year ago (1 children)

Arch breaks once in a while... Like anything else in my experience

[–] [email protected] 1 points 1 year ago (1 children)

But the arch users told me it never breaks, could they have lied to me?

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago)

In about two years, the update process broke twice... I had to manually add or remove packages so that the update process would complete successfully

[–] [email protected] 3 points 1 year ago

Use timeshift, easy peasy to set up. It's saved my bacon a couple of times now, only last week, when the kernel 6.4 came out, but my old nvidia driver wouldn't work with it. You just jump into tty, run timeshift --restore and chose a previous backup, takes all of ten minutes. I have it doing a back up every day to a second HD, keeping the last five, doesn't take up much space. I don't think there's an easier option than that.

[–] [email protected] 2 points 1 year ago (1 children)

Advice? Sure, setup timeshift backups.

Or if possible, switch to btrfs and install snapper + the grub integration. Will make it possible to go back to a previous state even from grub.

Not if you mess up grub tho, then you're screwed.

[–] [email protected] 0 points 1 year ago

You don't need timeshift because arch never breaks.

Source: arch users

load more comments
view more: next ›