Linux

Not really a "braking my linux setup", but still fun as hell! Back in university, a friend of mine got a new notebook at a time... we spent the night at the university hacking and they wanted to set the notebook up in the evening. They got to the point where they had to setup luks via the cryptsetup CLI. But they got stuck, it just wouldn't work. They tried for HOURS to debug why cryptsetup didn't let them setup LUKS on the drive.

At some point, in the middle of the night (literally something like 2 in the morning) they suddenly JUMPED from their seat and screamed "TYPE UPPERCASE 'YES' - FUCK!!!"

They debugged for about six hours and the conclusion was that cryptsetup asks "If you are sure you want to overwrite, type uppercase 'yes'". ... and they typed lowercase. For six hours. Literally.

The room was on the floor, holding their stomach laughing.

Accidentally deleted system Python, which on GNOME meant my DE was toast as well. Luckily very freshly set up, so no harm done.

Related note, add this in your shell profile:

bash
export PIP_REQUIRE_VIRTUALENV=true

proper scripting language
set PIP_REQUIRE_VIRTUALENV true

I used to work at this place that had a gigantic QNX install. I don't know if QNX that we used back then had any relation to q&x now They certainly don't look very close.

It was in the '90s and they had it set up so that particular nodes handled particular jobs. One node to handle boot images and serve as a net boot provider, one node handled all of the arcnet to ethernet communication, one node handled all the serial to mainframe, a number of the nodes were main worker nodes that collected data and operated machinery and diverters. All of these primary systems were on upper-end 386s or 486s ,they all had local hard disks.

The last class of node they called slave nodes. They were mainly designed for user data ingest, data scanning stations, touch screen terminals, simple things that weren't very high priority.

These nodes could have hard discs in them, and if they did, they would attempt to boot from them saving the net boot server a few cycles.

If for some reason they were unable to boot from their local hard drive, They would netboot format their local hard drive and rewrite their local file system.

If they were on able to rewrite their local file system they could still operate perfectly fine purely off the net boot. The Achilles heel of the system was that you had no idea that they had net booted unless you looked into the log files. If you boot it off your local hard drive of course your root file system would be on your local hard drive. If you had net booted, and it could not rebuild your local file system, your local root file / was actually the literal partition on the boot server. Because of the design of the network boot, nothing looked like it was remotely mounted.

SOP for problems on one of the slave nodes was to wipe the hard disk and reboot, in the process it would format the hard drive and either fix itself or show up as unreliable and you could then replace the disc or just leave the disc out of it. Of course If the local disk had failed and the box had already rebooted off netboot without a technician standing there to witness it, rm -Rf would wipe out the master boot node.

I wasn't the one that wiped it, but I fully understand why the guy did.

Turns out we were on a really old version of QNX, we were kind of a remote warehouse mostly automated. They just shut us down for about a week. Flew a team out. Rebuilt the system from newer software, and setup backups.

I installed python one time

A regular update I guess...

I've had the typical disasters with partition tables and boot loader mixups, but the one I keep coming back to is updating my Nvidia drivers too eagerly. Whether something gets messed up with an external monitor, or the laptop starts resisting switching away from the integrated GPU, or an electron app I use regularly that makes heavy use of 3D acceleration breaks, or I just need to bump the driver version in a reproducible system state record... it's just bad news.

Back when I started using Linux, I really wanted something that was super different from windows (I used Gnome 3 for like 3 years). I decided one day to try out Fedora cause, hey, I can live on the bleeding edge.

Second day I had it installed, I was having issues with the audio. Decided to try reinstalling pulse. Apt autoremoved it and somehow completely nuked the entire GUI. Stuck in terminal mode, I found that I had no ethernet to connect to, nor could I figure out how to connect to a wifi network with a password or download packages to a USB. After a couple hours, I gave up, wiped the drive, and went back to Mint.

Nowadays I'm happier in my little comfort zone.

I recently broke the networking stack by uninstalling ca-certificates

I was using a slightly risky command to delete unneeded packages, and for some reason ca-certificates was on the list

At least the fix was simple. Boot the rescue iso and reinstall them

I was testing a custom initramfs that would load a full root into a ramdisk, and when I was going to shut down I tried to run rm -rf --no-preserve-root / to see what would happen, since I was on a ramdisk anyway. The computer would not boot after that because it nuked the UEFI options.

I was running Fedora. Something like 27 or so. I needed drivers. I don't remember if it was AMD or Nvidia, but they were only available on RedHat.

So I downloaded the RedHat drivers for the GPU and forced it to install. It worked! It was great.

Then when I updated the distro to the next release... everything failed. It was dropping into grub, but no video was output. Ooof.

So I ended up enabling a terminal console and connecting to it via a serial port to debug. I had to completely uninstall that RPM and I was never happy that it was properly gone. So a few months later I ended up reinstalling the whole OS.

On the plus side, I learned a lot about grub and serial consoles. Worth it.

Just yesterday I overwrote some pacnew files and borked user authentication for myself. Very rough time

chown 777 everything

Actually, I have a story that I'd consider an achievement even though it was extremely stupid and by all accounts should've bricked the system but didnt.

So I was on windows and wanted to install linux as a dual-boot on the main drive. The problem was that my mobo didnt like this particular and the only flash drive I've had, dropping it out mid-boot, before I got any usable terminal, so a usual install method wasn't an option. So I had this crazy idea to start a vmware vm in windows and pass the linux iso and the boot drive directly to it and try to install it live over the running system. Unfortunately, vmware guys thought of this and there's a check that disallows passing the boot drive to vms. So i created a bunch of .vmdks for another drive and fiddled with them in notepad until I somehow managed to trick vmware and at some point it started booting the same windows copy that I was sitting on. I quickly powered it off, added the linux iso and proceeded to install like I usually would. It did involve some partition shuffling, but, somehow, it went smoothly, linux installed, grub caught on, and even windows somehow survived, even though it was physically moved around on the disk. It serms that vmware later patched this out, because later in an attempt to re-create the trick of running the same copy of windows twice, but after updates to both windows and vmware, I was met with the same old error that boot drive is not allowed when trying to add that same virtual drive I had laying around.

I suppose it doesn't quite qualify as breaking the system in a funny or stupid way but it certainly was one of those stupid things that was easy to fix after a ton of trouble shooting, ignoring the issue for a while and trying to fix it again.

So i had an old pc where I had a failed hard drive which I replaced. Obviously I also accidentally unplugged my optical disc drive and plugged it back in. Now that failed drive was just a data drive so the system should have booted up no problem since the os was on a SSD but instead it got a kernel panic and got stuck at boot. Since it was late I left it at that and came back to that the next day where it would still not boot. So I unplugged the disc drive and looked up what it could be. Tried a ton of different possible solutions but every time I added that disc drive it would panic.

I eventually kind of gave up and just didn't use that disc drive at all and just had it as a paperweight in the system. Unplugged and all that. When my replacement SSDs for my old data drive and backup drive came in I tried again to get that optical drive working but to no avail. So I unplugged it again, got it all set up and ran into another issue where for some reason Linux couldn't properly use my backup SSD. So I investigated that as well and trough some miracle found a post on the forum from my Mainboard manufacturer... Turns out that particular Mainboard had a data retention chip on it that didn't like Linux.

So naturally I just plugged everything into the data ports that were not controlled by that chip and it all worked as intended.

Stupid dumb chip on a Mainboard, all I had to do was try the simple idea of unplugging and trying a different connector but instead I did all that other stuff first that didn't work and cost me so much of my time.

Moral of the story, when in doubt try and put stuff on different connectors and see if that fixes it. Might just be a dead connector for all you know. Or an incompatible chip on the Mainboard.

FWIW I bought that Mainboard long before I switched to Linux and didn't plan at all to switch at the time. But that's a different story.

I mistyped my SU password when setup the OS...

I can't remember what I did to break it, but back when I was in high school I was tinkering right before class and rendered my laptop unbootable. I booted into an Arch Linux USB, chrooted into my install, found the config file I messed with, then reverted it. I booted back into my system and started the bell ringer assignment as quickly as I could. I had one minute left when the teacher walks by, looks at it, and says that I did a really good job. She never knew my laptop was unbootable just 2 minutes earlier.

A few years ago I was having obscure audio problems on Ubuntu so I tried replacing pulseaudio with pipewire. I was feeling pretty cocky with using the package manager so I tried

sudo apt install pipewire

Installed successfully, realized nothing changed, figured maybe I had to get rid of pulseaudio to make it stick.

sudo apt remove pulseaudio

Just two commands. Instant black screen, PC reboots into the terminal interface. No GUI. Rebooting again just brings me back to the terminal.

I fixed it eventually, but I'm really not very computer literate despite using Linux, so I was sweating bullets for a minute that I might have bricked it irreversibly or something.

and a need to find another PC to flash an archiso to a flash drive ('cause ofc I didn’t have one at the time).

you can do that from your phone using etchdroid

i don't remember ever breaking my system in a terrible way, but when i started using linux (with linux mint) i uninstalled ca-certificates and i think that uninstalled the whole DE

The only time was within a VM. I accidentally wrote

rm -rf ./* while my cwd was /

I use absolute paths with -rf now, to prevent the error again.

Every other breakage I had was with apt shitting itself. It has always been fixable just annoying.

I now use Fedora, to prevent the error again.

sudo apt upgrade -y

To this day I can’t figure out why it killed the GUI and all terminal commands on a Mint install…

I somehow locked myself out of sudo when trying to give my user permission to read serial devices.

Had to reinstall.

I wanted to move my Arch VM to bare metal, so I copied out all the important bits. Then I wanted to move that copy to a new drive so I could boot into it.

I THOUGHT I'd MV all the files in the Arch install's etc directory using sudo MV /etc ...

I also (somehow) mashed my install's etc with Arch's and bungled both, with no live CD to help.

I learned a thing or two about absolute file paths...

Wanted a cool bootscreen on my Nixos machine - commented out the bootloader to troubleshoot, why my meme-boot-picture wouldn't show - after rebooting, it loaded straight into the BIOS and finally realized what I had done... Was able to fix it thankfully