From Wikipedia, the free encyclopedia
Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).
Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.
Community icon by Alpár-Etele Méder, licensed under CC BY 3.0
It definitely sounds like a hardware issue since it has survived multiple disk wipes and distro changes.
If you're now getting I/O errors that won't even get you booted, it sounds to me like drive failure is imminent.
Yeah, I just ordered a new SSD. I'll give that a try when it arrives tomorrow. Smart says it passed, but suspect my SSD enough that I think it's worth it to just try another SSD.
Make sure smart is enabled in the bios. Too often I see it flipped off by default.
I have the same problem with an older macbook air and linux, and I bought a new SSD for it (will be here at the end of the month, since it's a special pre-ssd model). Husband, who's an engineer, said that such hard freezes are usually the ssd's fault, and not the memory's (he said memory creates other kinds of crashes, but not this kind of hard freeze).
Well, I just reinstalled on a new SSD this morning. Fingers crossed it all works out! It usually takes a few weeks for the issues to start happening each time, so I guess I'll just wait in agony until then. 🥲
Try swapping to BFQ io scheduler and see if that makes a difference.
Thank you for the suggestion. I am unfamiliar with this, but I am reading about it now.
For what it's worth, I've never had to change my io scheduler in the nearly twenty years I've used Linux. You can check your current scheduler with the following command: cat /sys/block/sda/queue/scheduler (change the block device to whatever yours is...sda, nvme0n1, etc.).
In my case, it was already bfq: one mq-deadline kyber [bfq]
Even with nvme drives which supposedly "don't need" to use BFQ, I STILL always swap it since it maintains responsiveness across the system during heavy IO loads. I used to have similar full system freezes when downloading steam games which notoriously overload your IO in Linux. BFQ was the solution every single time.
Edit Try following the instructions detailed in this post to add a systemd rule to set the scheduler: https://stackoverflow.com/questions/1009577/selecting-a-linux-i-o-scheduler
The second answer that shows an actual rules.d file example has always worked for me. If using nvme or old school spinning rust you'll need to change it up a bit. Instead of "noop" set it to "BFQ".
How long did you run the memtest for? Ideally it should run a couple of times, since just a single pass might not detect any errors.
But it's weird that it happens when you try to update. Could it maybe be related to your network hardware, either LAN or WiFi? If you're using WiFi, try LAN, or vice versa. Perhaps even a USB dongle, and disable the onboard network hardware completely.
It did 4 passes by default. It took almost 3 hours to run on 16 GB of RAM.
It's possible there is an issue with the WiFi. I had lots of WiFi issues on this laptop when I used Windows (micro freezing when using typing into an SSH shell, and pings would drop at the same time), but since switching to Linux, those went away. I'll definitely keep that in mind. I'll try using wired network if the issues come back after swapping SSDs.
Try firing up btop in a terminal before you kick off an update in another, that should give you a better indication of what's happening when the system hangs. Turn on kernel "show kernel threads" so you can spot anything kernel side eating CPU.
Check your kernel journal from the last boot after a freeze, there should be some indication of what went wrong before you rebooted the machine. journalctl -k -b -1 will show you what was going on with the kernel before the machine froze.
Edit: things to watch for in btop: CPU pegged at 100% and no disk activity? Look at the top process, there's your offender. Super high IO latency but otherwise the system looks normal? Try another drive. Memory completely used and swap endlessly thrashing? Find something to kill to make more memory available.
Turn on "show kernel threads" in btop, they're off by default, so you can see if something in the kernel is eating CPU time.
I will try this once I get my system back up and running tomorrow! I'm going to install the distro on a new SSD and see what happens.
I made a couple of edits above re: btop and troubleshooting, if you're not used to diagnosing hardware and kernel issues they might help
sounds like maaaaybe too little ram and no swap? what is your ram size and do you have any swap or zram enabled? i kinda doubt it because multiple distros should have a swap space or zram on by default on a fresh install but maybe not or you explicitly chose not to and it's running out of memory.
I have 16GB of RAM and 16GB of swap as a swap partition. Though I have also tried a 16GB swapfile and saw no difference. I don't know about zram.
If you haven't intentionally setup zram swap you're almost certainly not using it.
Even then the recent LRU swap changes have largely eliminated pathological swap behavior cascading into an unusable system state. Those changes went in like 2y ago and should have been picked up by most distros by now.
What hardware? And can you narrow down when during updates?
I had this problem on Arch on a 5 year old Lenovo laptop with an Nvidia 1660ti GPU. With judicious use of set -x I narrowed it down to systemd daemon-reload.
I actually changed my ext4 journal mode and added a pacman hook in that calls sync before any systemd hooks ran, after the second time half of the package updates got lost due to the freeze.
Because the problem only happened most times, and usually not soon after a reboot, I can't prove it, but the problem hasn't reoccurred since I switched the Nvidia driver to the open flavor.
It's a 2021 Asus Zephyrus G15 with an AMD CPU and an Nvidia GPU. I got an aftermarket SSD off of Amazon so I could dual boot with Windows, but I haven't booted back into Windows a single time since installing Linux. Though that might be a good test.
I can try set -x once I reinstall my distro and get it back up and running tomorrow, as it is currently borked. Since zypper does all the downloads then all of the installs, I was able to see that it always happened during the install phase, not the download phase.
I am definitely interested in the possibility of it being related to the proprietary Nvidia driver. When it happened yesterday, the proprietary Nvidia driver was being updated (not sure at that exact time. But it was in the list of packages to be updated). I'll keep an eye on that for sure.
I had a similar problem with hard lockups especially when doing package updates (Arch). After seeing a report on Gaming on Linux about the Nvidia 550 driver (I think it was that one) causing freezes, I uninstalled it and just ran on the intel igpu. Never had a single freeze again. Waited for 555 driver, installed that, and immediately got lockups during package updates (and randomly sometimes) again. I've now installed the nvidia-open package to see if it fixes it, and so far so good.
Well, I reinstalled this morning and I'm keeping the nouveau driver this time instead of going with the proprietary. I don't play play games on this laptop anymore since I set up sunshine/moonlight, so it shouldn't be a problem.
So now I'm thinking it might be the SSD, but I'm not even sure how to check that.
There's definitely some utilities for that. SMART or something? They'll at least show you rough drive health/wear.
Edit: nevermind, another responder recommended that already.