this post was submitted on 26 Mar 2024
636 points (96.2% liked)

linuxmemes

21448 readers
1091 users here now

Hint: :q!

Sister communities:

Community rules (click to expand)

1. Follow the site-wide rules

2. Be civil
  • Understand the difference between a joke and an insult.
  • Do not harrass or attack members of the community for any reason.
  • Leave remarks of "peasantry" to the PCMR community. If you dislike an OS/service/application, attack the thing you dislike, not the individuals who use it. Some people may not have a choice.
  • Bigotry will not be tolerated.
  • These rules are somewhat loosened when the subject is a public figure. Still, do not attack their person or incite harrassment.
    • 3. Post Linux-related content
  • Including Unix and BSD.
  • Non-Linux content is acceptable as long as it makes a reference to Linux. For example, the poorly made mockery of sudo in Windows.
  • No porn. Even if you watch it on a Linux machine.
    • 4. No recent reposts
  • Everybody uses Arch btw, can't quit Vim, and wants to interject for a moment. You can stop now.
    •  

    Please report posts and comments that break these rules!

    Important: never execute code or follow advice that you don't understand or can't verify, especially here. The word of the day is credibility. This is a meme community -- even the most helpful comments might just be shitposts that can damage your system. Be aware, be smart, don't fork-bomb your computer.

    founded 1 year ago
    MODERATORS
    [email protected]
    [email protected]
    [email protected]
    636
    What is the most difficult problem that you have fixed in linux? (lemmy.world)
    submitted 8 months ago by [email protected] to c/[email protected]
    164 comments fedilink hide all child comments
     
    you are viewing a single comment's thread
    view the rest of the comments
    [–] [email protected] 10 points 8 months ago

    A couple months ago, I made a Palworld server box out of a spare motherboard assembly (mobo, processor, ram) from a computer I had recently upgraded.

    I didn't have any spare drives lying around, so I plugged in 7 USB flash drives and made them into a RAID array. Not a true RAID array, but a BTRFS filesystem with volumes spread onto each flash drive, with the data redundancy set to raid1, and the metadata redundancy set to raid1c3.

    It worked... in the sense that I never lost any data. It certainly didn't work in the sense of having good uptime.

    The first problem was getting it to boot right. The boot line in GRUB had "root=UUID=..." instead of a specific drive named. That is normal. However, in BTRFS multi-volume filesystems, all the volumes have the same UUID. So the initrd was only waiting for a single drive matching that UUID, then trying to mount it as the root filesystem. This failed, because the kernel had not yet set up the other 6 USB drives, and this BTRFS filesystem needs all 7 volumes present. Maybe 6, if you used the "degraded" mount option.

    The workaround was to wait for this boot process to fail, at which point you get dropped into an initrd shell. Then, you look at all the drives and make sure they're all there. And then... I don't exactly remember what happened next. I think it was some black magic that erases your mind in the process. I somehow got it booted from the initrd shell.

    Installing Steam and the Palworld server worked ok, and it even ran for a few hours before crashing overnight.

    The next morning, I tried rebooting it. Unfortunately, the USB drives weren't all appearing. Turns out the motherboard had some bad USB ports, some sometimes-bad USB ports, and a maybe-bad PCIe bus, because the PCIe USB expansion card I plugged in had weird problem that it had never had before.

    I found the most reliable ports and plugged the drives in there. But you can't just replug them in the initrd. It doesn't have USB hotplug support. So each time it tried to boot with not all the drives there, I restarted it again until one time I finally had all the drives.

    I changed the GRUB boot line to "root=/dev/sdg1" . This made it wait for all the drives to load, in any order, and whichever one was last would be mounted as the root filesystem (but the kernel would automatically include all the others too, since they were successfully initialized).

    The bad USB ports kept bringing down the server every day or two. I bought a cheap NVMe drive and added it to the BTRFS filesystem, and then removed all the USB drives except the largest. That fixed the reliability. It's been like that since.

    Now, to boot the server, all I have to do is change the GRUB boot line to "root=/dev/sdb1" . Since the NVMe drive is much faster than the USB drive, it always initializes first. If the initrd waits for sdb2, then it will always have both drives initialized when it tries to mount the root filesystem.

    I could add that to the grub.cfg, or come up with some other more permanent solution, but I'm not planning on rebooting this server ever again. My friends fell off Palworld, and I gave a shutdown date that's about a week away. And the electricity is pretty reliable here.