When you are bored, backup a VM then hard kill it and see if it manage to restart properly.
Software should be able to recover from that.
If it doesn't, troubleshoot.
this post was submitted on 03 Mar 2024
268 points (96.9% liked)
Selfhosted
38768 readers
369 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
That reminds me of Netflix's Chaos Monkey (basically in office hours this tool will randomly kill stuff).
When I built my home server this is what I did with all VMs. Learned how to change the start up delay time in esxi and ensured everything came back online with no issues from a cold built.
Rip VMware.
load more comments
(12 replies)
IMHO you're optimizing for the wrong thing. 100% availability is not something that's attainable for a self-hoster without driving yourself crazy.
Like the other comment suggested, I'd rather invest time into having machines and services come back up smoothly after reboots.
That being said, an UPS may be relevant to your setup in other ways. For example it can allow a parity RAID array to shut down cleanly and reduce the risk of write holes. But that's just one example, and an UPS is just one solution for that (others being ZFS, or non-parity RAID, or SAS/SATA controller cards with built-in battery and/or hardware RAID support etc.)
I agree that 99.999% uptime is a pipedream for most home labs, but I personally think a UPS is worth it, if only to give yourself the option to gracefully shut down systems in the event of a power outage.
Eventually, I'll get a working script that checks the battery backup for mains power loss and handle the graceful shutdown for me, but right now that extra 10-15 minutes of battery backup is enough for a manual effort.
Some of the nicer models of UPS have little servers built in for remote management, and also communicate to their tenants via USB or Serial or Emergency Power Off (EPO) Port.
You shouldn't have to write a script that polls battery status, the UPS should tell you. Be told, don't ask.
load more comments
(2 replies)
Did the services fail to come back due to the bad reboot, or would they have failed to come back on a clean reboot? I ugly reboot my stuff all the time, and unless the hardware fails, i can be pretty sure its all going to come back. Getting your stuff to survive reboot is probably a better spend of effort.
I didn't mean to imply that Services actually broke. Only that they didn't come back after a reboot. A clean reboot may have caused some of the same issues because, I'm learning as I go. Some services are restarted by systemctl, some by cron, some....manual. This is certainly a wake up call that I need standardize and simplify the way the services are started.
We've all.committed that sin before. Its better to rely on it surviving the reboot than to try prevent the reboot.
Also worth looking into some form of uptime monitoring software. When something goes down, you want to know about it asap.
And documenting your setup never hurts :D
On the uptime monitoring I've been quite happy with uptime kuma, but... If you put it on the same host that's down... Well, that's not going to work :p (I nearly made that mistake)
It's not the most detailed thing, but I just use a free account on cron-job.org to send a head request every two minutes to a few services that are reachable from the internet (either just their homepage or some ping endpoint in the API) and then used the status page functionality to have a simple second status page on a third party server.
You can do a bit more on their paid tier, but so far I didn't need that.
On the other hand, you could try if a free tier/cheap small vps on one of the many cloud providers is sufficient for an uptime Kuma installation. Just don't use the same cloud provider as all other of your services run in.
load more comments
(1 replies)
load more comments
(1 replies)
load more comments
(1 replies)
Yeah an unclean reboot shouldn’t break anything as long as it wasn’t doing anything when it went down. I’ve never had any issues when I have to crash a computer unless it was stuck doing an update.
I present to you the holy hardware compatibility table:
https://networkupstools.org/stable-hcl.html
Anything not listed there is not worth buying.
A lot of stuff on there isn't worth buying either, like anything from APC. If you want good stuff, just get Eaton.
But also you have to understand that UPSes aren't set and forget. The batteries need replacement every 3-5 years. And they're not for extended outages, they're mostly to bridge the gap between mains power going out and a generator starting up.
Personally I just have everything running from docker-compose, so I run one command and everything not running gets started. I don't worry about stuff being down for a bit.
Eatons batteries are usually really simple to switch, see
https://www.eaton.com/content/dam/eaton/products/backup-power-ups-surge-it-power-distribution/backup-power-ups/eaton-5s-ups/eaton-5s-120v-user-manual-700-1000-1500-lcd.pdf
For me they are meant for allowing a graceful shutdown in a powerout scenario and to protect the hardware behind them from power surges.
That's why you integrate with NUT. So you can automate a graceful shutdown when battery levels drop to a set level.
Tl;dr apc bad? I have 4 cyberpower so no experience with them
They got bought. They started to suck.
Now they've been 'sploited, they're overpriced, and they still schlepp the same bad software.
load more comments
(3 replies)
A general tip on buying UPSes: look for second hand ones - people often don't realise you can just replace the battery in them (or can't be bothered) so you can get fancier/larger ones very cheap.
Also, a larger capacity one is better, and it's likely you'll find a secondhand one with more capacity/features for a similar price.
load more comments
(2 replies)
Though I wonder if even besides adding an uninterruptible power supply (UPS) (writing acronym out for anyone else who would've had to Google it), this might be a useful exercise recovering from outages in general. This is coming from someone who hasn't actually done any self hosting of my own, but you saying you're still finding down services reminds me of when I learned the benefit of testing system backups as part of making them.
I was lucky in that I didn't have any data loss, but restoring from my backup took a lot more manual work than I'd anticipated, and it came at an awkward time. Since then, my restoring from backup process is way more streamlined.
Laptops
My favorite part about using an old laptop as a 24/7/365 plugged-in server is the anticipation of when the lithium battery will explode from overcharging.
"overcharging" doesn't exist. There are two circuits preventing the battery from being charged beyond 100%: the usual battery controller, and normally another protection circuit in the battery cell. Sitting at 100% and being warm all the time is enough for a significant hit on the cell's longetivity though. An easy measure that is possible on many laptops (like thinkpads) is to set a threshold where to stop charging at. Ideal for longetivity is around 60%. Also ensure good cooling.
Sorry for being pedantic, but as an electricial engineer it annoys me that there's more wrong information about li-po/-ion batteries, chargers and even usb wall warts and usb power delivery than there's correct information.
load more comments
(3 replies)
If you say it quickly enough it may sound plausible to some but this is not how battery technology works, as explained by @[email protected]
load more comments
(1 replies)
I'm a big fan of running home stuff on old laptops for this reason. Most UPSs give you a few minutes to shut down, laptops (depending on what you run) could give you plenty of extra run time and plenty of margin for a shutdown contingency.
Small, good value, quiet, power efficient, built in battery backup and server terminal. Laptops are dope for home labs!
A UPS should always be your first or second purchase if only for power conditioning and brown-out protection.
You should buy a UPS if those things are concerns for you. If not, then don't.
load more comments
(3 replies)
-
UPS, good idea.
-
backups too.
This is why I have about five of these bad boys: CyberPower CP1500PFCLCD.
One is in my utility room for my cable modem and our chest freezer, three back up my homelab and wifi AP, and one is for my office.
They've been bulletproof through storms, and when we've lost power, but not Internet I can't keep on working.
The big thing to look for is number of battery+surge outlets vs just surge outlets. Typically they top out at 1500VA - the more overhead for what you're powering, the longer you can go without mains power.
A screen/display is helpful for at-a-glance information like expected runtime, current output, etc.
Never heard of someone using a UPS on a Fridge/Freezer. Does it make a difference? Seems like the UPS would just died after 10-20 minutes and not really make much difference to your freezer.
load more comments
(1 replies)
UPS with usb allows you to configure a script to properly shutdown your server when a power outage happens and the UPS battery is about to run out.
This is why I gave up self hosting. It's great when it works but it just becomes an expensive second job. I still have Plex/Jellyfin etc but for emails and password vaults I just pay for external services.
if you don't want it to feel like a second job, you could always quit your first!
I could have the best self hosted setup.... living in a van, down by the river!
I like to host as many services as possible and I’m fine with it being a second job at times since this is my main hobby, but I actually agree with you on your examples. The three things I won’t self-host are:
-
Emails - I am not willing to put in the effort on this. Plus, my ISP blocks those ports so I’d already be into using a VPS even if I wanted to host this. I’d rather just pay someone else, like Proton.
-
Password manager - I actually did self-host Bitwarden for a long time, but after thinking about it for a while, I decided to take the pay someone else approach here too. I’m pretty sure I’m doing everything correctly, but I’m not a security expert. I’d rather be 100% sure my passwords are in safe hands rather than be 95% sure that I’m doing everything right on this one.
-
Lemmy - I’ve heard about (luckily never seen) CSAM attacks on Lemmy/Kbin and will not risk that kind of content being downloaded because I’m federated with an instance dealing with those attacks. I’m happy to throw a couple bucks at lemmy.world’s Patreon and let them handle that.
load more comments
(1 replies)
Yeah if you self host, a UPS is very important.
Not APC. (At least for Windows) trashy software.
It doesn't need the software, it gets recognized by the system as a battery
load more comments
(3 replies)
load more comments
(4 replies)
I use a laptop and external jbod covered with a low power ups. As other said, the point is to bridge powergaps now long term working powerless. I live in the countrisied, so small powergaps happens specially when my photovoltaic don't produce (no, i have no battery accumulators, too expensive)
Figure out how much power your servers use on average with the help of a wattage meter, then enter that number and how many minutes battery backup you want in Eatons UPS Power Calculator to find a suitable unit. I'm sure other vendors have similar tools too.
view more: next ›