this post was submitted on 05 Aug 2023
44 points (100.0% liked)
Reddthat Support -> Has moved
305 readers
1 users here now
Reddthat Community Support Forum
Before posting, have you read the rules?
Introductory Required Reading
You are ready to start your adventure on Reddthat but are still unsure? That's fine! You've come to the right place.
- Ideas? Post-em
- Issues? Post-em
- Queries? Post-em
- Ideas to help Reddthat? Post-em
Alternative Support Forums
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Updates hiding in the comments again!
We are now using v0.18.3!
There was extended downtime because docker wouldn't cooperate AT ALL.
The nginx proxy container would not resolve the DNS. So after rebuilding the containers twice and investigating the docker network settings, a "simple" reboot of the server fixed it!
upstream lemmy-ui
&upstream lemmy
. These are DNS entries which are cached for a period of time. So if a new container comes online it doesn't actually find the new containers because it cached all the IPs thatlemmy-ui
resolves too. (In this example it would have been only 1, and then we add more containers the proxy would never find them). 4.1 You can read more here: http://forum.nginx.org/read.php?2,215830,215832#msg-215832I get notified whenever reddthat goes down, most of the time it coincided with me banning users and removing content. So I didn't look into it much, but honestly the uptime isn't great. (Red is <95% uptime, which means we were down for 1 hour!).
Actually, it is terrible.
With the changes we've made i'll be monitoring it over the next 48 hours and confirm that we no longer have any real issues. Then i'll make a real announcement.
Thanks all for joining our little adventure!
Tiff
For number 4, can you set a cron job to constantly flush DNS cache?
It's the internal nginx cache. It /shouldn't/ be a problem once I update the configuration to handle it.
We can add a
resolver
line withvalid=5s
so it will recheck every 5 seconds instead of whatever the internal docker TTL cache is.