this post was submitted on 01 Jul 2024
148 points (97.4% liked)
Lemmy.ca's Main Community
2758 readers
2 users here now
Welcome to lemmy.ca's c/main!
Since everyone on lemmy.ca gets subscribed here, this is the place to chat about the goings on at lemmy.ca, support-type items, suggestions, etc.
Announcements can be found at https://lemmy.ca/c/meta
For support related to this instance, use https://lemmy.ca/c/lemmy_ca_support
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Something got into a weird state and restarting either the backend or frontend didn't help. Taking the entire stack down and then bringing it back up, resolved it.
It's weird since it crashed at 1am and at 3am we gradually restart all backend and frontends, so that automatic restart should have fixed it too. All the containers reported healthy, but nginx wasn't reporting any available frontends.
I suspect some sort of weird lemmy bug, but we'll just have to improve monitoring for now and try to debug this more if it happens again.