this post was submitted on 05 Jul 2023
102 points (94.7% liked)

Lemmy.World Announcements

28383 readers
1 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to [email protected] e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 1 year ago
MODERATORS
 

Another day, another update.

More troubleshooting was done today. What did we do:

  • Yesterday evening @phiresky@[email protected] did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
  • @[email protected] created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
  • We started using this image, and saw a big drop in CPU usage and disk load.
  • We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
  • We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
  • We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @[email protected] , @[email protected] , @[email protected], @[email protected] , @[email protected] , @[email protected] for their help!

And not to forget, thanks to @[email protected] and @[email protected] for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 40 points 1 year ago* (last edited 1 year ago) (6 children)

server load is too low, everyone upvote more stuff so i can optimize more

edit: guess there is some more work to be done 😁

[–] [email protected] 5 points 1 year ago (2 children)

Upvote causes an endless spinner on Liftoff. 😁

[–] [email protected] 3 points 1 year ago (1 children)

I'm getting 504 gateway time outs when I try to upvote

[–] [email protected] 1 points 1 year ago (1 children)

For me it works way better than before

[–] [email protected] 1 points 1 year ago

seems like it may have been a temporary issue. It's clearing back up.

[–] [email protected] 1 points 1 year ago

It doesn't for me actually. Maybe just on Lemmy.world?

[–] [email protected] 4 points 1 year ago (1 children)

I don't understand your graph. It says you are measuring gigabit/sec but shouldn't the true performance rating be gigabeans/sec for a Lemmy instance?

[–] [email protected] 1 points 1 year ago

Web-ui is very smooth rn.. is this .world?
πŸ˜…
Joke aside, the improvement is like heaven and earth. Love it!. Good work teams!

load more comments (2 replies)
[–] [email protected] 8 points 1 year ago (4 children)

Test:

Upvote if you can see this comment. πŸ‘

load more comments (4 replies)
[–] [email protected] 4 points 1 year ago

You guys had better quit it with all this amazing transparency or it's going to completely ruin every other service for me. Seriously though amazing work and amazing communication.

[–] [email protected] 4 points 1 year ago

I love the smell of updates in the morning.

[–] [email protected] 3 points 1 year ago

The instance seems to be much better. Posting and commenting is not taking as long and loading times are way better. I hope things can stay this good or even get better.

[–] [email protected] 3 points 1 year ago* (last edited 1 year ago) (4 children)

As a data engineer, I'd be interested in hearing more about the SQL troubleshooting.

EDIT: It looks like [email protected] is a good place to subscribe to for more technical info on some of these performance improvements.

Also the Lemmy GitHub of course contains more information on bugs/enhancements/etc.

[–] [email protected] 2 points 1 year ago

Same, my job is like 80% SQL, so it'd be cool to see what is used in the background and maybe help improve things.

[–] [email protected] 1 points 1 year ago
[–] [email protected] 1 points 1 year ago (1 children)

Same here! If you find a way to contribute please give us an update!

[–] [email protected] 1 points 1 year ago

Just edited my comment!

[–] [email protected] 1 points 1 year ago
[–] [email protected] 2 points 1 year ago (1 children)

Thank you guys for your awesome work!

Also to other people: DONATE TO FOSS PROJECTS. If 50.000 people donate only 0.5€, we have 25.000€ for funding the servers, coding, motivating/ people etc. Just don't take a cup of coffee for 1 day. We are already 2 millions in Lemmy instances. We can build a decentralized world together!!

load more comments (1 replies)
[–] [email protected] 2 points 1 year ago (1 children)

Appreciate that these updates use the yyyy-mm-dd format :D

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago)

ISO-8601. The only correct format!

[–] [email protected] 2 points 1 year ago (2 children)

This is why having a big popular instance isn't all bad. It helps detect and fix the scaling problems and inefficiencies for all the other 1000s of instances out there!

[–] [email protected] 1 points 1 year ago (1 children)

This, if everyone kept just spreading out to smaller instances as suggested in the beginning, while still a sensible thing to do, no one would have noticed these performance issues. We need to think a few years out, assuming Lemmy succeeds and Reddit dies, and expect that "small instance" will mean 50k users.

[–] [email protected] 3 points 1 year ago* (last edited 1 year ago)

I sincerely doubt reddit will die anytime soon, it'll just exist as its own thing that it's new target audience gets bored with and moves on from in a few years when something new and flashy catches their eye in the app store. Just like they do all the other apps designed in exactly the same fashion that reddit is currently morphing into.

Meanwhile Lemmy will be slowly building it's communities up to be what reddit used to be.

[–] [email protected] 1 points 1 year ago

I'm actually kinda waiting a few releases to start promoting my instance anywhere, letting some other brave instance admins work the kinks out a bit first.

[–] [email protected] 2 points 1 year ago

The change is noticeable. Good job guys.

Thanks for the updates.

[–] [email protected] 1 points 1 year ago

Thank you so much for the hard work, time and money you spend into making lemmy.world run very smoothly. This much transparency is awesome for something that's being used so massively.

[–] [email protected] 1 points 1 year ago (1 children)

Even though i'm not from this instance, this is such a nice way of keeping the users posted about changes. I wish more companies (I know this is not a company) went straight to the point, instead of using vague terms like "improved stability, fixed few issues with an update" when things are changed. I hope all instance owners follow this trend.

[–] [email protected] 3 points 1 year ago

The owner of your instance has been a big help. You've also chosen a good instance!

[–] [email protected] 1 points 1 year ago

Thanks for the update! Things seem way speedier now ^^

[–] [email protected] 1 points 1 year ago

Thanks for the updates! Seeing the details of how you work through these early issues is valuable to those of us thinking of starting an instance.

[–] [email protected] 0 points 1 year ago (1 children)

Everytime I open a post and go back to previous page it scrolls back to top. Is this fixable? Im on windows 11, chrome.

[–] [email protected] 0 points 1 year ago (1 children)

Try wefwef. It remembers exactly where you were when you press back.

[–] [email protected] 1 points 1 year ago
[–] [email protected] 0 points 1 year ago

Thank you so much! I will be donating a few cappuccinos your way when my next check arrives. I really appreciate how awesome of a community you’ve brought together & all of the transparency with the updates (and the frequency) is astounding! Keep up the great work but don’t forget to take breaks :)

[–] [email protected] 0 points 1 year ago (1 children)

Am I getting this correct: the whole lemmy.world instance run in one single container on one single host?

[–] [email protected] 0 points 1 year ago (1 children)

You'd be surprised at how much performance this kind of setup can squeeze off. Often the limitation is more on the DB/storage than network handling and processing power.

[–] [email protected] 1 points 1 year ago

This. Most of the time, the bottleneck will be the database backend.

Curious if lemmy.world uses separate reader/writer instances.

[–] [email protected] 0 points 1 year ago* (last edited 1 year ago) (1 children)

It blows my mind with the amount of traffic you guys must be getting that you are only running one container and not running in a k8s cluster with multiple pods (or similar container orchestration system)

Edit: misread that a second was coming up, but still crazy that this doesn’t take some multi node cluster with multiple pods. Fucking awesome

[–] [email protected] 0 points 1 year ago (1 children)

A lot of containerization is unnecessary and perpetuated by people who don't know what they're doing.

[–] [email protected] 1 points 1 year ago (1 children)

Kinda shitty dig there but whatever, I’m a certified k8s admin, I would say I’m pretty damn good with containers.

With the amount of traffic I’m assuming lemmy.world gets I was just imagining it taking advantage of something more robust than just running Docker. It was more a comment on being impressed that lemmy must be damn efficient.

[–] [email protected] 1 points 1 year ago (1 children)

Heck yeah. I didn't mean to cast aspersions. I'm just tired of this trend of using technologies "just because" rather than having a legitimate need for them.

[–] [email protected] 2 points 1 year ago

No worries :D

[–] [email protected] 0 points 1 year ago* (last edited 1 year ago) (1 children)

Would HAProxy work better as a load balancer? For work we switched due to some issues with NGINX; so far, the service has been much more consistent with pretty much no downtime, even when restarting server hosts.

[–] [email protected] 1 points 1 year ago

Can be considered, for now it's working ...

load more comments
view more: next β€Ί