this post was submitted on 04 Jul 2023
994 points (99.1% liked)

Lemmy.World Announcements

29104 readers
3 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to [email protected] e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won't work as desired.. we see several issues:

Performance

  • Loading is mostly OK, but sometimes things take forever
  • We (and you) see many 502 errors, resulting in empty pages etc.
  • System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

  • Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
  • 2FA seems to be a problem for many people. It doesn't always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @[email protected] is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 254 points 1 year ago (3 children)

I'm just excited to be back in the Wild West again -- all of the big players had bumps, at least this one is working to fix them.

[–] [email protected] 72 points 1 year ago (1 children)
[–] [email protected] 13 points 1 year ago

I’d rather have to deal with hiccups and bumps along the way, because the community only grows more each time.

[–] [email protected] 25 points 1 year ago (1 children)

I still remember early reddit days of 502 it went through, 504 try once more.

load more comments (1 replies)
[–] [email protected] 14 points 1 year ago

Yee-haw, mother fuckers.

[–] [email protected] 110 points 1 year ago* (last edited 1 year ago) (11 children)

That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

Lemmy has a memory leak? Or, should I say, a "lemmory leak"?

[–] [email protected] 43 points 1 year ago (1 children)

A pretty bad one at that...

[–] [email protected] 33 points 1 year ago (4 children)
[–] [email protected] 55 points 1 year ago (1 children)

Rust makes holes and that's how leaks happen

[–] [email protected] 10 points 1 year ago

Underrated comment

[–] [email protected] 27 points 1 year ago

Rust protects you from segfaulting and trying to access deallocated memory, but doesn't protect you from just deciding to keep everything in memory. That's a design choice. The original developers probably didn't expect such a deluge of users.

[–] [email protected] 12 points 1 year ago* (last edited 1 year ago)

Leaking memory is safe

[–] [email protected] 9 points 1 year ago

I’m calling it - if there’s actually a memory leak in the Rust code, it’s gonna be the in memory queues because the DB’s iops can’t cope with the number of users.

[–] [email protected] 13 points 1 year ago

On some of the latest release candidates I think so.

[–] [email protected] 12 points 1 year ago

Lemmy has a memory leak? Or, should I say, a β€œlemmory leak”?

A lemmory meek, obviously!

load more comments (8 replies)
[–] [email protected] 94 points 1 year ago

Really appreciate all the time and effort you all put in especially while Lemmy is growing so fast. Couldn't happen without you!

[–] [email protected] 56 points 1 year ago (1 children)

As a game dev for bigwigs I know all too well about memory leaks, and so very much appreciate your patch notes, updates, and transparency. You're doing great with such fast exponential growth

πŸ’™ Thanks for your hard work!

[–] [email protected] 11 points 1 year ago

Amen. So happy to be done with Reddit.

[–] [email protected] 53 points 1 year ago (2 children)

What was that? We're going to need more and better hardware soon, and you have a Patreon and a paypal on the sidebar?

Yeah, that sounds pretty reasonable, we can work with that.

[–] [email protected] 13 points 1 year ago (2 children)
[–] [email protected] 11 points 1 year ago (1 children)
[–] [email protected] 9 points 1 year ago (1 children)

I'm thoroughly confused by most of what is going on. I'm using kbin on mobile. The interface isn't terrible and it fills the random surfing void that I am used to.

[–] [email protected] 18 points 1 year ago (1 children)

Just a heads up, you're using kbin which is a different "website" than lemmy. You are also seeing a thread that originated in lemmy. That means both kbin and lemmy can see each other's content, but the website UI is different between them so if you read about some UI element and can't find it in kbin that might be one of the reasons.

That being said, kbin also has a sidebar with useful information. In the mobile site you can find it at the very bottom on the page, since it doesn't fit to the side in such a small screen.

[–] [email protected] 11 points 1 year ago* (last edited 1 year ago) (1 children)

Omg I never scrolled that far down lmao, now I can go block 196 and the other shit posting magazines

load more comments (1 replies)
load more comments (1 replies)
[–] [email protected] 10 points 1 year ago

Just signed up for the 8€/month tier, hopefully my blue checkmark will arrive soon!

[–] [email protected] 50 points 1 year ago

Could I get a discord invite? I’m an ex sysadmin with a. Lot of free time

[–] [email protected] 45 points 1 year ago (1 children)

@ruud > That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

Hmm, makes me curious if there is a Lemmy memory leak, or simply that the load wants to stabilize above of the RAM you have? I hope contributions can help you with another 32 GB RAM? Thank you for your work! 🍻

[–] [email protected] 48 points 1 year ago (3 children)

We have 128GB of RAM. It just skyrockets after a while!

[–] [email protected] 21 points 1 year ago

@ruud Oh damn. This spontaneously sounds crazy but I’m admittedly a novice at servers on this scale.

load more comments (2 replies)
[–] [email protected] 44 points 1 year ago (3 children)

As somebody who flocked to Voat during the height of the Ellen Pao controversy and remembered the site being rendered unusable for whole days at a time from the Reddit Hug of Death, I'm remarkably surprised at how well Lemmy.world has held up. I thought the fediverse would have truly crumbled from this exodus.

load more comments (3 replies)
[–] [email protected] 39 points 1 year ago (1 children)

The work you're doing is greatly appreciated! It's like you invited half the internet into your house. I feel like I should've brought a cake or something

[–] [email protected] 13 points 1 year ago (1 children)

You should consider setting up a small donation to the site, if you can afford it. If you throw $5 a month at them, you're not only covering your own hosting fees, but covering for some of the folks who can't afford to as well. And maybe Ruud gets to buy some beans.

load more comments (1 replies)
[–] [email protected] 37 points 1 year ago (1 children)

It’s a ton better this afternoon! Thank you!!

[–] [email protected] 14 points 1 year ago

I’ve noticed that too!

[–] [email protected] 37 points 1 year ago

Of course these performance issues are a bit annoying, but I gotta say that I love these updates and explanations here. Great communication, keep it up, please!

[–] [email protected] 34 points 1 year ago* (last edited 1 year ago) (2 children)

That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

who'd have thought memory leaks would be possible in Rust 🀯

(sorry not sorry Rust devs)

load more comments (2 replies)
[–] [email protected] 34 points 1 year ago* (last edited 1 year ago) (1 children)

Huge respect for what you've built here, but it might be worth reaching out to the lemm.ee admin. I only know enough DevOps and cloud hosting to be dangerous, not helpful. But his instance seems stable and scalable. He might be able to offer some insight into the issues here

[–] [email protected] 55 points 1 year ago

Yes he's one of the other admins in our Discord, he's very helpful!

[–] [email protected] 31 points 1 year ago (1 children)

Keep up the good work!

I created an account in lemm.ee until the issues are fixed. Then I will happily go back to my lemmy.world account.

[–] [email protected] 41 points 1 year ago

Lemm.ee is also a good choice!

[–] [email protected] 28 points 1 year ago (1 children)

Thank you so much for your hard work and for fixing everything tirelessly, so that we can waste some time with posting beans and stuff lol.

Seriously, you're doing a great job <3

load more comments (1 replies)
[–] [email protected] 10 points 1 year ago* (last edited 1 year ago)

Thank you for your hard work! β™₯️

[–] [email protected] 10 points 1 year ago

Thanks for the transparency and keep up the good work!

[–] [email protected] 9 points 1 year ago (5 children)

Thanks u/ruud!

How are you managing all of this in addition to having a life outside of Lemmy? This kind of went from 0-60 in a blink.

[–] [email protected] 22 points 1 year ago (1 children)

Same happened with mastodon.world in November. Family goes first, then work, and then all of my hobbies, of which this is one. (But the one taking up most time at the moment...)

load more comments (1 replies)
load more comments (4 replies)
load more comments
view more: next β€Ί