this post was submitted on 17 Jun 2023
19 points (91.3% liked)

Lemmy

12579 readers
45 users here now

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to [email protected].

founded 4 years ago
MODERATORS
[email protected]
19
How can Lemmy scale? (self.lemmy)
submitted 1 year ago by HelloLemmySup to c/[email protected]
21 comments fedilink hide all child comments
 

I like the idea with Lemmy/kbin and the fediverse but theres something I dont understand perhaps.

If in the future Lemmy is very popular and someone wants to add their own server and federate with everyone then from that moment that new instance will get all new comments, posts, etc. from all other instances its federated with and must save them in its db. This means if Lemmy gets popular forget about little guys helping out spread the “load” because every intance still must take and save all new data. Thats a lot of processing power and storage. How can this work? I see in the future only a few instances will survive.

If somehow each instance was a node and only took care of its posts and comments and forward them to others upon request I can understand scaling but this is not how it works AFAIK. Another way would be with consensus algorithms where a node saves more thsn its own data but still not all.

you are viewing a single comment's thread
view the rest of the comments
[–] HelloLemmySup 1 points 1 year ago (1 children)

I think its not so much storage as it is requests I am worried. If a small instance wants to join and a few users subscribe to a few big communities then it needs to potentially proccess a lot of updates from the pubsub. I would imagine these messages are optimized so you get many updates within the same message but still.

Can the small instance be federated only one way? Meaning big can see small and comment in small but small cant see big communities (only comments made from big to its own small communities?)

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago)

In the future, we'll likely gain similar migration tools to mastodon. This means we'll be able to "split" any instances that get too large to function, if such a thing ever happens.

If half of the users move away, but stayed subscribed to a given sub on that overloaded server, this would still reduce the load significantly, because any interaction now goes to the new server, once, when it is synced. And then that server handles pushing it out to all the users.

As for custom setups, I don't see why they wouldn't be possible. The server software would simply have to be made to work that way. AFAIK ActivityPub, the standard, doesn't have anything in it that would make federation all or nothing.