Lemmy

12884 readers

20 users here now

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to [email protected].

founded 5 years ago

MODERATORS

[email protected]

Hypothetically, if all lemmy users were on instances of a very small number (< 20), would this be better or worse for scaling? (lemmy.ml)

submitted 2 years ago by [email protected] to c/[email protected]

11 comments fedilink hide all child comments

And I guess this question is two parts: 1. Regarding the current lemmy implementation, and 2. The activityPub protocol in general

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 5 points 2 years ago (1 children)

Every complex system (and federated systems like Lemmy qualify) has more than one potential bottleneck that can become a problem in different conditions.

Right now, the common performance bottleneck for Lemmy instances is heavy database reads caused by users browsing. Many of these queries are written inefficiently and can be optimized, there are things that can be done in Postgres to scale as well. But browse traffic is one kind of workload load that can reach limits, and it gets stressed when lots users are active on one big instance.
Federated networks CAN experience federated replication load when there are lots of instances to deliver federation messages to. If I comment on this post, and the server hosting the community has to deliver the comment to (pinky to mouth) one million instances... that's a different kind of workload and it gets stressed when there are lots of different instances subscribed to a single community.

The goldilocks zone is where there is a medium number of medium sized instances. Then each federation message can efficiently power browse traffic for a lot of users, and no one instance gets overwhelmed with browse traffic.

In practice, this is not how networks organize. There will both be instances that are "too large" and also lots of small instances. Right now, the Lemmy network is small and federation traffic is not a meaningful bottleneck. Browse traffic is, and that's what the devs are working on. But with time, the limits of both these things can be pushed further out improving scalability of the etwork in both directions.

[–] [email protected] 1 points 2 years ago

Makes sense, I also think the original post would make more sense if it were talking about the distribution of large communities. For instance, if there wasn't so many of the largest communities on the same 2-4 servers, then I think browsing would be a lot smoother. In addition, I think users who say care a lot more about a certain community may choose to use that other less populated server to be their home, as it is the home to that community.

(Just a thought, I have done ZERO work with federation or ActivityPub)