Asklemmy

Doesn’t the fact that every Lemmy server has a copy of every federated post mean that if Lemmy takes off, only a few people with strong donation feeds can afford to survive?

It's not precisely true that every Lemmy instance stores every post. A given Lemmy instance will store a given post if and only if:

• A user on the storing instance is "interested" in that post. Being interested has a somewhat complicated definition that I myself don't fully understand, but two examples of being interested include being subscribed to the community BEFORE the post was made (importantly, the storing instance doesn't fetch the forever backlog of posts just because someone subscribes... rather... it asks to receive FUTURE posts), or a user searches a post by url.
• Subject to some caching policy. I just learned this today from another comment, and maybe it doesn't apply to Lemmy, or doesn't apply yet... but we already know that the storing instance doesn't fetch historical posts going back forever. It could also decide to forget posts older than a certain time.

The first of these is most important though, because it means that posts and comments that no one is interested in don't get shipped around the federated network. And this leads to the property that the size/cost of a Lemmy instance is going to depend on the size of the "active" usage. A single user Lemmy instance subscribing to a handful of communities will always be small and cheap, because it doesn't subscribe to much content. A bigger Lemmy instance need not scale to the entirely of content in the lemmyverse, but rather to the "active set" of posts and comments its users interact with this month. That could get big, but what the Lemmy devs are saying (sorry no link, I've read too many posts lately to remember all my sources) is that user-traffic browsing the local DB of the Lemmy instance is dwarfing the replication load, which is great news because user browsing is much easier to optimize than federated replication.

If there’s an active forum (sub-lemmy?) on a server that has to spin down, the history stays on the remaining active ones, but I assume the only option is forking?

(FYI, the thing you subscribe to is called a community in Lemmy. Some folks say sublemmy, but this is a redditism that isn't used in the code or official docs. It's a "community", which is why the url for a community is hxxp://my.lemmy.social/c/mycommunity. The "c" in the middle stands for community.)

Well, we've already talked about caching and expiry. It's not clear to me than any Lemmy instance other than the one that hosts the community is required to keep the ENTIRE post/comment history (though yeah the active/recent ones will be all over the federated network).

I haven't lived through a major instance shutdown, maybe an old-timer can weigh in here. Speculating, I'd think there would be 2 options:

• BEFORE the old instance shuts down (or using a third big community like [[email protected]](/c/[email protected]) to coordinate), make a new community on a different server and the mods post telling everyone to subscribe there. The new community would be... well... new. It wouldn't have the old posts, it would be made from scratch. The only things that would bind it to the old community are the mods that come over, the users that follow them, and the culture.
• Optionally, using database tricks, or a migration tool (that I don't think currently exists, but could almost certainly be created)... after doing all of the above... someone direct db access on both instances (aka an admin) could copy the the old posts to the new community. This might be a terrible idea for federation reasons, or it might be prohibitively complicated because of the db schema... but it feels to me like it COULD work. I'm not aware of it having been done before, or of any tooling that makes it "easier" for someone who isn't pretty strong at doing complex data migrations in Postgres.

But the crux of your question is a real concern. If the admins of a major Lemmy instance hosting a bunch of big communities gets bored and shuts down, or gets pissed and takes their toys and goes home... I believe that event would be fairly disruptive but not an existential threat to the lemmyverse unless it happens frequently and nobody responds by making tools to do community migrations. But it's my belief that if this happened once or twice people would improve the capabilities around community migration.

Doesn’t the fact that every Lemmy server has a copy of every federated post mean that if Lemmy takes off, only a few people with strong donation feeds can afford to survive?

Yes I’ve seen something like that on mastodon already. Though the caching is scaled by time so you can just say to cache only last 24 hours (or less) which will scale down storage requirements.

There's definitely scope for this.

I know I would never pay for facebook / twitter / reddit. Not 1 cent, but I'm happily contributing to fosstodon on patreon each month - and I don't think I'm alone.

I think there's a not-insignificant segment of the population that are weary enough of the advertising revenue model that we're happy to pay to avoid it. The expectation that everything on the net should be free has ruined the net IMO.

We're all learning. And I want to stress -- and maybe should have in my original reply -- that I wasn't really trying to criticize. My reply was just going to be so far down, I figured it'd only be noticed by a handful of people.

The way you intuited things is the way a lot of people have intuited things on the Fediverse. There were big blow-ups on Mastodon about this 6 months ago, as users spoke out about instance admins defederating from other Mastodon instances who hosted accounts that chronically broke local rules.

It became clear that people interpreted these defederations as denying new local users' access to "parts of Mastodon", as if there were was some central object called "Mastodon" and local admins were being nannies about letting some of their users wander over to some parts of it, when in reality it was a case of the local admins refusing to host content from other websites that gave shelter to individuals who were abusive or otherwise breaking rules that existed on the local site.

If that makes sense.

If you can grok that we're basically all on different, independent web forums, and there's just an implicit agreement between the forums to cross-post and share content, you can better grok why somethings that will happen here happen.

Thank you for the response. EDIT: I read another comment (also from you) about the way federation actually works and I was totally wrong in my assumptions so I deleted them. I suppose I should be leaving these questions up to the people building the platform, rather than a layperson such as myself.

Can't help but ask though, is the federation somewhat comparable to a blockchain? In a blockchain each member has a full copy of the chain, and in the federation each instance has a copy of all parts of the chain it cares about. If one peer goes down, the other peers can continue to function. Cool.

This comment should be stickied or included in some sort of tutorial though.

https://lemmy.world/comment/20357

I found another thread discussing this topic and most people seem to expect that there will be natural consolidation and fragmentation of communities over time. There was also the suggestion that individual communities could federate such that posts from two different meme communities in different instances could be merged into the same feed on the user end, which sounds like a pretty good compromise.

https://lemmy.ml/post/1163258

It'll be interesting to see how similar communities from different instances will work out. I've noticed a lot of Technology instances from beehaw.org, lemmy.ml, lemmy.ca, and others.

How will the fediverse as a whole determine which Technology instance is the "main" one? I assume it'll be whichever one has the largest subscriber count accumulated over time.