Hello!
I am sunaurus, the head admin of lemm.ee. Ever since I created my instance, I have been following a lot of public and private discussion channels between different parties involved with Lemmy. As I’m sure many others have also noticed, the discussions in such channels sometimes get heated, and in fact recently, I feel like there has been a constant trend in these discussions towards a lot of demands, hostility, negativity, and a general lack of empathy between different participants in the Lemmy network.
I am writing this post for a few reasons:
- I would like add a bit of positivity by expressing my gratitude towards every single person who has helped improve Lemmy.
- I want to speak up in defense of different people who have been receiving negativity lately.
- There are a few false rumors spreading on Lemmy, which I would like to try and counteract with very simple evidence.
- I want to remind everybody that at the end of the day, all of us care about building and improving Lemmy. We all have the same goal, and it’s too easy to lose sight of that.
I will split up what I want to say in this post by different user groups - users, mods, admins and developers. I understand that many people belong to several (or even all) of these groups, but I just want to highlight the value of, and express my gratitude to each group separately.
Users
At the end of the day, Lemmy would not be worth anything without the users. Users bring Lemmy to life by posting great content, getting involved in discussions in comments, helping surface interesting content for others through voting and even keeping the platform clean through reports. I am extremely thankful for all the users who have given me so much enjoyment on this platform.
I believe that users often get treated unfairly on Lemmy based on what instance they are participating from. I’m sure so many of you have noticed comments around Lemmy along the lines of “Oh, another user from , I’m going to completely ignore your stupid takes”. I’ve also many cases of people treating users as second-class citizen if they are not on the same instance - for example, I’ve seen users who are active and valuable participants in communities on another instance receive comments like “why are you participating in our discussions, go back to your own instance”. In my opinion this is completely counterproductive to the whole idea of federation. On a human level, I can understand it - you’re far more likely to notice or remember what instance somebody is posting from if you have a negative experience. As a result, as time goes by, people tend to develop negative views of each instance, despite potentially having had many positive interactions with other users of those same instances. The message I want to put out here is that instances, especially bigger ones, are not monoliths - do not judge users based on what instance they are browsing Lemmy from, judge them by their actual words and actions.
Mods
There are some excellent communities already on Lemmy, and these communities are all continuously being built up and maintained by mods. Mods put in huge amounts of their free time and energy in order to provide spaces for all Lemmy users. They form the first line of defense against bad actors, they keep communities alive and often receive no praise, only criticism. I am very grateful to everybody who has dedicated time to building communities on Lemmy.
Users rarely notice the lengths mods go to in order to keep communities running smoothly - mods more often than not only get noticed when users disagree with some mod actions. I believe mods deserve a lot better than this. Constructive criticism can of course be useful to improve communities, but it must be balanced with empathy and kindness towards people who have been putting in effort to provide something for users. Remember that there is another human being reading your words when you start writing about the mods of any particular community. Users who are not happy with mods of a certain community always have the opportunity to start their own community and run it as they like.
Admins
Admins provide two main key functions for the network:
- Taking care of the actual infrastructure of Lemmy
- Working as a higher level defense against bad actors, in cases where mods are not enough
I can tell from my own experience that being an admin of a bigger instance requires constant energy and attention. I don’t believe that there is a single medium-to-big instance where the admins have not put in hundreds (if not thousands) of hours of their free time, as well as in many cases, probably their own money. This is a service which admins provide for free, and it is necessary in order to keep the Lemmy network healthy. I have endless respect for anybody who is willing to put themselves in the position of a Lemmy admin.
I have seen awful messages towards admins from all the other groups listed here, including other admins. These messages range from condescending and rude, to downright hateful. I have seen admins treated as useless and their work taken for granted. I have seen people getting frustrated with admins for not spending every waking minute on Lemmy. I have seen some users consistently spreading provably false rumors about particular admins in an effort to tarnish their reputation on Lemmy.
Before you take out frustration on admins, please remember that they are also humans who have been working tirelessly to improve Lemmy in their own way.
Also, a reminder: the absolute best feature of Lemmy is that users are free to pick their instance - and as a result, users are also free to pick their admins. Even more than that, users can always become their own admins by spinning up their own instance. Yes, this requires dedication, effort, and research, but that’s exactly my point. It’s not easy running an instance, and mistreating people who do this as a free service is completely unacceptable.
Developers
Lemmy development has been lead by a few key maintainers, with a massive amount of smaller contributors. The software is constantly being improved at a very good pace, and everybody is able to benefit from this effort at no cost whatsoever. I am extremely grateful to everybody who has participated in the development of the Lemmy software, and other related software, as without you folks, none of us would even be here now.
There seems to be a huge amount of people with very little appreciation of the work that has gone into the software. I’m sure many of you have seen countless messages where people express that the devs should be doing more in one way or another. “They should work faster”, “they should prioritize this obviously most important feature”, “they should be available 24/7 to offer support”, etc. I just want to take a moment here and acknowledge what core maintainers have already done for Lemmy:
- Years worth of work on the code itself
- Offering support to the community and other admins
- Reviewing literally thousands of pull requests on GitHub
- Acting fast in stressful situations where the Lemmy network has been overloaded
- Not abandoning the project in the face of constant hateful users
- Sacrificing literally hundreds of thousands of euros in missed salaries which they could have been getting if they were working for a tech company instead of working on Lemmy
I also want to take this moment to discredit some rumors which I have seen repeated too many times:
- Rumor: Lemmy devs do not accept outside code contributions
This is completely false - the maintainers are completely open to (and even constantly asking for) contributions. When somebody starts contributing, they will receive support and code reviews very quickly. I can tell you that I have experienced this myself several times, but that’s anecdotal, so let me also provide evidence:
a. Contributors list for the Lemmy backend: https://github.com/LemmyNet/lemmy/graphs/contributors
b. Contributors list for Lemmy UI: https://github.com/LemmyNet/lemmy-ui/graphs/contributors
Both of these lists include 100 different names, and that’s only because GitHub literally caps these pages to 100 users. Actually, the amount of different contributors is even bigger. If Lemmy devs did not accept and encourage outside contributions, then there would be no way for these lists to be so big.
- Rumor: Lemmy devs work too slowly
This is an extremely entitled and frankly stupid claim. I try to keep on top of the changes made in the Lemmy repo, and let me tell you, the pace of improvement is very good.
I very firmly believe that if the network started downgrading to Lemmy versions from ~8 months ago, the whole network would just collapse, as none of the instances could keep up with the current volume. That is to say, we have come an extremely long way since last summer alone.
Let me provide some more evidence. Take a look at the Pulse page for the Lemmy backend on GitHub: https://github.com/LemmyNet/lemmy/pulse. As of writing this, Lemmy devs have merged 18 pull requests in the week leading up to this post - that’s an average of 2.5 merged PRs per day. This is extremely good for a project with a small underfunded team.
- Rumor: Lemmy devs do not prioritize the important issues
There are two sides to this. First of all, there are endless users who turn to the Lemmy devs with what they believe is the most important issue and should immediately be prioritized - the problem is that almost none of these endless users have the same view of what the most important issue actually is! In that sense, it’s literally impossible to please everybody, because everybody wants different things.
On the other hand, even when Lemmy devs do prioritize things which some users have been desperately asking for, I have on several occasions seen a dismissive response along the lines of “too little too late”. Basically, the demands made are often unrealistic and impossible to meet.
If you are somebody who feels like Lemmy devs are not doing enough, I would ask you to please take a step back, look at the actual contributions which they have made, and consider how you yourself would feel if after making such a massive contribution, you would still need to listen to countless strangers on the internet tell you how you’re not good enough in their opinion.
Conclusion
Lastly, I am very thankful to anybody who took the time to read to the end of this post. Again, my goal is to try and defuse some of the hostility, as well as to put out a message of gratitude and positivity. I am very interested in the success of Lemmy as a whole, and that is much easier to achieve and maintain if we all work together. Thank you, I hope you're doing well, and have a nice weekend!
Thank you so much for writing this, it articulated a lot of things I've been thinking about this past week. So many of us are all spending our most valuable resource, time, trying make this a better place, in whatever way we can, and none of these 4 groups (some of us are members of more than one) deserve any vitriol or negativity for their efforts. We all need to think about ways we can reduce that negativity, and not all of them can be fixed with computer code, or at the admin level. We all need to improve how we interact with people, and treat them the way we'd like to be treated, with a view towards their well-being.
I'd like to follow your good example, and send out my genuine thanks to all the users, admins, developers, those doing server support, and contributors in any form to lemmy, and it's ecosystem of apps and tools. Members of all of these groups are absolutely vital, and this place is only possible because of our cooperation.
Thank you for all the work you do!
I heard you guys have recently told Beehaw to quit using Lemmy, is that not correct, or how does this square with what you are saying here?
Heard where? I heard that Beehaw is considering not using Lemmy, but they've said that for a long time without changing still. But maybe they will. But it's all up to them and they can control their instance how they like :)
Beehaw has just published a thing explaining their current situation and it includes some conversations with Lemmy devs where it looks like they are being told to fuck off and stop being a part of Lemmy: https://docs.beehaw.org/docs/important-questions-decisions-and-reflections/beehaw-lemmy-and-a-vision-of-the-fediverse/
Both side seems to be frustrated at each other there. Beehaw admins accused lemmy devs of not listening to their requests for better moderation tools, while lemmy devs accused beehaw admins of demanding more work without contributing anything or gave them appreciation. I hope the conflict is just temporary and they can reconcile later when things have cooled off.
The difference between open source vs paid-for software, and the lack of articulation of what entitlement is (and the harm it causes contributors to open source projects), is one of the root reasons for a lot of frustrations this past week. We've even added a specific no entitlement clause to our code of conduct a few days ago to try to avoid this in the future.
In short, entitlement is insulting or demanding behavior towards anyone for not doing what you want them to do, or not doing it fast enough.
Lemmy is developed by 2-4 devs, but used by >40k ppl. This massive disparity means it is absolutely impossible for us to solve every issue, and please everyone.
We make no demands on anyone, and don't force anyone to use lemmy, and encourage ppl to do the open source thing, and improve / work on issues we don't have time for. We gladly review PRs, as anyone can transparently see on the github.
Some of the beehaw admins on the other hand, are making demands, whilst refusing to do the open source thing and help add the features they'd like added. At this stage we've come to an impasse, where they'll likely just move to another platform, where the developers of that new platform will experience the exact same entitlement timeline: request for features, frustration that they're not getting completed fast enough, lashing out at developers, a similar developer response, then burnout for all parties.
The only way forward is for people to realize that entitlement has no place in open source, and that making demands on other people is not acceptable for any party.
From my discussion with some beehaw admins and sublinks devs, a problem is they they feel like their code contributions will not even be accepted for lack of shared goals. For example showing voting totals.
I keep saying that the most elegant solution here is to develop a plugin framework, to allow admins to customize their instance experience even when it diverges from the lemmy dev vision. If it's made software agnostic it would allow people like me scratch our own itches. Ironic I know but it's for this reason I think this should be one of the main priorities, as it will unleash a lot more Foss power on the project. It's a big reason why projects like WordPress and godot are having great success
They've never opened a single PR, whilst the github shows us merging tons of PRs from third parties, so that seems like negative speculation on their part.
The lemmy API already has open vote totals on everything (score, upvote, downvote), and I also made a PR adding a user preference setting for how to display scores for your user.
I believe there's an open issue for a plugin framework, but that would need to be fully worked out. If it's just simple preferences, there are tons of sample PRs to learn from.
I'm quite confused about some people's adverseness to learning Rust; it's been the voted the most favorite developer language for many years in a row now (for good reason), rust frameworks frequently top the fastest web server benchmarks now, and every real developer has to learn new languages and frameworks every few months to keep up to speed anyway. Just as an example, I was waiting for a messageease(an android keyboard) replacement, and nothing came close. I taught myself kotlin, and android programming, and made one, and I'm an incredibly slow learner and middling programmer.
Speaking or myself, it's simply a matter of time. Even doing the fediverse stuff I'm already doing is stretching me beyond my limits.
About the prs, ye I don't want to do the broken telephone here. The point is there's clearly a sort of disconnect between how welcoming the lemmy developers believe they are and how they are perceived by some other people. It's not an easy problem to solve tbh and requires some honest discussion in good faith with the affected parties.
About the open voting totals, I remember there was a conflict about this. Weren't they are some point hidden on the api? As in if you request the overall voting total for a user you alway get 0?
It's exactly the same for us. There are hundreds of open issues for Lemmy and we can't work on all of them.
By voting totals you mean the karma score? We intentionally decided not to show that because it has many negative effects. It was accidentally still exposed in the api so we removed that.
You misunderstand. For you, lemmy is your full-time job, so working on feature A instead of feature B, is still working on lemmy. For me being a lemmy admin is a side-thing to my main FOSS project, and on top of that I also have my day job.
yes exactly. But this is something that you diverge from what other admins might want. So they cannot submit it as a feature even if they want to which is what I was alluding to earlier as to why you might not see the PRs
Sure, thats because we develop for the majority of the userbase and not what some (real or imaginary) admins might want. Its impossible to make everyone happy so we have to choose what works for most people, and hiding karma is clearly very popular.
They're free to not run the upstream version but their own patched one. Not being willing to write code will certainly not make that feature landing any more likely.
Plugins in Rust are a whole can of worms, what with AOT compiling and static linking, you can't just monkey patch everything like in PHP. Essentially a plugin system would be a way to organise patches, to dependency inject admin-provided libraries instead of default implementations. Having a collection of patches should help designing such a thing, Personally I wouldn't even consider designing such a thing before it's clear what it will need to do, where it needs to provide hooks, also, that it will be used.
There's much better ways to do plug-ins than this. Just check how the Django-wiki does it or how godot does it.
If rust can't achieve what python or c++ does, then that would unexpectedly disappointing
Python is a dynamic language just as PHP, you can monkey patch there. Neither are anywhere close to the performance level of Rust or C++. I wouldn't call their approach better: It's very brittle, nothing tends to be well-specced, things easily become bug-compatible and a nightmare to maintain.
Godot has fixed extension points, which on top of that are accessible via gdscript or C#, not C++. Things you can hook into. Figuring out for which things that should be possible for, and how the interfaces should look like, is not something where eyeballing works.
Doing it the godot way would be rather pointless with lemmy as there's no need to extend the server's functionality while it's running, you can just replace the whole binary which is way simpler to do but the sensible way to go about it dynamically would be to use wasm. In both cases you'd need to figure out exactly where plugins should be able to hook into, there's no way around that without eating PHP's and Python's performance and maintainability costs. Also, rewrite everything.
There's no monkey patching needed! There's way better approaches with signals and events and shit!
Even if there was a performance hit. That's something for the plugin users to eat. The point is they have the option. Beehaw.org might accept the extra 1ms delay to have the tools they need.
And yes, the effort needed to support plug-ins is worth it. This is why all successful community driven software does it!
Why introduce a dynamic scripting layer if you can dependency-inject a crate, compile your custom version, and replace whatever is running just as if you'd be restarting the server, picking up where it left because all the important stuff is ACID? (Or at least I hope that's how lemmy works, never actually had a look at the code).
Certainly less development effort, gets as cheap as putting the function to be customised in a lazy static. The difficult part is the API that goes along with it, providing a stable interface to plugin writers because if you don't have that you can just as well tell people to patch the function directly. And at the same time lemmy's version number starts with 0 which doesn't exactly sound compatible with API stability, and bugs need fixing. As such it comes down to the same thing: If people want a plugin system, they'll have to contribute as dessalines and nutomic seem to be perfectly busy keeping the ship afloat.
because maintaining, compiling and deploying custom forks is an order more complex than loading some third party plugins?
And because with plugins each admin can mix and match what they need
Compiling a custom binary can be streamlined (Rust certainly already has a quite painless in-bult mechanism), and you can mix and match dependency injection that's not an issue. It may not be a fancy wheel but it's a sturdy wheel and a wheel that doesn't need to be invented yet.
I think you severely overestimate the capabilities of the average admin
If you can follow a tutorial and docs to edit a config file and start the server then you can follow a tutorial and docs to edit a config file and start a rebuild script.
My brother's in christ, you need to deploy that shit somehow...
Also lol at the idea that every tutorial is the same difficulty
Deploy? I'm a programmer, not devops. I'm barred from production.
Erm. Anyway. If you do it right you can have one nix flake that people can edit and the thing itself then decides whether it can use a standard binary or needs to do a custom build, no difference to the admin. The code needs to be compiled one way or the other, with Rust it just happens to be simpler, and just as automatable, to do it ahead of time instead of at runtime.
"if you do this right" sounds like wishful thinking. Sorry but every other software uses plugins for his sort of thing for a reason. As a devops, sysadmin, and developer, what you're describing sounds absolutely unusable for the vast majority of its target audience.
It's still plugins. It's still configurable without need to write code, or know Rust. Just the compilation step occurs ahead of time instead of at runtime.
And plenty of software is written that way btw though more often in the commercial and embedded sector. This goes into the general direction of a Software product line. As you mentioned godot: Game engines also tend to do it, simply for performance reasons. You can implement quest logic in a dynamic layer, but if you need custom physics or such it needs to get baked into the binary. Godot calls those things modules instead of plugins but really it's the same thing at a different point in compilation time.
You realize Godot supports modules and plugin for a reason, right? Like it would be absolutely inane if Godot asked gamedevs to recompile Godot from source to add shared functionality. And this is with a piece of software that is already aimed at developers and software houses, most of whom are familiar with compiling from source.
Lemmy admins are not software houses. Most barely have enough knowledge to run the ansible playbooks and you're suggesting they set up CI/CD pipelines!
They do, for a lot of things: Not everything you can do with modules can be done with plugins, while everything that can be done with plugins can be done with modules. And bevy requires it for everything: If you want a scripting layer you have to bring your own, the project doesn't, and won't, provide one. The envisioned editor indeed involves building everything from scratch, not just loading things into a pre-compiled runtime.
Nah I suggest they run them. Either as-is or after tweaking some knobs. And frankly speaking running a server is not like opening MS Paint, if you don't have some background technical knowledge if the software doesn't eat you then the next botnet will.
And yet, they support plugins, even through they're technically unnecessary since modules exist. What a waste of core developer time, eh?
I mean, there's a reason why Godot is slowly-but-surely taking over the world of Game Engines and I have never heard of bevy before now. Ease of use and ease of development trumps performance every time (as far as "market share" is concerned).
Sorry but as someone who does have the extensive experience with those things, there's nothing accurate about that statement.
Which is where allowing more developer resources to be injected into the project, with, oh I don't know, plugins, would help. We're looking to expand the pool of people who are technically able to run fediverse servers. Not shrink it.
Godot set out to be an open source unity, bevy set out to finally build an ECS-centric game engine with performance as its top goal. Bevy is also quite a bit younger, and not yet feature complete, but also far from a small hobby project it's serious business.
Why would an admin care if setting a particular setting in a config file causes compilation of a custom binary, or the standard binary, once started, compiling/loading a dynamic plugin? There's no discernible end-user difference. In both cases you do the equivalent of
./lemmyctl [start|reload] --config my-config.jsonor such.My peep, do you even know how lemmy hosting works or are you speaking out of your ass? :D
I might be a tad out of touch, which can probably be inferred by the
ctlpart, but it's the concept that matters, and this applies to any daemon: You configure, you start, you possibly change the config, and you tell the thing to reload the config.On my system everything gets switched all at once with
nixos-rebuild switchorhome-manager switch: File systems get mounted and unmounted, daemons stopped and started, the whole system gets upgraded if necessary. It can be that easy. If it isn't, maybe that's a thing to look into before looking into plugins.A plugin framework sounds great. I would be happy to see a PR for this from Beehaw, Sublinks or anyone else.
Sounds like a case where they're both right, too.
One thing I have never understood and keep repeating in this context: Beehaw has >7k$ balance. If they really have a few issues that would solve 90% of the problems, why not putting a 500/1000/2000$ bounty of that feature.
I believe they mentioned that in their essay I linked to in this thread yesterday, they looked into starting a bounty system for new features but the Lemmy devs told them not to do it so the idea was abandoned.
I am curious about the details of that conversation, because I remember reading Dev's comments in some post on Lemmy where they mentioned this option.
What? Even if they wanted to, the devs can't control who uses Lemmy. It's FOSS
I don't know about the devs specifically, but maybe advised is a better term. Beehaw themselves don't really want ro be using Lemmy as it doesn't have the moderation tools they want.
I'm not sure beehaw idea of what they want works in the federated space