Hi there,
Some providers are rejecting mail from the IP. In the meanwhile send me a PM with your account name and email and I'll get you sorted.
Thank you
Hi all,
I somewhat explain some of the recent downtime here https://sh.itjust.works/comment/1341394
Postmortems are really great when they are done well and you have a team of sysadmins to collaborate with. As you probably know most instance operators and moderators are volunteering their time (and sometimes money) to keep these instances online. As the lemmyverse matures in age and development things will get better. In the meanwhile there will be some hiccups along the way.
When I have some extra time I plan on sharing more details about all of this. In the meanwhile check in on our chat. I'm pretty active there and can always provide additional insights on whats going on during an outage.
This is an issue with Lemmy. You cannot delete private messages, it needs to be done from the database. Luckily this instance and lemmy.world have implemented some measures to have these links removed automagically. If other instance admins are reading this and would like to know more about the measures we put into place feel free to reach out to me via pm or over on our matrix channel.
It helps monitor reports at the moment.
Hey all,
There has been a few issues over the last couple of weeks. Here's a rundown of some of the issues we've had.
-
Broken Images: A few weeks ago we had issues with broken images. This was due to me migrating our local image storage to object storage (like AWS S3). When I did the switch it broke all old images as they needed to be migrated. Pictrs at that time did not support concurrent uploads which means the migration would have taken days or weeks to complete during which time the image service would be offline. Instead I waiting for a newer version to be released that supported concurrent connections and did the migration in about 30-40 minutes one evening.
-
5xx server errors: Some of you may have experience a lemmy page with an error code on it. This was due to me trying to implement an additional proxy to shield and mitigate future risks. While rolling this out I hit a few blockers that caused downtime as we worked to rectify it. I'm glad to say that as of Today this has since been implemented.
In the addition to the above, the lemmyverse and especially this instance has been under bot attack almost daily. These bot attacks are eating resources and causing a query floods.
Lemmy is still very young and in its early phases. In time these issues will slowly go away.
P.S you don't have to worry about me leaving you guys hanging.
I disabled this function until I finish the development of some mod tools. If you want to create a new community send one of the admins a PM and we'll get it created for you. I plan on having community creation turned back on again shortly.
Hey all,
Another update on this. I just completed migrating the entire file store over to object storage.
Images should now all be fixed.
Honestly these type of scat spamming has been really discouraging on me. I spend time and money so that the community can have a reliable and nice instance to call home, only to have a few (not sure what to call them) people try to ruin it for everyone. If these people are reading this comment, just know that every time you make these kinds of unwanted posts you are taking my time away from my family and young kids to address the reports.
My nights lately have been mostly used for coding some extra moderation tools that will moderate these in the future.
I sincerely apologize for all of you who have been exposed to literal shit images over the last week. I’m working as fast as I can and will implement something soon.
In the meanwhile i’ve added a few new admins to help with this moderation.
I’m actually quite amazed by how little would be needed to keep this instance fully funded and even provide other services at the same time. If just 1/4 of the users gave 25cents per month we would have enough money to cover the hosting, new hardware and some paid development/administrative work! The problem however still resides with not having a fiscal host to be the entity needed to accept donations.
Working on some alternatives at the moment.
Offloaded the images to an S3 compatible host. Will take a bit but I will be migrating the older images shortly
Hi there,
I've upped it to 50 characters.
Hey all, I’m always excited to upgrade to the latest versions that become available. Unless the upgrade is related to security I will tend to wait a few days to give other instances a chance to share their experiences. Before upgrading I usually go over the issues that have been submitted to help understand the risks and solutions to mitigate them.
That being said the new version does bring in a whole bunch of performance gains (mostly related to the database) that i’m looking forward to.
Lemmy.world is due to upgrade later today and I plan on upgrading shortly after them (either Monday or Tuesday).
I will put up a notification up prior to the upgrade.
Thanks