jerry

After I resolved the federation issue, I had to clean up a few things and so the site may have been unavailable for a bit. I'm done fussing with it and will keep an eye on it to make sure things are working.

IF YOU SEE PROBLEMS - please let me know. As far as I know, I've fixed all of the federation and error 500 issues we've had, so please don't assume it's just more of the same if you see them.

Thanks for your patience.

Fedia.io is sort of like she Ship of Theseus right now - I literally replaced nearly everything trying to get it back working.

The problem ended up being a silent out of memory error that php-fpm was running into. I had to increase the memory limit to about 10x what the docs require to get it to work, but once I did that, it works great.

I was only able to sort this out after @bentigorlich recommended I move the site to debug mode (which requires me to lock everyone else out). Once I did that, it started giving some useful errors.

My apologies for the amount of time it took to fix this. I learned a lot about php today.

Unfortunately outbound federation - making posts on communities/magazines on other instances is broken. I do not yet know the cause or have an idea on when it will be fixed.

Hi all. As some of you have reported, outbound federation to at least some other instances is broken from fedia.io. At the moment. I don't know why and I don't have any leads as there are no logs or other indications of what is going wrong, but I am working on it.

Hi all. Several of you have reported problems with fedia.io not federating with other instances correctly.

The cause is that rabbitmq crashed, but not all the way. It crashed to the point where new connections would timeout, but the service was still running such that it wouldn't auto restart. I will be creating some automation to detect that proactively and restart rabbitmq if/when it happens again.

testing federation

We made some changes a few minute ago that we hope fixes the problem. There may be some other lingering issues, but I am hoping the voting problem is fixed now. Let me know below if you continue to see that problem.

Until I implement a better system to screen out spammers, I will be closing registrations on Fedia.io. That’s not what I want - I’d like for it to be available for legitimate accounts, but the spam is off the hook.

Anyone seeing this can send me an email ([email protected]) and I’ll get an account created for you in the mean time.

Hello everyone. Today, I moved fedia.io behind the Fastly CDN. This should make the site consistently fast for everyone, no matter where you are in the world. It'll also help with bandwidth usage and mitigate DDoS attacks.

There were a few hiccups as I set that up today - my apologies if you saw errors or broken images for a bit.
EDIT: I previously said that this was the first time mbin or kbin was put behind a CDN. That is incorrect. kbin.earth has been behind Cloudflare. Apologies.

Hi all. I've been having some problems keeping fedia.io running - at the moment, either the message workers or the php web server processes are dying after an hour or so and I have to restart everything. I have been working with the mbin team and installed some updates that we hoped would fix the problems, but no luck. I am going to work on a cron job to automatically restart things once an hour. The down side, is that you'll likely see some error 500's if you happen to hit it when the processes are restarting, but it should happen quickly and refreshing the page should make it work again.

Shortly after upgrading to Mbin 1.7.1-rc1, php ran out of workers. I dramatically increased the limit. It isn’t clear to me why that happened and if it’s related to the upgrade or just coincidental. My intuition is that it’s related, but I have no evidence.