this post was submitted on 12 Jun 2023
617 points (99.8% liked)
Technology
1928 readers
7 users here now
Rumors, happenings, and innovations in the technology sphere. If it's technological news, it probably belongs here.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Too much load? Reddit is down.
Not enough load? Believe it or not, also down.
I'd love to know what it is about subreddits going private that caused issues.
Maybe some overload caused by a process having to dig deeper to find best/top posts?
apparently that's exactly the case.
That is an interesting aspect no engineer could have foreseen!
You'd be surprised how much critical infrastructure was implemented through trial and error and has just been left like that for years...
Anything less than 99% of infrastructure working that way would be surprising. Everything is held together with scotch tape and scotch whisky.
I'll be sure to repeat that last line to my fellow team members :D
I like this idea. I imagine that with the top subs being dark the automated top posts that get scrounged up may be too terrifying for the front page and they hit the panic button while they scramble to curate through the absolute worst filth they've ever seen.
“It’s merely coincidence. But starting Wednesday, our servers will be more robust and you can browse the site using our official app.” - Spez, while sniffing a decanter of human shit
God we need indefinite blackouts.
It’s entirely possible that they’ve made some assumptions about what a “normal” level of traffic looks like when writing code for their backend, which has caused some things to break when that has changed.
Not our fault if their code is shit.
How is that an example of bad code?
Honestly, it’s probably not - if I’m actually right this is likely an issue that Reddit’s engineers never predicted would happen so never planned for it. I was being hyperbolic.
It's not reactive. A proper reactive system can handle fluctuations in usage patterns more robustly.
I'm having a hard time believing the claim that Reddit's code isn't reactive.
Wouldn't be surprised if it's just a gigantic mess of nested if-else statements.
Gotos all the way down
Maybe, but this was a huge increase in usage. Reddit never expected to deal with anywhere near thousands of subs going private simultaneously.
The servers run on the tears of bitter whiny CEOs.
Reddit is hosted on AWS after all...
Tildes' dev Deimos used to work for Reddit and had this guess https://tildes.net/~tech/163e/reddit_appears_to_be_down_during_blackout_day_1#comment-87v1.
They're lying. Fish swim, birds fly, sun shines, Reddit lies.
Probably a drop in usage flagged some internal test
Want Free API? Straight to down status.
Want cheaper API? Also straight to down status
Not enough people on Reddit because of protests? Also straight to down status
This comment is so good an upvote won’t do justice (without awards, a classic comment such as this now has some merit.. it’s a new day boys & girls, a good day)
If Beehaw offered awards I would actually buy them, at least the money would be going towards keeping the lights on for a project that isn't actively trying to screw over users for profit.
Give them some gold. Oh wait…
Rebelling moderators, we have a special jail for rebelling moderators.
thank you, this comment made my day
Lol, this made me chuckle out loud. Good job Sausage man!
When Reddit forcibly opens everything back up:
“Who’s there?”
“Hired mods?”
"Wait, you all are getting paid?"
Reddit has an annual "moderator summit", a rah! rah! yay for moderators! event for moderators, mostly of large or super large subreddits.
At last year's summit, Spez gave his 'keynote' talk where among other things he claimed that they were researching ways to pay moderators for their work, by giving them a cut of ... something. It was all sort of wonky and nebulous and likely just something he thought of that morning in the shower.
Is that what the subreddit coins or subredidt points idea was about?
I don't think so. I think that's a whole other mess.
If the volunteer mods hold their ground and force Reddit corporate to oust them, Reddit would need to step in to fill the void.
They'll find some people.
The reality is, not having (good enough) mods will take a while to really hurt the bottom line. Subs will slowly deteriorate.
But I'm 100% sure, within a few weeks you can establish a new order of more servile mods.
People on Reddit complain about the mods enough as it is. (And I include myself in that. I've had some less than stellar mod encounters in the past.) However, if Reddit were to force out existing mods and replace them with mods willing to toe the company line (and possibly ban people for mentioning the blackout, complaining about Reddit, or mentioning alternatives), it would just result in more user dissatisfaction.
Reddit won't go out overnight. There are too many people who post there. However, this could turn into a snowball effect. Rebelling mods are replaced by bootlickers. Dissent is crushed in order to make it seem like everything is hunky dory before the IPO. Power users flee to alternatives like Lemmy. Slowly, normal users hear that some of their favorite content is on this new service and sign up. Reddit usage drops little by little until it's limping around as a shell of its former self.
Whatever causes the website to have trouble, I'm all for it, right now.
I already wondered if I got lightning-banned for sending too many API requests in a short time, when I used a script to auto-edit all my comments and text-posts.
Verge: Reddit crashed because of the growing subreddit blackout
My hypothesis is that it's probably because so much of Reddit posting is automated by their own bot network now that they DDOS'd themselves trying to auto-post to subs that are suddenly locked. Like they didn't even bother tracking which subs would be blacking out and like...write exceptions to their post schedule.
A significant number. Fantastic. I'm not sure I believe the stability issues, I'm just a a tin foil hat kind of guy though. I guess it's possible.
Reddit didn't design their systems around needing to deal with a huge number of subs going private all at the same time. It's not surprising that it caused a short outage.
Ah, "expected", such a wonderful word! They expected for their infrastructure to explode, just according to keikaku...
Bold of you to assume they had a keikaku to begin with