TechTakes

I was reminded of a particular anecdote I need to get out of my system.

There was a time on a particular fiction series' fan forum's IRC channel when I had to convince a very enthusiastic teenage fan that the instructions they found on 4chan to open a portal to a parallel world of that particular work of fantasy fiction was actually a method for synthesizing and inhaling poison gas.

The weird part was that they admitted they knew it was probably just a cruel prank, but were still willing to try just in case it was real. We had to actually find and link articles where the same reaction was exhibited as an actual documented suicide method, if an unreliable one at that, to convince them not to go for it.

By my estimate that person was about high school age, almost certainly over 13 and probably over 15. Mostly their behavior seemed normal for the age, certainly overenthusiastic about the fandom and with obvious signs of teenage ennui, but both of those are typical for someone in their mid-teens. It was just this strange incident of extreme gullibility and self-destructive devotion to a fandom that really stuck with me.

The way things are going, KnowYourMeme will be the only reputable journalistic outlet in a few years.

Okay, quick prediction time:

• Even if character.ai manages to win the lawsuit, this is probably gonna be the company's death knell. Even if the fallout of this incident doesn't lead to heavy regulation coming down on them, the death of one of their users is gonna cause horrific damage to their (already pretty poor AFAIK) reputation.

• On a larger scale, chatbot apps like Replika and character.ai (if not chatbots in general) are probably gonna go into a serious decline thanks to this - the idea that "our chatbot can potentially kill you" is now firmly planted in the public's mind, and I suspect their userbase is gonna blow their lid from how heavily the major apps are gonna lock their shit down.

Wait, the guy was Simone-ing himself with a waifu chatbot? I heard the site name and outcome and assumed this was a nudification bullying ordeal. That at least would have worrisome questions about making bad things possible versus making bad things easier, and whether general-purpose applications excuse foolproof access to the worst abuses of a given technology. This... this is dumb. This is really fucking dumb.

accusing the company of blame and that the teen was deceived and manipulated by a chatbot

I'm not convinced they could reliably make these things recommend a brand of dish soap. The fuck kind of manipulation is the company supposed to be at fault for?

This is so stupid that I don't even expect the general public to clutch their pearls. How do you describe this in any way that a normal rando's immediate reaction isn't just '... what?' This teen wasn't bullied by an imaginary girlfriend. Unless I've missed something, the bot did not somehow suggest this course of action, unprovoked. (Then again, it's based on Game Of Thrones, so that might come up.)

Side note, fuck commenters like Akhilesh for calling a 'that's fucked up, we're gonna try to prevent that' response as if that means 'the company has accepted this.' What would it look like if it looked like what you wanted? Flat denial? This attitude is a trap. They do nothing, people cry they do nothing. They do something, even something pointless, the same people cry that it's proof of guilt.