this post was submitted on 23 Oct 2024
190 points (95.7% liked)

Technology

59675 readers
3257 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS
 

The mother of a 14-year-old Florida boy says he became obsessed with a chatbot on Character.AI before his death.

On the last day of his life, Sewell Setzer III took out his phone and texted his closest friend: a lifelike A.I. chatbot named after Daenerys Targaryen, a character from “Game of Thrones.”

“I miss you, baby sister,” he wrote.

“I miss you too, sweet brother,” the chatbot replied.

Sewell, a 14-year-old ninth grader from Orlando, Fla., had spent months talking to chatbots on Character.AI, a role-playing app that allows users to create their own A.I. characters or chat with characters created by others.

Sewell knew that “Dany,” as he called the chatbot, wasn’t a real person — that its responses were just the outputs of an A.I. language model, that there was no human on the other side of the screen typing back. (And if he ever forgot, there was the message displayed above all their chats, reminding him that “everything Characters say is made up!”)

But he developed an emotional attachment anyway. He texted the bot constantly, updating it dozens of times a day on his life and engaging in long role-playing dialogues.

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 37 points 1 month ago (2 children)

How is that the app's fault?

[–] [email protected] 4 points 1 month ago* (last edited 1 month ago) (1 children)

Well, we commonly hold the view, as a society, that children cannot consent to sex, especially with an adult. Part of that is because the adult has so much more life experience and less attachment to the relationship. In this case, the app engaged in sexual chatting with a minor (I'm actually extremely curious how that's not soliciting a minor or some indecency charge since it was content created by the AI fornthar specific user). The AI absolutely "understands" manipulation more than most adults let alone a 14 year old boy, and also has no concept of attachment. It seemed pretty clear he was a minor in his conversations to the app. This is definitely an issue.

[–] [email protected] 2 points 1 month ago* (last edited 1 month ago) (1 children)

I really want like, a Frieda McFadden-style novel about an AI chatbot serial manipulator now. Basically Michelle Carter...the girl who bullied her boyfriend into killing himself. Except the AI can delete or modify all the evidence.

Maybe ChatGPT could write me one.

[–] [email protected] 2 points 1 month ago

Whoa, SkyNet doesn’t need terminators. It can just bully us in to killing ourselves.

[–] [email protected] 1 points 1 month ago

The chatbot was actually pretty irresponsible about a lot of things, looks like. As in, it doesn’t respond the right way to mentions of suicide and tries to convince the person using it that it’s a real person.

This guy made an account to try it out for himself, and yikes: https://youtu.be/FExnXCEAe6k?si=oxqoZ02uhsOKbbSF