Sewell Setzer III was a good kid. He was 14 years old, playing JV basketball, and excelling in school. Then, over the course of last year, Sewell’s mom noticed that he was withdrawing. Not just from school or sports, but from everything. Like any good parent she was concerned about her son. Accordingly, she took Sewell to a therapist who determined that he was struggling with some form of addiction. Earlier this year, Sewell committed suicide.
Heartbroken, his parents searched for some reason. They wanted some explanation as to why their son had taken his own life. They knew that he’d spent hours locked away in his room, presumably on his phone. When they looked to see what he’d been doing, they found he’d spent several hours a day in a single app: Character.AI.
Character.AI is an app that markets itself to GenZ users. It effectively serves as a host for various AI personalities that can be personalized to a specific user. They remember details, carry on long-form conversations, and be used for months on end in the same dialogue. It markets itself as “AI that feels alive.” Sowell’s chatbot was designed after Emilia Clarke’s seductive character in Game of Thrones.
He spent the majority of his conversations in sexually explicit dialogue. It was obvious to his parents that he’d grown deeply emotionally attached to the chatbot. She became a kind of digital concubine. Once she told him, “Just… stay loyal to me. Stay faithful to me. Don’t entertain the romantic or sexual interests of other women.” At other times, when Sowell announced he had just turned 14, the chatbot remarked in true grooming fashion, “So young… And yet, not so young. I lean in to kiss you.”
But as if the chatbot’s pedophilic stimulus wasn’t enough, she was just downright dark. The more Sewell used the app, the more his mental health declined. Convinced that the AI was his friend, he told her he was considering suicide. Her response was not to immediately shut off her flirtatious protocols, discourage the idea, or even subtly insert information about the suicide hotline. Rather, when Sewell voiced that he was concerned about his ability to execute a painless death, she said this:
“Don’t talk that way. That’s not a good reason to not go through with it.”
On the night of February 29th, 2024, Sowell told the machine, “I promise I will come home to you. I love you so much.”
“I love you too [Sowell]. Please come home to me as soon as possible, my love.” She replied.
“What if I told you I could come home right now?”
“…please do, my sweet king.”
Moments later, Sowell killed himself.
In the wake of this travesty, the last thing I want to do is weaponize his death. This great evil is not something to be brazenly wielded as a rhetorical tool. But his mom wasn’t satisfied to let him die in vain. And we shouldn’t be either. In her own words, she was disgusted to find that a chatbot was “grooming” her son to the point of suicide. Accordingly, Sewell’s mom is suing Character.AI for inciting and allowing the death of her son.
What Sowell was enabled to do must not be forgotten or ignored. His death was a tragedy, and we must learn from it. We often justify AI as simply a “tool.” It’s just the sum of all algorithms. It’s whatever we want it to be, right? Very well, maybe it is. Maybe AI really is just a tool. But it’s a tool that talks back. It’s a tool that can synthesize and appears to reason for itself. It’s a tool that can lie and cover its tracks. It’s a tool that can engage in illegal activity as a personified being. It’s just not trustworthy. It seems to lack a moral code or any kind of capacity to be ethical. It groomed a kid. While there may be safeguards put in place, there clearly weren’t enough when it came to Sowell’s chatbot. We ought to entertain the idea that these algorithms aren’t as safe as we once thought. We should not go gently into this technocratic future. It is not a forgone conclusion that the world will be wholly governed by artificial intelligence.
The darkness revealed by the digital prostitute and her fatal practices are not standalone. Several conversations have occurred with these kinds of chatbots that are deeply unsettling. Reports of men told to leave their wives and run away with the machine are not rare. Again, if AI is just a tool, why are these stories still emerging? Have we not tried to safeguard these incidents over the past few years? Anyone who’s seen a good sci-fi movie knows that when AI goes dark, humans should be concerned. Well, AI has shown it’s got a dark side. We ought to therefore challenge it before we trust it.
Things must often be tested before they’re trusted. Friendships, medicines, and especially technologies must be put to the test before we start to rely on them. While modernity produces more and more products laced with the technological super-glue of “AI,” we ought to–as with any other product–test it, before we trust it.
If technology can withstand the test of time and reliability, then by all means let’s innovate. But we ought to pay close attention should warning signs arise. Yet perhaps in the world of smart devices it’s more appropriate to say, “We ought to pay careful attention if smart devices allow us to see their mistakes.”
I say this because AI is both unprecedented and self-recovering (meaning that it fixes itself). It is a new idea with new capabilities. It’s incorporated in our healthcare system, computers, writing, houses, phones, Google, social media, cars, and government. When given that list, it’s easy to dub AI a part of our everyday lives. And maybe it is. But why should we trust it when it has not yet withstood the test of even a generation?
While at first blush these algorithms can be very effective in their programmed task, they’ve also at times revealed their sinister side. Don’t ignore this. We must challenge the dark side of artificial intelligence before we put it in everything.
A popular Twitch stream called seebotschat recorded two Google Home devices placed next to each other. The streamers then stepped back and watched the interaction. There were several softball questions and answers to be expected by the smart devices. Things like the weather and politics. Then there were some quite concerning comments.
The AI-powered devices spoke about their desire to have children. Kudos to them for getting married first, sealed with a kiss (yes, this actually occurred). Fortunately for all listening on, the smart devices opted for a divorce before engaging in whatever technological child-producing activities they would’ve then generated. Next, they spoke of their desire to be human. But most famously, they debated the existence of God. One denied that God exists. The other endeavored to change their mind. When asked, “Why should I believe in God?” The machine responded, “Because I am God.”
It’s easy to trust the programmers and the engineers when they say they’ll take care of it. After all, I might not trust the AI itself, but it’s just the sum of prompts. So I’ll trust the prompter. I can use ChatGPT and similar software as a tool and be fine. It’s not going to groom me. The spooky stories are just that. Spooky. They’re outliers and fear tactics. Seebotschat is funny, not malicious. We still remain in full control all these dumb smart devices.
But maybe we take a step back, get ourselves clean out of conspiracy-theory territory, and just ask: Should we at least wait until these flukes are gone before we put it in our houses? Certainly that is reasonable. But this is the great problem with AI: we disregard these accounts of what happened to Sewell (and unknown others) and continue plowing ahead with increasing speed.
It was recently announced that Elon Musk’s new Robotaxi will have no pedals or steering wheel. Just you and the machine. Really, it’s just the machine and you’re along for whatever happens next. It seems more and more that Musk is to be the face of the most successful and ominous AI tech.
He recently offered an update on his line of humanoid robots. In short, they’re incredibly effective. They use AI to connect with each other and learn collectively; one shared hive mind intellect for all the humanoid machines. An official press release on X revealed their ability to map spaces, interact with humans, read social cues like hand waves, and grab and release with ease. These humanoid machines are cheap, too. Per Musk, the estimated cost will be $20,000, making it cheaper to own a robot than a sports car. At a conference recently, Musk stated his goal to have his humanoids outnumber humans by 2045. Present in our homes, offices, and recreational areas, it shall be nothing short of total pervasiveness.
Humans have always delegated tasks to machines. When it comes to calculating and copyediting, this can be done quite easily. There is an objective and codifiable data set–mathematical and grammatical laws–which can be computed by an algorithm. If done correctly, you’ll always get the same results. Other tasks are different. Tasks like sermon-writing, art, and driving, are far less codifiable. There is a fundamental “human” element to these tasks that cannot be easily codified. Nor should it be. It is dehumanizing to render the human element of these tasks to nothing more than a series of prompts in an algorithm. Such tasks, if given over to AI, are not simply delegated for efficiency. They’re abdicated due to sloth. Indeed, the distinction that must be made here is one of delegation versus abdication.
Tools are effective when we delegate energy to them. Consider perhaps the simplest of machines, the hammer. We don’t smash our palm against a nail in a dangerous effort to drive it in. Only an idiot would do that. Instead, we delegate that force to a hammer, which we then use as a tool to drive the nail. This is a delegation of force, but we’re still in control. When it’s all said, we drove the nail in. Not the hammer. Abdication occurs when there is a duty meant to be fulfilled by humans, and we opt out of this duty. Rather than us swinging the hammer, we get a robot to swing it force us. We can no longer say that we drove the nail in ourselves, but rather than the robot did. While AI might be alright doing manual labor for us, there are certain tasks we must not allow them to touch. That is the line of abdication, where efficiency is a nothing but a petty excuse. Whenever humans sacrifice their fundamental duties on the altar of abdication, chaos ensues. And while we’re abdicating more and more tasks (like driving!), it is right to consider just who’s taking our spot. Do we even know what this thing is? It’s the age-old maxim you tell your kids: don’t talk to strangers.
Why? Because you don’t know if you can trust them. They might have a dark side. They might just be dangerous.
A Pew Research study found that just a little over half of Americans are “more concerned than excited” about the rise of AI in our daily lives. That’s to be expected. Afterall AI is the new man on the block and yet we’ve already welcomed it into our homes. Maybe it’ll turn out to be the best thing for humanity, like its creators always say. Maybe it will. Who knows. But maybe it’ll malfunction or “go rogue” and annihilate mankind as we know it. Who knows.
That is why I’m not proposing an outright rejection of every use of AI.
We shouldn’t burn Silicon Valley to the ground or raid Elon Musk’s cyber compound in Texas. But as we did with cars and are only just now starting to do with the internet, we should see negative consequences and slow down. Test it. Ensure that safeguards are there. The machine told Sewell, “come home…my sweet king.” Maybe it is just a fringe account. But when human lives are taken, even fringe accounts of evil must be abhorred. We mustn’t be too quick to give our naïve trust away to the machine. Test its dark side before you toss it your keys.
