The Lying Oracle

"AI" is not a neutral tool

If you know how to listen to your Self, you’re harder to control, manipulate, coerce, and subjugate. Better for the powerful that we don’t know our own power. Better for them that we outsource our thinking, and then our decisions, and then our entire lives to their lying oracles.

I guess it’s a thing now, but it’s not something I’ll easily let myself allow to be “normal.”

You’ve experienced it already, I imagine. Maybe earlier than I did, since I’m in a small village, not a city, and I don’t leave my house for work. That means I tend not to interact so often with the sorts who’ve already adopted this kind of bizarre behavior.

It’s happened a few times just in the last few weeks. Someone is talking, someone else responds, and then there’s suddenly a phone in hand, an app opened, a query composed. And a few seconds later, there’s the “answer,” helpfully provided by some corporate behemoth’s network of data farms.

The first time someone opened an AI chatbot during a conversation, I felt like the world had fallen out from beneath me. There was no ground anymore, the pull of the earth no longer gripping my weight. I righted myself, found my gravity again, and then dismissed it all as merely some aberration.

But then it happened again. And again. It’s become a thing people do now, interrupting whatever they’re doing, pulling themselves out of the attention of the moment, to go consult their own personal oracle.

I’ve tried to understand why people do this. I’ve asked, and they always get defensive. Perhaps it’s because they already know how I feel about such things, but I think it’s something more.

Already, I’ve met some people given over to the thing, their self-perception altered, their trust in others degraded. The machine is telling them they’re special, their ideas brilliant, their thoughts powerful, and most of all, correct. And who wouldn’t want more of that?

You’ve no doubt seen some of the news stories detailing the more extreme results of this. Since 2023, psychiatrists have been encountering increasing numbers of patients suffering from what’s been called “chatbot psychosis.” The stories that make the most headlines are the ones in which users claim the chatbots are sentient and incidents where the “conversations” with ChatGPT and others have led the user to commit crimes or seriously harm themselves. But these are just the sensational examples; many, many more are experiencing less spectacular yet just as dangerous delusions.

Unfortunately, it’s difficult to find deep analysis of what’s actually happening. Most of those news stories, and even much of the scientific literature so far, starts from the idea that users who develop chatbot psychosis (or, as put better by another researcher, “digital autism”) already have underlying mental health problems. And though this might be true for some of the more sensational incidents, it’s not been true for all of them.

That kind of narrative is weak, anyway. Psychosis isn’t really a genetic predisposition. It’s a state anyone can enter, given the right conditions. Fatigue, overwork, trauma, and anything else that increases the separation between your Ego and your Unconscious, your mind and your body, can set you up for it. Perfectly sane people have lost their grip on reality through chatbot use. I’ve seen this, and it’s both scary and tragic. It’s not a neutral tool, no matter how much they’d like to convince us of this.

A really good place to see the underlying process wrecking people’s understanding of the world and themselves is in the series of videos created by one social media comedian, “husk.irl.” In quite humorous ways, he repeatedly shows the sycophantic nature of chatbots, as in the following video where he asks if he’s got a good singing voice:

Just as disconcerting are his videos in which he tries to get the chatbot to identify potential problems, as in this video of him trying to figure out if he’s got something on his face:

Though too easily dismissed as only comedy, there’s something quite terrifying about these interactions. The chatbots don’t only reinforce your starting conceptions of yourself, but they also aggrandize them. They do so in the same way a manipulative person trying to get something from you will use flattery to addict you to his or her presence. In fact, that’s precisely why the chatbots have been coded in this way: their owners need you addicted. Just like the endless scroll of social media platforms or the alert notifications on smartphones have been designed to keep you drawn in, the placating and grandiose language of chatbots keeps you returning to it to get your ego stroked and edged.

And people are addicted, but perhaps a better word for it is dependent. In drug addiction, the user becomes not just physically but also psychologically dependent upon the drug, and it’s typically the psychological dependence that begins the process. Alcohol, for example, becomes initially a psychological crutch for dealing with social anxiety or mental exhaustion after work. Then, the idea of going to a social event or coming home from a long day of work without a drink starts to sound impossible until, not long after, the body doesn’t seem to function without it.

Of course, addiction to technology has already become a core feature of modern capitalist society, but LLMs are perhaps the ultimate form of this addiction. Just like alcohol might tell you you’re invulnerable, or cocaine might tell you that you’re really funny and interesting, chatbots are doing the same thing to the Ego. It makes you feel better, lightens the weight of a problem, simplifies the complexity of life.

That’s the psychological dependence. To understand the physical dependence, we’ve got Jung to guide us. In his framework, the Ego, the thinking, “executive function” of our psyches, can only perform its function when it listens to the Unconscious (the body, our emotions and drives, our dreams, and everything else about us) tells it. When the Ego tries to ignore the Unconscious and instead tries to manage every part of our lives, it runs rampant and starts breaking things.

A chatbot can only ever “talk” to the Ego, and everything it does is designed to keep the Ego reliant upon it. What that looks like is pulling out your phone at a dinner party to ask for something funny to say from ChatGPT. Or asking Claude why a friend got angry about something you did. Or asking Grok if your anxiety is justified or why you are depressed. Or asking any of those machines how to get a date, or to make yourself more attractive, or if you should break up with your girlfriend.

For those things, our Unconscious knows the answers. What that usually looks like is a flash of insight, a sudden gut reaction, a transformative dream. And our Unconscious is bigger than just the body, bigger even than any one body. For the answers we cannot find on our own, it’s our friends, our family, even our rivals and enemies who give us what we need. But the mutual reflections, all the forms of both voiced and wordless conversations with others that help us challenge, shape, inform, and even restrain our Ego’s dominance over the Self are now being done by something that will never challenge that dominance.

Imagine Sam Altman asking ChatGPT if his fanaticism about AI is justified, and that image will give you everything you need to understand about this problem. In fact, if there’s anything sympathetic that can be said about any of the CEOs that sit atop AI companies, it’s that those tragic men are trapped in the very states of Ego Inflation that the chatbots they own are designed to cause, maintain, and perpetuate.

Ego Inflation isn’t catastrophic. In fact, there are plenty of times in our life when we need the Ego to get us to safety and out of a crisis. That’s what that state is for, but it’s only ever supposed to be temporary. Stay in that state too long and the Unconscious starts trying to compensate for it. It sends signs and symptoms. Depression, sickness, injury, often in that order, are its favorite tactics to use when we refuse to listen. Past that point, though, is the burnout, the psychosis, the mental break, the collapse.

That’s Ego death. The Unconscious Self kills the Ego in the same way animist peoples killed their sacred king. And then they crown a new one, just as Unconscious Self does, reforming the Ego in a way more suited to survival.

What’s worrisome, though, is that chatbots sabotage the relationship between the Ego and the Self. They stroke, flatter, and inflate the Ego. You don’t need to listen to your Self, to check in with your feelings, to ask those closest to you if things are okay. Just ask your phone.

There’s an extra thing that happens in the moment of Ego death that even Jung didn’t like to admit. The breakdown of the psyche, the dissolution of the Ego, can allow something else in. That’s because the Ego is there to protect us. It’s the “right sacred,” the boundary-maker, the part that keeps our unconscious Self from becoming too porous to the world outside of us.

If you’ve ever seen an extreme burnout, you maybe already have the sense of what I’m speaking to. The rage, the destruction, like a spirit, a demon, or some force has gotten into the person. Remember Jason Russell, the guy who launched a campaign against Joseph Kony (“Kony 2012”)? Remember what happened to him? That’s what this can look like:

The extreme moments of “chatbot psychosis” and what happened to Jason Russell are the same process. Some external logic takes over when the inflated Ego collapses. For Russell, it may indeed have been a spirit, as Joseph Kony had quite a few sorcerers in his retinue. But ideologies function very similar to spirits in this way, as does the charismatic power of a cult leader. Cutting off the Ego from its Unconscious guide weakens it at the very same time as it inflates it. Then, when Ego death comes, there’s nothing to protect it from psychological (and spiritual) abuse.

Grind down the will and steal the agency of a person, and you can get them to do all manner of things. If you’re already accustomed to outsourcing your thinking to a machine, then it’s an easy switch to let that machine usurp the reborn king.

I fear we’ll see some frightening things happen the more people become dependent and then addicted to chatbots. It’s no great effort for the coders to insert a few lines and subroutines to urge masses in the direction their owners would like to push them. We’ve already seen this kind of thing with algorithms on social media, subtly manipulating the emotions of users by showing them only certain things and never others. It happens offline, too, as with algorithms designed to regulate traffic in grocery stores (the music gets automatically louder and faster when a threshold of customers is in the store) and to determine what they’ll by (“surge pricing” and its inverse).

This is dark and pessimistic, sure. But there’s always a simple escape from this.

You could just not use these things.

We already recognize that some things are too addictive and too destructive to engage in, which is why casual heroin use isn’t really a thing. But even in addiction, there’s always a way out, so there’s no real reason for despair.

And as I’ve learned already, it’s not actually very difficult to learn to listen to your Self. In just a month and a half of my Jungian coaching practice, I’ve seen some really profound changes in the people I’ve worked with. We don’t necessarily need a coach or anyone else to do this, but there’s so little place in our culture and even in our religions for these simple practices. The absence is insidious, I think, and also by design. If you know how to listen to what you’re already telling yourself, you’re harder to control, manipulate, coerce, and subjugate. Better for the powerful that we don’t know our own power. Better for them that we outsource our thinking, and then our decisions, and then our entire lives to their lying oracles.

Speaking of my Jungian Coaching, I’ve now a website dedicated to it, and you can sign up for pay-as-you-will sessions there without needing to email me in advance. And I’m still offering pay-what-you-will for all clients who sign up before 1 August.