Are we trusting AI too much?

AI sounds confident. It always sounds confident. It produces fluent, authoritative answers regardless of whether those answers are true. It does not hesitate, qualify, or express doubt. It does not signal the difference between what it knows and what it has inferred.

This is not a flaw. It is by design. The systems are built to generate plausible, well-formed responses. Plausibility and accuracy are not the same thing.

In Episode 3 of The AI Values Podcast, Edosa Odaro and Lindley Gooden explore what happens when we mistake one for the other.

The hospital story

Edosa opens the episode with a story from his own family. A relative was in a holiday apartment abroad. Everything in the fridge was labelled in a foreign language. There was a small child in the group with a nut allergy. They did what millions of people do every day: they asked AI.

The AI responded with the certainty of something that had considered all the angles. It gave a clear, confident answer about which foods were safe. They ended up in hospital. The doctor said it could have been an emergency.

"The AI wasn't malfunctioning. It was doing exactly what it always does - generating a plausible answer, regardless of whether that answer was true. Confidence is not intelligence."

This story is not exceptional. It is ordinary. It is what happens when systems that generate plausible responses are used in situations where accuracy is the only thing that matters.

‍

AI made up a quote about me

Lindley tells a story from his own professional experience. He was researching misinformation and asked AI to generate examples of false or misleading content for analysis purposes. One of the examples it produced was a quote. The quote was attributed to Lindley himself. It cited the European Union as a source. It was perfectly framed - aligned with his known positions, written in a register consistent with his public voice, attached to an authoritative-sounding institutional reference.

He was one second from sharing it. He stopped. He checked. The quote did not exist anywhere. The EU source did not exist. None of it was real.

"What nearly got me was not the inaccuracy. It was the precision. The AI gave me exactly what my instincts were primed to accept. It confirmed my expertise. It cited authority I respect. My positivity bias closed the gap."

This is what makes AI hallucinations particularly dangerous in professional contexts. The errors are not random. They are plausible. They are tailored to the context. They are designed - not deliberately, but structurally - to bypass the critical instincts of the person asking.

‍

Why executives trust AI faster than anyone

The episode explores why trust in AI tends to be strongest at the top of organisations - and why that is also where the stakes are highest. Edosa describes the conditions that produce executive over-trust: vendor demonstrations that are polished and impressive; promises that are enormous; the social pressure of publicly committing to AI as a strategic priority.

The problem is structural. When a leader publicly commits to trusting AI, they remove the cultural permission for anyone beneath them to question it. The organisational signal - AI is the answer - makes it harder to raise concerns, report problems, or acknowledge that the system is producing outputs that need checking.

"Executives trust AI faster than individuals do. And the stakes are higher. When a leader publicly commits to AI, they remove the permission for anyone below them to question it."

‍

The satnav effect: how AI is making us think less

Edosa introduces what he calls the satnav effect. Before GPS navigation became ubiquitous, most drivers had a working mental map of their local area. They could plan routes, estimate times, and recognise when they had taken a wrong turn. That capability has atrophied for many people who now depend entirely on navigation systems.

The same effect is visible with AI tools. When people outsource judgment to AI - research, analysis, decision support - the cognitive muscles that those tasks develop begin to weaken. The question is not whether AI makes individual tasks easier. It is what we lose in the process of no longer doing those tasks ourselves.

"Sat nav reduced our spatial awareness. AI tools may be doing the same to judgment, critical thinking, and the ability to identify when an answer does not feel right."

‍

If we trust AI too much we stop asking questions

The episode closes with what Lindley identifies as the most fundamental risk of AI over-trust: we stop asking questions. Asking questions is the primary mechanism by which we catch errors, surface assumptions, and challenge conclusions. It is the skill that makes everything else possible.

If AI produces confident answers and we accept them without scrutiny, we do not merely risk individual errors. We risk the gradual erosion of the cognitive infrastructure that makes good judgment possible in the first place.

‍

Listen to Episode 3

‍

Going deeper: the trust problem is a design problem

The two stories at the heart of Episode 3 - a family in hospital because of a confident AI answer about food allergies, and a journalist nearly sharing a fabricated quote attributed to himself - are not edge cases. They are illustrations of a design characteristic that is consistent across all large language models: confidence is uniform, regardless of accuracy. The systems do not know when they are wrong. They generate responses that feel equally authoritative whether they are describing something well-documented or fabricating something entirely.

The reason Lindley's story is particularly instructive is the positivity bias he identifies. The hallucinated quote was not obviously wrong. It was precisely tailored to be plausible to him specifically. It confirmed his existing positions. It cited sources he trusted. It was formatted like something he might have said. The AI did not produce a random error. It produced an error that was specifically designed - by the mechanics of how these systems work - to bypass his critical instincts. This is the nature of the risk. It is not that AI produces bad outputs. It is that it produces bad outputs that look like good ones.

The organisations that manage AI trust well do not do so by trusting AI less. They do so by building structured verification into their workflows. They distinguish between decisions where AI error is low-stakes and reversible, and decisions where it is high-stakes and consequential. They maintain cultural permission at every level to question AI outputs without that questioning being read as resistance to the technology. And they invest in the human skills - critical thinking, source verification, domain expertise - that make verification possible. Trust in AI should not be unconditional. It should be proportional.