When confident AI gets it wrong

Generative AI makes this particularly difficult because its outputs can sound complete, polished and authoritative. That confidence can encourage people to accept an answer before they have properly tested its evidence, assumptions or relevance. As AI becomes part of everyday work, the ability to question what it produces becomes a leadership capability as much as a technical one.

In Episode 21 of The AI Values Podcast, Professor Julie Wall joins hosts Edosa Odaro and Lindley Gooden to explore critical thinking, the future of work, deepfakes, fairness, human judgement and the responsibilities that emerge when AI enters higher-stakes decisions. 

Across the conversation, one principle keeps returning: technical capability tells leaders what a system can do. Responsible deployment requires judgement about how that capability should be used.

Confidence can make AI harder to question

One of generative AI's strengths also creates a risk.

Its answers often arrive quickly and with considerable confidence. For someone trying to understand a subject, prepare a document or complete a piece of work, the quality of the language can make the output feel more authoritative than the evidence behind it.

Julie sees this directly in education.

“The answers are so confident in and of themselves that when you're looking at them, I find this that you think, well, could I say this any better?”

The concern she raises is overreliance. AI can be genuinely useful to students and professionals, but usefulness can gradually become dependence when people stop challenging what the system gives them.

That makes critical thinking increasingly valuable.

Julie argues that people need to verify AI-generated material and actively challenge its outputs. Universities already teach students to evaluate books, websites and research papers according to their quality and reliability. AI-generated information now needs to sit within that same discipline of scrutiny.

For organisations, this reaches far beyond education. Employees are increasingly using AI to research, draft, analyse and summarise information. Leaders therefore need people who can recognise when an answer requires further investigation and who remain confident enough in their own expertise to challenge a convincing machine-generated response.

The first answer should rarely be the last

Critical thinking also changes how people interact with AI.

The quality of an output depends partly on the quality of the interaction around it. Accepting the first response limits the opportunity to expose assumptions, explore alternatives or identify missing information.

Julie describes the process simply:

“I think it's about asking the right question and knowing that the first question isn't the end question.”

This matters because effective AI use requires more than knowing how to enter a prompt. People need enough subject knowledge and judgement to recognise what should be questioned next.

That has implications for workforce development.

AI literacy should include the ability to interrogate outputs, compare evidence and refine a line of enquiry. Employees also need to understand where AI-generated information requires independent verification, particularly when the consequences of an error are significant.

As AI becomes more common across professions, those skills will increasingly belong to people outside technical teams.

Julie expects AI to become part of everyday work in much the same way that email and search engines already are. Routine activities such as gathering information, preparing documents and basic coding are likely to become increasingly automated, creating more space for work that depends on human capabilities.

The organisational opportunity lies partly in how that space is used.

Technical performance can hide human differences

An AI system can perform well against a technical measure and still behave poorly when it encounters the diversity of people it is supposed to serve.

Julie illustrates this through research into AI-enabled educational tools.

Many such systems use technologies such as emotion recognition and sentiment analysis to infer engagement or attention. The difficulty comes when a behaviour is interpreted as meaning the same thing for every person.

Julie discusses gaze aversion as an example. For one group of users it may be interpreted as disengagement, while for a neurodivergent learner the same behaviour may carry a different meaning. A model trained around general assumptions about behaviour can therefore produce technically plausible interpretations that fit some users poorly.

This has wider consequences for AI governance.

Leaders need to understand the assumptions embedded in the signals an AI system uses. Performance statistics alone cannot show whether those assumptions work equally well across different groups or contexts.

That becomes especially significant when an AI output influences access, treatment, assessment or another consequential decision.

Technical testing therefore needs to be accompanied by questions about users, context, fairness and how the output will eventually be applied.

Prediction still requires human judgement

The same problem becomes more serious in higher-stakes settings.

Julie describes research involving the use of AI to detect credibility through voice patterns in an insurance context. The system could identify features such as uncertainty, hedging and over-explaining.

Yet identifying a pattern does not establish why that pattern exists.

A person might sound uncertain because they lack credibility. They could also be stressed, communicate differently or respond unusually to the situation they are in.

Julie explains the governance problem:

“How reliable are the signals being used that the AI is reading and making a decision on?”

From there, further responsibilities emerge. Organisations need to understand how the output will be used, who remains accountable for the eventual decision and how bias will be identified and addressed.

This distinction becomes essential when leaders evaluate AI systems.

Pattern recognition can reveal information that deserves attention. Human judgement determines what that information means within the context of an actual decision.

A prediction should therefore have a clearly defined role in the decision process. Leaders need to understand how much authority the system has, when a person must intervene and who carries responsibility for the outcome.

Deepfakes create uncertainty as well as deception

The risks of AI-generated content extend beyond incorrect answers.

Deepfake technology can produce convincing synthetic voices, images and video with increasingly limited source material. Julie argues that the resulting problem reaches into public trust.

“A convincing deepfake doesn't need to fool everybody. What it needs to do is create uncertainty and it's the uncertainty where the problem is for most people.”

That distinction matters.

A fake does not need universal acceptance to cause damage. It can create enough doubt around an authentic recording, statement or event to make people uncertain about what they should believe.

Julie points out that deepfake technology is becoming accessible beyond high-profile targets. She discusses an example involving a US high-school principal whose voice was allegedly imitated in a recording containing controversial statements, illustrating how synthetic media can affect ordinary people and institutions as the tools become easier to use.

Detection technology is part of the response. Education and verification are equally important.

For organisations, this creates new questions around evidence and trust. Executives may increasingly need procedures for authenticating sensitive communications, responding to suspected synthetic content and establishing which sources employees should trust during a fast-moving incident.

Deepfakes therefore create a governance problem alongside a technical one. Organisations need to prepare for an environment in which seeing or hearing something may provide insufficient evidence that it is genuine.

Listen to Episode 21

Going deeper: judgement becomes part of AI infrastructure

Episode 21 points towards a broader organisational consequence of widespread AI adoption. As systems become easier to use, the quality of human judgement around them becomes more significant.

Traditional technology governance can concentrate heavily on the system itself: its security, performance, data, accuracy and compliance. Those controls remain necessary, but they cannot determine whether every output is appropriate in the context where it will eventually be used.

That responsibility sits partly with people.

An employee using generative AI needs enough understanding to recognise when an answer deserves challenge. A manager using an AI-generated recommendation needs to understand its limitations. A board considering an AI-enabled process needs visibility into how decisions are made, who can intervene and where accountability remains.

Organisations therefore need to treat critical thinking as part of AI readiness.

Training should go beyond explaining which tools employees may use. People need practical experience in checking sources, challenging assumptions, identifying uncertainty and understanding how confident language can conceal weak evidence.

The same principle applies to system design. AI should support human decision-making in ways that preserve meaningful judgement, particularly where the consequences affect people's opportunities, finances, health or rights.

This also changes how boards should think about value. A technically capable system can still create poor outcomes when its assumptions fit the wrong population, its outputs are accepted too readily or responsibility becomes unclear.

The strongest organisations will build the human capability to question AI alongside the technical capability to deploy it. That means creating environments where challenging an AI output is expected, where accountability remains visible and where speed does not remove the opportunity for judgement.

AI can make information easier to generate.

Leadership still determines what deserves to be believed and acted upon.