Why are you using AI?

AI can make a bad process move faster. It can expose problems that were already there, produce convincing answers that people trust too readily and, as systems gain greater autonomy, allow the consequences of poorly defined objectives to travel further through an organisation before somebody intervenes.

That makes the starting point for leaders surprisingly simple: understand the outcome first.

In Episode 20 of The AI Values Podcast, board-level digital and technology leader Mat Mallett joins hosts Edosa Odaro and Lindley Gooden to discuss what happens when AI leaves the demonstration environment and enters real organisations. Drawing on experience across sectors including healthcare and space, Mallett repeatedly brings the conversation back to purpose, preparation and human judgement.

His question for leaders is direct:

“What's the outcome you're trying to get to? And very much, why are you using AI to do it?”

Answering that properly can prevent wasted investment and give organisations a clearer basis for deciding where AI belongs.

AI can expose problems faster

AI carries a powerful promise for organisations. Processes can become quicker, large volumes of information can be analysed and work that previously consumed significant amounts of employee time can be automated.

That promise creates pressure to deploy, particularly when leaders are surrounded by claims about what the latest systems can achieve. Mallett warns against assuming that introducing new technology will resolve underlying organisational problems.

“There is my concern is there's an over-reliance on this is the latest tech tool and it's going to solve all our problems.”

His experience points towards a more complicated reality.

AI can examine information across an organisation and reveal connections that humans may struggle to see. A problem that appears to originate in one department may have its cause somewhere entirely different.

Mallett describes the possibility this way:

“You think your problems over there to the right, but actually it starts over here on the left and you need to sort that first.”

That diagnostic capability can be valuable, but finding a problem still leaves leaders responsible for deciding what happens next. They need to interpret what the system has identified, understand the organisational context and determine an appropriate response.

AI may accelerate diagnosis. The organisation still needs people capable of exercising judgement once the problem becomes visible.

Real deployment requires preparation

The distance between an AI prototype and a dependable operational system can be substantial.

A demonstration may show that something is technically possible. Production environments introduce customers, employees, unusual cases, imperfect data, existing systems and real consequences when something goes wrong.

Mallett draws on his experience of automating processes:

“I can reduce it down to 10% of the time it takes. But that takes a lot of planning in advance and a lot of preparation in advance.”

Preparation therefore becomes part of the value calculation.

Teams need to understand the different journeys a system may take, anticipate exceptions and establish escalation routes. Human intervention points need to be designed into the process, with clear responsibility for what happens when the technology behaves unexpectedly.

For boards, a successful pilot should therefore be treated as the beginning of a deeper operational conversation. Leaders need to understand how the system was tested, how unusual cases will be handled, where human intervention occurs and who carries responsibility when an output is wrong.

Those details reveal considerably more about deployment readiness than an impressive demonstration.

Generative AI still demands judgement

Mallett uses a memorable analogy for today's generative AI systems.

“They're like a teenager. They think they know everything. They think they have the answers to everything. But they hallucinate badly.”

The comparison captures one of the difficulties facing organisations. Generative AI can communicate with remarkable confidence, and that fluency can make its output easy to accept, particularly when employees are working quickly or using AI in areas where they have limited expertise.

Mallett identifies reliance as a particular concern:

“My concern is people relying on the outcomes of these generative AI solutions at the moment.”

This places AI literacy alongside technical governance. Employees need to understand the capabilities and limitations of the systems they use, while retaining enough critical judgement to question convincing outputs.

The same principle applies in the boardroom.

Mallett describes using AI to summarise board papers after he has read them. In that example, AI helps him return to and organise information he has already considered, while his own understanding of the material remains active.

That distinction provides a useful model for organisational use. AI can support comprehension, synthesis and analysis, while leaders remain responsible for interrogating what it produces and deciding how much confidence to place in the result.

Autonomy raises the stakes

Greater autonomy changes the governance challenge because the system can move from generating information towards taking action.

An AI recommendation allows a human to decide what happens next. An autonomous or agentic system may interact with other systems, execute processes, or make multiple decisions before a person becomes involved.

Edosa frames the issue directly:

“When these things become autonomous, how do we ensure that, you know, we can protect the good and, you know, in effect, avoid the bad?”

Mallett's response emphasises human checkpoints, vendor accountability and guardrails.

“It's very crucial to have the human in the loop checkpoints.”

Human oversight needs substance behind it. A person sitting somewhere in a workflow provides limited protection if they lack the information, expertise or authority required to challenge a system and stop an action.

Organisations therefore need to define when intervention is expected and what authority the human retains. Escalation processes become particularly important when uncertainty is high or the consequences of an incorrect action are significant.

Technology providers also sit within this accountability structure.

“They need to be kept to account because ultimately they are the creators of these solutions.”

Third-party AI creates dependencies that leaders need to understand. Organisations should know which controls sit with the provider, which remain within the organisation and how responsibility will be handled when systems behave unexpectedly.

As autonomy increases, those boundaries become more consequential.

Personal data changes the risk

AI systems can process volumes of information that no individual could realistically review.

Mallett highlights what this means for personal data:

“AI, on the other hand, can and it works 24 seven and it can map systems globally at lightning speed and pull a complete profile of everything to do with my life, your life, Edosa's life, you know, everyone.”

The implications extend into inference and influence.

Systems can identify relationships between behaviours, interests and people. Recommendation engines can use those connections to provide relevant information, while repeated recommendations can also shape what people encounter and which perspectives remain visible to them.

Mallett raises the potential loss of diversity of thought as part of this concern.

For leaders, this broadens the way AI risk should be considered. Governance needs to account for how data is combined, what systems can infer from it and how those inferences influence decisions or behaviour when applied repeatedly at scale.

Direct disclosure of a confidential record is only one form of potential harm. What a system can infer about an individual can also carry consequences.

Boards need space to question the answer

Boards already operate under significant information pressure. AI can help organise, summarise and analyse some of that material, but greater speed creates its own governance challenge.

Mallett describes the pressure facing boards:

“I think it's the wealth of information. I think it's the speed of change that is overwhelming.”

The ability to generate a rapid answer can create an expectation that decisions should also become rapid. Yet significant organisational decisions still require context, challenge and judgement.

This is especially important when AI produces fluent analysis. A polished answer can appear authoritative before its assumptions, sources or reasoning have been properly examined.

Mallett returns to a simple human capability:

“We still need to be able to step back and go, is it right?”

For boards, that ability to pause and interrogate an answer becomes part of responsible AI use. AI can reduce the effort required to process information, giving directors more capacity to examine implications, challenge assumptions and consider consequences.

The value comes from what leaders do with that additional capacity.

Purpose has to survive the hype

Mallett has worked with technology in environments ranging from healthcare to space, yet his recurring focus remains grounded in the outcome an organisation is trying to achieve.

That focus becomes increasingly valuable as new capabilities create pressure for organisations to demonstrate that they are keeping up. Generative AI, agentic systems, quantum computing and future forms of autonomous technology can all attract attention, while organisational value still depends on what leaders choose to do with them.

Mallett describes his own approach:

“How is this going to benefit the business? How is this going to benefit the teams I'm working with? How is this going to actually bring public value? You know, can I do it another way that's cheaper?”

Purpose provides a filter for those decisions.

Leaders need to understand the benefit they expect to create, the people affected and the dependencies introduced by the technology. Cost, risk and accountability belong in the same assessment because each can change the value of the eventual outcome.

A leadership team should therefore be able to explain why AI has been chosen for a particular problem and what success will look like before deployment begins.

Listen to Episode 20

Going deeper: accountability becomes more valuable as AI gains autonomy

Episode 20 points towards a governance challenge that becomes more significant as AI systems gain the ability to act. Organisations have spent decades building accountability structures around people, with roles carrying authority, decisions moving through escalation routes and responsibility being assigned when something goes wrong.

Autonomous AI complicates those structures.

An agent may make hundreds of decisions, several systems may interact and a third-party model may sit inside an internal process. Employees can also come to rely on recommendations without fully understanding how they were generated, making responsibility increasingly difficult to locate when outcomes emerge from a combination of human and automated actions.

Boards therefore need to design accountability as part of the system from the beginning. Every significant AI deployment should have a defined purpose, an accountable owner and clear boundaries around the authority given to the technology.

Human intervention points also need to work in practice. The person responsible for oversight needs enough information to understand what is happening and sufficient authority to intervene when necessary. A nominal approval step offers little protection when the reviewer cannot meaningfully challenge the system.

Vendor relationships require the same clarity. Using technology supplied by another organisation does not remove the consequences for the organisation deploying it, so leaders need to understand what providers control, what remains under internal control and how responsibility is divided when systems behave unexpectedly.

Measurement should follow the same discipline. A faster process can still produce poor outcomes, while a cheaper process can transfer costs or risks elsewhere. Highly capable systems may also encourage people to place more confidence in their outputs than their reliability warrants.

Successful deployment therefore depends on leaders maintaining sight of the wider system around the technology: its purpose, the people affected, the data it uses, the controls around it, the consequences it creates and the accountability structures that govern it.

Keeping those elements connected becomes increasingly important as AI gains greater independence.