AI needs new rules

Organisations may be making a fundamental mistake with AI before they even begin implementing it.

They are taking assumptions developed for conventional technology and applying them to systems that behave differently. Executives expect predictable answers. Business cases focus on automation and cost reduction. Governance attempts to constrain uncertainty. Yet generative AI is probabilistic by design. If leaders misunderstand that distinction, they risk designing the wrong systems, expecting the wrong outcomes and governing the wrong risks.

In Episode 16 of The AI Values Podcast, Santiago Castro, Director of Data & AI Strategy at Pay.UK, joins Edosa Odaro to examine what happens when organisations confront a new technology using old assumptions. Their conversation moves from deterministic systems and AI experimentation to cost reduction, human judgement, risk, organisational values and the future of work.

The central leadership challenge is not simply learning how to use AI.

It is learning when established ways of thinking no longer fit.

We are using old frameworks for a new technology

Traditional software gives organisations a useful expectation: defined inputs should produce defined outputs.

Generative AI changes that relationship.

As Castro explains:

"We often tend to look at new problems with old frameworks."

Much of organisational technology has been built around deterministic thinking. Code is written to perform particular operations. Systems are tested against expected outcomes. Leaders become accustomed to technologies behaving predictably within specified parameters.

Generative AI introduces probability into that model.

Castro argues that organisations can therefore misuse AI when they approach it expecting the certainty associated with conventional software.

That misunderstanding can begin with something as simple as a search box. A user sees a familiar interface, asks a question, receives an unsatisfactory answer and concludes that the technology has failed.

But interacting with generative AI is not necessarily equivalent to retrieving information from a search engine. It can require dialogue, iteration, challenge and further questioning.

That changes the skills required of the human using it.

"You need to be active you need to be curious you need to ask the next question you need to research"

For leaders, this distinction matters far beyond prompting.

If AI behaves differently from previous technologies, organisations may need different expectations for design, testing, adoption, governance and performance. The risk is not merely that employees use AI badly. It is that the organisation builds its AI strategy around assumptions that no longer hold.

Stop treating AI as an automation project

The same old assumptions appear in organisational transformation.

A process exists. Leaders identify inefficiency. Technology is introduced to automate it.

AI makes that approach tempting because automation can produce visible savings. But Castro argues that simply automating an existing process may miss the larger opportunity.

"automated process is almost like a process upgrade but what we need is real transformation is real redesign"

This is a significant distinction for executives.

Automation asks how technology can perform an existing activity more efficiently.

Redesign asks whether the activity, process, service or business model should work differently now that new capabilities exist.

The first can reduce costs. The second can create new value.

Castro points to earlier digital shifts that produced companies and business models that would have been difficult to construct by merely improving what existed before. Connectivity, data and applications did not simply make old processes faster. They enabled different ways of connecting businesses and customers.

Generative AI creates a similar strategic question.

Boards should therefore be cautious about AI programmes whose ambition begins and ends with automating today's organisation.

The better question may be what tomorrow's organisation could do differently.

Cost cutting can become a strategic trap

AI's potential to reduce costs is difficult for organisations to ignore.

But a cost-cutting strategy and a value-creation strategy can lead businesses in very different directions.

Castro illustrates this through two hypothetical companies. Both begin with 100 employees.

Company A uses AI primarily to automate processes and reduce its workforce. It captures an immediate financial benefit.

Company B keeps its people and uses AI to stimulate ideas, develop new services and find different ways of serving customers. Its objective is expansion rather than contraction.

"cutting costs that's a very tactical kind of approach right short term"

This does not mean efficiency is irrelevant. Organisations have legitimate reasons to reduce unnecessary expenditure.

The governance question is what happens next.

If AI is used principally to shrink the existing organisation, leadership needs to consider whether short-term efficiency could weaken longer-term competitiveness, organisational knowledge or the capacity to innovate.

Castro's alternative is augmentation.

Employees already possess knowledge of customers, processes, markets and organisational realities. Combining that knowledge with AI may allow an organisation to create something new rather than simply perform the old work with fewer people.

That changes the executive conversation from what can AI remove? to what can people and AI create together?

Experimentation needs permission to fail

Organisations often demand certainty from technology investments.

Generative AI makes that difficult.

Castro compares the challenge with pharmaceutical experimentation, where multiple attempts may fail before one produces a significant result.

"we're in an experiment face and we're confused because we're still learning and experimenting"

That uncertainty does not eliminate accountability.

It changes how accountability should work.

If every AI experiment must demonstrate an immediate return, teams will naturally gravitate towards familiar and easily measurable applications. That may favour incremental automation over genuinely new ideas.

A more useful model separates experimentation from production.

During experimentation, organisations can allow wider exploration, simulation and rapid testing. Ideas can fail without becoming organisational failures. Once an application moves towards production, governance can become tighter. Controls, accountability and clear operating boundaries become increasingly important.

Castro offers boardroom decision-making as one example of where experimentation could create value. AI could review material, surface forgotten points or introduce perspectives that participants have avoided raising because of organisational hierarchy or group dynamics.

"what are we missing"

That simple question captures an important use of AI.

The system does not necessarily need to make the decision. It can help humans notice what their existing process failed to surface.

Human oversight must have a real boundary

Experimentation cannot mean giving AI authority everywhere.

Some decisions carry consequences that demand stronger boundaries.

Castro describes a financial-services example in which AI helped customers understand complicated financial products. Rather than expecting people to work through material they might struggle to engage with, an AI interface could explain the information conversationally and answer questions.

But there was a hard boundary.

"any point the person tries to ask for financial advice they I switch and say I cannot talk about that I'll pass you a human"

The distinction is instructive.

Explaining information was one task. Giving regulated financial advice was another.

The organisation did not need to choose between using AI everywhere and rejecting AI entirely. It needed to understand where the technology was appropriate and where responsibility should return to a person.

This is what meaningful human oversight should look like.

A human in the loop cannot simply be a governance phrase added to an AI policy. Leaders need to define the circumstances in which human judgement becomes mandatory, who has authority to intervene and what the AI system must do when it reaches that boundary.

For Castro, some areas should remain clear no-go zones.

"we shouldn't go there with with life-threatening decisions like that with AI"

Healthcare, defence and other high-consequence environments make this question especially urgent.

The more serious the consequence, the less acceptable it becomes to treat human oversight as an afterthought.

Values need to survive contact with reality

AI governance becomes harder when there is no single definition of value.

An AI system might reduce fraud while making the customer experience worse. It might increase efficiency while reducing employment. It might create financial value for one part of an organisation while introducing operational risk elsewhere.

Different stakeholders can therefore look at the same system and reach different conclusions about whether it is creating value.

That makes perspective part of governance.

Castro argues that AI can itself help people examine a problem from different points of view. He describes creating AI agents based on authors whose thinking he values, then asking the same difficult question from several perspectives.

The purpose is not to allow AI to make the final judgement.

It is to broaden the material available to the human making it.

"you will make up your mind"

This matters because organisational values are not necessarily permanent instructions that can be written once and mechanically applied forever.

Context changes. Markets change. Societies change. Consequences become visible that were not apparent when a decision was originally made.

"a value in itself is not something that is cast in stone forever"

For boards, that creates a responsibility not merely to define organisational values, but to revisit how those values apply as circumstances change.

Governance therefore needs memory as well as rules.

Leaders should be able to explain what decision was made, which values informed it, what evidence was available and why that judgement was reasonable at the time. They should also be prepared to reassess it when reality changes.

Listen to Episode 16

‍

Going deeper: AI governance must match the technology

Episode 16 raises a larger institutional problem.

Many organisations are adopting generative AI while their management systems remain designed for deterministic technology.

That mismatch can appear everywhere.

Procurement may expect predictable functionality before approving an investment. Risk teams may attempt to remove uncertainty rather than decide where uncertainty is acceptable. Business cases may reward immediate efficiency while overlooking experimentation. Governance committees may concentrate on controlling outputs without asking whether the underlying process should have been redesigned.

Boards therefore need to distinguish between at least three different environments.

There are areas where deterministic behaviour is required. There are areas where generative exploration is useful. And there are areas where the consequences are serious enough that human judgement must retain authority.

Treating all three identically creates poor governance.

The deeper issue is accountability.

A probabilistic system does not remove the organisation's responsibility for what happens around it. If anything, uncertainty makes the design of responsibility more important. Someone still needs to decide the objective, define the boundaries, assess the consequences and determine when a human must intervene.

The same principle applies to workforce decisions.

AI can be used to remove people from existing processes. It can also be used to increase what those people are capable of creating. Those approaches may produce very different organisations several years from now.

Leadership therefore needs to decide what kind of institution it is trying to build, not merely which AI tools it wants to deploy.

The technology may be probabilistic.

Accountability cannot be.