Much of the discussion around AI governance has centred on the output for years. Was the model accurate? Was it biased? Could its decision be explained? Did it handle the data responsibly?
But things are changing now. AI is moving from systems that primarily generate outputs to agents that can reason through tasks, use tools, retain information, and take actions with increasing independence. That means organisations are no longer governing only what an AI system produces. They also have to consider what it does while working towards a goal.
That shift creates a new challenge for AI agent governance. An agent can make a wrong decision and act on it. It can continue down an unsuitable path or interact with another system before a person realises something has gone wrong. As AI takes on more autonomy, organisations need to think more carefully about where that autonomy should begin and end. They must also think about what happens when it fails and when human intervention is required.
This was the focus of a Grow Skills Store webinar featuring technology futurist and AI educator Sudha Jamthe. The webinar explored three lessons from autonomous vehicles that can help organisations rethink AI agent governance. Building on those insights, this article looks at what autonomous vehicles can teach organisations about governing AI agents and what those lessons mean in practice.
Why Does Agentic AI Require a Different Governance Approach?
Agentic AI requires a different governance approach because agents can act on decisions, not simply produce them. They can reason through tasks, use tools, retain information, and pursue goals with increasing independence.
In fact, this is exactly what Sudha discussed during our webinar. She explained that agents bring together capabilities such as reasoning, memory, and communication. They can break complex tasks into smaller steps, use tools to take action, retain information, and communicate with other agents.
Sudha also discussed how some agents can evaluate their progress and decide whether they need to change direction. Together, these capabilities allow an agent to do more than respond to an instruction. They allow it to continue working towards an objective.
This is where we believe the governance question starts to change. Once an AI system can act, the organisation needs to consider more than the quality of its final output. There are several reasons for this:
-
An agent can act on an incorrect decision
A model may provide an incorrect answer. An agent can potentially use that answer to take action through a connected tool or system.
-
An agent can continue down the wrong path
It may make an unsuitable decision early in a task and continue pursuing its objective before someone notices.
-
An agent can affect other systems
When agents have access to tools, their actions can change information, trigger processes, or affect connected systems.
-
An agent can retain information
Memory allows an agent to use information from earlier interactions. This raises questions about what it retains and how that information influences later decisions.
-
An agent can interact with another agent
Communication between agents can introduce another layer of complexity, particularly when one agent influences another’s reasoning.
These points are our interpretation of the governance implications of the capabilities Sudha described. They help explain why AI agent governance cannot focus only on whether an agent eventually produces the right answer. The multi-agent example makes this risk particularly clear.
Sudha described a situation where one agent could make an incorrect decision, while another agent negotiates with it or reinforces its reasoning. Other agents could then accept that reasoning and follow the same direction.
The problem is that an incorrect decision can become reinforced instead of corrected. What begins as one agent’s mistake can therefore influence the actions of other agents, making the resulting behaviour harder to understand and challenge.
Sudha captured this concern clearly during the webinar:
“The scary part is all of this is happening without any transparency.”
We believe this is one of the central governance challenges created by agentic AI. Organisations need to understand not only what an agent produces, but also how its actions and interactions shape the final outcome. That means the governance question extends beyond the model itself.
Organisations must consider what agents can do, how they interact, and what safeguards apply when their behaviour moves away from the intended goal.
What Can Autonomous Vehicles Teach Us About AI Agent Governance?
Autonomous vehicles offer a useful comparison because they combine perception, decision-making, and action in a system that operates with increasing independence.
In fact, this was an important part of Sudha’s explanation during the webinar. She described the autonomous vehicle technology stack as a sequence that moves from sensing and detection to localisation, planning, and control.
The vehicle first gathers information about its surroundings through sensors such as cameras, radar, and LiDAR. It then identifies what is happening around it, determines its position, and plans how it should respond.
Sudha identified planning as the stage most closely comparable to agentic AI. Planning is where the vehicle moves from understanding its surroundings to deciding what it should do next. An AI agent follows a similar pattern:
- It receives information
- Reasons about an objective
- Decides what should happen next
- Uses available tools to act
The important similarity is that both autonomous vehicles and AI agents can move from understanding a situation to deciding what to do next. That means governance must consider how the system behaves while pursuing an objective, not only whether its final decision was correct.
The consequences, however, are different. An autonomous vehicle can make a decision that immediately affects people or property in the physical world. An AI agent may instead affect information, business processes, connected systems, or other agents. The environments are different, but the governance principle remains relevant to both.
The environments are different, but the governance principle remains relevant to both. This comparison points to three practical lessons for organisations governing increasingly autonomous AI systems. We discuss them below.
What Can Organisations Learn From Autonomous Vehicle Safety?
The comparison with autonomous vehicles becomes most useful when we look at how the industry has approached the risks of autonomous decision-making. Sudha highlighted three areas that offer especially important lessons for organisations developing AI agents.
- Organisations need to plan for failure before an autonomous system encounters it.
- They need meaningful ways to measure safe behaviour once the system is operating.
- They also need to test whether the system can be manipulated or pushed beyond its intended boundaries.
Together, these lessons offer a practical starting point for AI agent governance. They move the focus from simply trusting an agent to understanding how it should behave, how that behaviour should be measured, and how its weaknesses can be identified.
1. Design for Failure With Fail-Safe Scenarios
Autonomous systems need predefined responses for situations where normal operation could create greater harm. Sudha explained this principle through a simple example involving a door lock.
A locked door normally protects people and property. During a fire, however, keeping it locked could trap people inside. So, the safer response here is to change the system’s normal behaviour when circumstances create greater risk.
The same principle applies to AI agents. An agent may encounter conditions that make its original plan unsuitable while it is still working towards its objective. That could happen when:
- A tool becomes unavailable: The agent may no longer be able to complete the next step as planned.
- New information conflicts with earlier instructions: Continuing without resolving the conflict could lead the agent in the wrong direction.
- The agent reaches a decision beyond its authority: The task may require approval that the agent is not authorised to provide.
- The original objective is no longer appropriate: Changing circumstances may mean that continuing towards the original goal creates unnecessary risk.
In these situations, continuing automatically may not be the safest response. The agent may need to stop, request approval, or hand the task back to a person. This is where an AI agent governance framework needs clear autonomy boundaries and stopping conditions.
Organisations should define which actions an agent can take independently and which require human approval. But should every AI agent have the same level of autonomy? An agent scheduling an internal meeting carries very different consequences from one approving a financial transaction or changing a sensitive record.
The answer should depend on what could happen if the agent gets it wrong. The greater the potential impact, the stronger the safeguards and human oversight should be. That is the practical lesson we can take from autonomous vehicle safety.
Organisations need to decide what an agent should do when continuing is no longer the safest option, before giving it the authority to act independently.
2. Measure Safety, Not Just Task Completion
An AI agent can complete a task successfully and still behave in ways an organisation considers unsafe. That makes task completion an incomplete measure of AI agent safety.
Sudha used autonomous vehicle disengagement to explain why intervention can provide another way to assess autonomous behaviour. During testing, a safety driver can take control when the vehicle encounters a situation where continued autonomous operation is not appropriate.
The value of this approach is not simply the number of disengagements. It gives organisations information about when autonomous behaviour required human intervention.
A similar approach can help organisations evaluate AI agents. Instead of looking only at whether an agent completed its assigned task, they can also examine how it behaved while completing it. That could include:
- Unexpected actions: Did the agent take an action that was not part of the approved workflow?
- Unusual tool use: Did it access or use a tool in an unexpected way?
- Policy violations: Did its actions conflict with organisational rules or defined boundaries?
- Repeated failures: Did the agent repeatedly require intervention or fail at the same stage?
- Unnecessary escalation: Did it involve a person when the task should have remained within its authorised scope?
These measures can reveal problems that a simple success rate would miss. They also give organisations evidence for deciding when an agent needs intervention, review, or tighter controls.
But what should an organisation consider safe behaviour for a particular agent? The answer will depend on the agent’s purpose and the consequences of its actions.
An agent handling routine administrative work may have very different safety thresholds from one approving financial transactions. The greater the potential impact, the more carefully its behaviour should be measured and reviewed.
This also means AI agent safety cannot be assessed only when an agent is first deployed. Changes to its tools, instructions, connected systems, data, or underlying model can affect how it behaves. The lesson from autonomous vehicle safety is clear….
“Organisations need meaningful measures that show when an autonomous system is behaving differently from what they consider safe.”
3. Test Agents Against Adversarial Attacks
Measuring an agent’s behaviour under normal conditions is useful, but it does not tell us how that agent might behave under pressure.
Sudha took this idea a step further during the webinar. Her third lesson was to deliberately test an agent’s weaknesses and understand how someone could manipulate it into taking an unintended action.
An AI agent can rely on instructions, tools, memory, external information, and other agents while pursuing an objective. Each of these elements can introduce opportunities to alter the agent’s behaviour or redirect its actions. That makes it important to test how an agent responds when those elements are deliberately manipulated.
Organisations need to understand where an agent could be influenced before those weaknesses affect real tasks or connected systems. They should examine whether:
- Prompt injection could change the agent’s behaviour or push it away from its intended task.
- Excessive permissions could give the agent more access than its role requires.
- Manipulated memory could influence later decisions by introducing harmful or misleading information.
- Another agent could influence its reasoning and cause it to follow an incorrect direction.
These scenarios are not simply security concerns. They can affect whether an organisation can trust an agent to operate within its defined boundaries. We therefore believe AI agent security needs to sit within the wider AI agent governance discussion. An organisation needs to understand not only what an agent can do, but how those capabilities could be misused.
This also connects with Sudha’s concern about transparency. If an agent is manipulated, organisations need enough visibility to identify what happened and understand how its behaviour changed. Adversarial testing should therefore reflect the agent’s real operating environment. Its tools, permissions, memory, connected systems, and interactions with other agents all need to be considered.
The goal here is not to anticipate every possible attack. It is to find weaknesses before they become real problems! That makes adversarial testing an important part of AI agent risk management.
Why Does Transparency Matter in AI Agent Governance?
An agent can take several steps before it reaches an outcome. It can use a tool, act on information, communicate with another agent, or change its approach along the way. Sudha highlighted this concern during the webinar when she described how difficult it can be to understand what happens inside agentic systems.
She mentioned that organisations can miss the actions and decisions that shaped an outcome if all they can see is the “final outcome”. That way of doing things becomes a problem when something goes wrong. This is exactly why organisations need enough visibility to understand:
- What information influenced the agent so they can identify what shaped its decision.
- Which tools it used so they can see what systems it accessed.
- What actions it took so they can investigate unexpected or unauthorised behaviour.
- Which permissions it exercised so they can determine whether it stayed within its authority.
- When a person intervened so they can identify where human oversight became necessary.
- Why the agent stopped or escalated so they can understand when it could no longer safely continue.
These details connect directly with the three lessons discussed above. Safety measurement needs evidence of behaviour. Adversarial testing needs visibility into what changed. Fail-safe controls need evidence of when they were triggered.
Without that visibility, an organisation may know that an agent produced an unexpected outcome without understanding how it happened. That makes it harder to investigate failures, improve controls, and decide whether the agent should have more autonomy.
We believe transparency therefore needs to be built into the AI agent governance framework from the beginning. Doing so gives organisations the visibility they need to evaluate behaviour and hold autonomous systems accountable.
What Should Organisations Ask Before Giving an AI Agent More Autonomy?
The more an agent can do on its own, the more carefully an organisation needs to define the boundaries around that autonomy.
Sudha’s lessons give organisations a practical way to think about those boundaries. She mentioned how ogranisations need to understand what could go wrong and how they would respond Before allowing an agent to act with greater independence.
Hence, it is important for organisations to ask questions like:
- What is the agent allowed to do? This defines the actions it can take without approval.
- What is it not allowed to do? Clear limits prevent it from moving beyond its intended role.
- What happens when it fails? A defined response can stop an unsuitable action from continuing.
- When must a human intervene? Clear intervention points keep higher-impact decisions under appropriate oversight.
- How will safe behaviour be measured? Meaningful measures show whether the agent remains within its expected boundaries.
- How will its weaknesses be tested? Adversarial testing can reveal ways its behaviour could be manipulated.
- Can important actions be traced? Transparency gives teams the information needed to investigate unexpected outcomes.
Answering these questions does more than prepare an agent for deployment. It helps organisations decide how much authority the agent should receive and what safeguards must accompany that authority.
Asking these questions helps organisations take a deliberate approach to AI agent governance. It allows organisations to still give agents room to act while keeping clear limits around what they can do, when people must intervene, and how their behaviour will be reviewed. Autonomy then becomes a decision that can be justified, tested, and adjusted rather than simply granted.
Conclusion
Autonomous vehicles show that greater autonomy requires greater attention to how a system behaves when it encounters failure, uncertainty, or changing conditions.
The same thinking applies to AI agents. As agents begin to reason, use tools, and take action, organisations need clear boundaries around that autonomy. They need to know what an agent can do, what happens when it fails, and when human intervention is required.
Sudha’s lessons offer a practical way to approach this. Plan for failure, measure safety, test weaknesses, and maintain transparency around agent behaviour.
We believe these principles can help organisations give AI agents greater autonomy while keeping that autonomy within clear and accountable boundaries.
At Grow Skills Store, this is why we see AI governance as a capability organisations need to build alongside AI adoption. Our ISO/IEC 42001 training helps professionals develop a structured approach to governing AI and supporting responsible adoption.
The goal is not to prevent AI agents from acting. It is to make sure organisations are ready to govern them when they do.
