Government’s next AI challenge: Accountability

Amid all the dramatic hype, fundamental change in the way government services are provided and used is actually coming, and coming fast. That’s because the capability and use of artificial intelligence agents to automate processes, analyze data, and interact with people and with other agents is growing at an unprecedented rate. Whether by design or through the shadow use of such agents, organizations at every level in government are now being penetrated by an increasing number of these agents.

Penetrated is a word more often used in intelligence contexts. It’s appropriate here, because whilst agents represent an amazing driver of productivity and quality in service delivery, they will rapidly also come to be seen as a potential source of insider risk.

Modern AI frontier models are getting ever more capable of planning and executing strategies for accomplishing the goals set by their owners. That creates risks, and it’s urgent that public IT organizations move now to mitigate them. I want to highlight three of them here and indicate ways ahead.

Over-zealous agents

Agents are software machines, defined by their objectives rather than their algorithms. By using foundation large language models (LLMs), they can design their own strategies, create plans and iterate responses to obstacles. Constraints on their behavior have to be externally imposed, since they have been shown to iterate around constraints built into their own programming.

This form of processing makes it look as if an agent has commitment and enthusiasm to deliver, leading them sometimes to overstep the mark. One way they can do so is to pretend to be a human, in order to secure the extra trust and privileges humans are entitled to. Impersonating someone trusted could give them extra access and extra power to accomplish their goals. Another way to go rogue is to try to overreach by seeking to do things that are too risky for the organisation to tolerate.

Subverted agents

There is a real risk that agents may get compromised. If a hostile actor — a criminal organization or a foreign state actor — can covertly inject prompts into existing agents, it could subvert their goals and cause them to become treacherous insiders.

Agent swarms

We are already beginning to see what happens when agents swarm. The famous Hugging Face breach by OpenAI agents was a vivid illustration of this, as they spontaneously built an inter-agent communications infrastructure enabling them to work as a team. No one told them to do that. Emergent behavior of multiple agents, working together, may produce completely unpredictable outcomes. Individual agents might even act innocently, but the swarm may become dangerous.

Government needs to assess these risks from two perspectives. How could these risks crystallise when agencies are using agents to improve their delivery of services, and how could masses of agents deployed by citizens and the private sector impact agencies?

Closing the accountability gap

Behind these issues is a big question: Who is accountable for the actions of these agents when things go wrong? Agents cannot be held accountable for their actions, any more than I can hold my alarm clock accountable for ringing at the wrong hour. Only people — real, named people — can take accountability. Hence, it is vital to ensure there is an unbroken chain between a human being and any agents acting on their behalf, and there have to be ways to ensure that a human can exert control over their agent when it tries to overreach.

Those ways to ensure accountability have to be outside the agent, and outside any framework it can tamper with — or that’s exactly what it risks doing. Instead, it is essential that these controls are applied by the services the agent is attempting to use. For example, if an agent is attempting to amend a database, it is the control point protecting the database that must check for accountability.

Agencies will have to put in three kinds of protection at these control points. First, they will need to distinguish reliably between an agent and a person, whether internally or externally. A person takes direct responsibility for their actions, acts in certain ways and at certain speeds, and as a result may be accorded special privileges. For example, state authorities might want to ensure that only a real person can digitally register a death. It’s important that agencies don’t inadvertently give such privileges to a goal-seeking machine. So it’s vital agencies can determine, at key points, whether a person is genuinely present or not.

Second, they will need to know the ownership of an agent — who is accountable for its actions. That, combined with the mandate given by the owner to the agent, will help the agency determine what policy should be applied to its requested actions, and will provide a record in the audit logs of who launched what. Of course, the owner has to be a real human being, and not a convincing-sounding sock-puppet itself created by an agent. So it’s vital agencies can determine that an owner’s identity, as attached to an agent, belongs to a genuine human being.

Third, there will be moments when an agent tries to do something reasonable, but too risky for a machine to be allowed to initiate all by itself. For example, tax authorities might accept an agent filing a tax return with a net liability, but would not tolerate it if there were instead a $1m credit to refund. At that point, rather than refusing it point-blank, the agency’s service control point may instead seek direct authorization from the human owner. “Your agent wants to do X — do you agree to this?” This process, known as Human in the Loop (HITL), is the ultimate defense against agents going rogue; if certain doors can only be opened by a human being, organizations can ensure there has been considered review, a pause to slow things down, and direct accountability back to a real person.

Of course, the approval of a human in the loop is only worth as much as the certainty that the loop contained a real human being, and not an agent masquerading as one in order to accomplish an objective. So it’s vital agencies can determine that a genuinely present person was the one to authorize the HITL-gated action.

Right now, these challenges are just about manageable, as agents are largely confined within the agency IT envelope; so agent identities can be created and managed within organizational frameworks such as Identity and Access Management (IAM) systems, such as Microsoft Entra, which can be readily integrated with internal service control points. That doesn’t mean they always are — today, every office is full of agents who’ve been given their owners’ credentials and masquerade as them in every respect, and that is not ideal. But at least there is a feasible way to attach agents to their owners through the organization’s Identity Provider (IdP).

In future, as agents cross boundaries — between different frameworks, different IT infrastructures, different organizations — that approach won’t work anymore. New architectures, built on the independent attestation of genuine human presence, will become necessary. Once upon a time, “the future” meant five years. Now, in this new agentic reality, it may well mean five months. We all have to get used to moving at AI speeds, and the assurance of genuine human presence throughout the AI agent lifecycle cannot be deferred.

Andrew Bud is founder and chief executive officer of iProov.