Who is Accountable When An AI Agent Makes a Mistake?

In 2022, a customer asked Air Canada’s chatbot about bereavement fares and the bot invented a policy on the fly. The customer booked the flight, claimed the discount, was refused, then subsequently sued and won the case. The tribunal’s reasoning was simple. The airline owned whatever its AI told the customer.

That case and other similar stories keep coming up in conversations, and it’s a super interesting issue because it isn’t really about the mechanics of a chatbot. It’s about the ethical lines businesses are now constantly crossing without really meaning to. Up until very recently, most enterprise AI has lived in an assistive posture. Models mostly just drafted and recommended. It’s been advisory work, with advisory liability. But the moment systems began taking action inside real workflows, that arrangement changed, and most organisations are only halfway through working out what that means.

This is the gap that frameworks like AIUC-1 are trying to fill. AIUC-1 defines what responsible AI agent behaviour should look like at enterprise scale. It isn’t insurance, and it isn’t designed to slow adoption in any deliberate way. It’s an early attempt at a shared reference point for how autonomous systems get evaluated and governed once they start interacting with real business processes. It’s part of the developing rulebook for how these things behave inside organisations.

The pattern is familiar in the history of business development. Algorithmic trading triggered flash crashes, but the response wasn’t to abandon algorithmic trading. It was to introduce circuit breakers. Cloud computing created a new category of security exposure, but the answer wasn’t to walk away from the cloud. It was a generation of compliance frameworks that made the cloud something a board could approve. AI agents are now approaching the same junction.

You can’t underwrite what you can’t standardise. That is the bottleneck most AI strategies are about to hit. Capability keeps advancing, and the technology will keep getting better. What’s harder to build, and what tends to lag by a year or two, is the surrounding trust architecture that lets a CFO sign off without hesitation.

Industries that sit close to operational risk recognise this faster than most. Real estate and mining have spent decades building structure around physical and financial exposure, and when new forms of leverage appear in those sectors the conversation moves quickly past what the technology can do and toward how the exposure will be priced or contained. AI agents are starting to trigger that same instinct.

The interesting shift is that the next bottleneck for most AI programs may not be technical at all. It’s increasingly a question of insurability and governance maturity, and the onus is moving to each business to demonstrate that its autonomous systems are operating inside something the market can recognise.

Air Canada paid out a few hundred dollars in 2024, and took their chatbot offline for “maintenance.” But the more useful number is how many CFOs read that ruling and started asking different questions about the agents already running inside their organisations.