Robodebt didn't fail for lack of a tool. It failed for lack of the disciplines.
What Australia's most damaging automation failure teaches anyone now deploying AI in government and regulated business.
Between July 2015 and November 2019, an automated debt-recovery scheme known as Robodebt issued roughly 433,000 Australians with Centrelink debt notices that, together, demanded at least $1.76 billion. Many of those debts were not real.
The Royal Commission into the scheme, whose final report Commissioner Catherine Holmes AC SC delivered on 7 July 2023, did not soften its language. Robodebt, it found, was "a crude and cruel mechanism, neither fair nor legal," that "made many people feel like criminals." In essence, the Commission wrote, "people were traumatised on the off-chance they might owe money."
Two things before going further. First, the human cost was real and serious, and no analysis from the outside should reduce it to a case study. Second, this is not a story about software that malfunctioned. It is a story about disciplines that were missing. That distinction matters for everyone now rushing to automate decisions with AI.
It wasn't the automation. It was the absence of guardrails.
The technical heart of Robodebt was a shortcut called income averaging. The scheme took a person's annual income as reported to the tax office, spread it evenly across the year, used that crude average to assume fortnightly earnings, and raised a debt wherever those assumed earnings clashed with Centrelink's records. But real income is lumpy. People work seasonally, change jobs, take leave. Averaging invented fortnights of earning that never happened, then turned them into debts.
The scheme then reversed the burden. Instead of the government proving a debt was owed, people were told to prove they did not owe it, often for income going back years, with payslips they no longer had.
Automation did not make those choices. People did. The algorithm only executed them faster, at a scale no manual process could reach, which is exactly why the harm spread so wide. A tool did not cause Robodebt. The absence of four basic governance disciplines did. And I am not going to claim that any tool would have stopped it, because that would be both untrue and beside the point. What follows is about the disciplines, not the software.
The four disciplines Robodebt was missing
1. A lawful basis, checked before launch, with the power to halt
Concerns about whether income averaging was even lawful existed inside government before the scheme scaled. They did not stop it. A debt raised this way was later conceded to be not validly made, and the approach was ultimately found unlawful.
The discipline: lawfulness is the first gate, not the last. Someone has to ask "do we actually have a lawful basis for this?" before a system that affects people goes live, and that question has to carry a real veto. A governance process where legal doubt cannot actually halt the thing is not governance. It is paperwork.
2. Human oversight of decisions that affect people
Robodebt automated a consequential decision, asserting that a citizen owed the state money, and removed meaningful human judgement from the loop at the scale that mattered. Decisions that change someone's life were made by a formula and a letter.
The discipline: the higher the stakes for a real person, the more a human has to be accountable for the individual decision, not just for the system that makes it. Automate the clerical work. Do not automate away the judgement on a decision that can take food off someone's table.
3. A challenge and redress path for the people affected
When the notices landed, the people on the receiving end had little genuine ability to contest them. The onus was reversed, the evidence demanded was often impossible to produce, and the path to push back was opaque and intimidating.
The discipline: if a system can be wrong about a person, that person needs an obvious, usable way to challenge it and be made whole. Build the appeal path before you build the scale. A system with no working redress is a system that has quietly decided it cannot be wrong.
4. Named accountability
One of the most damning threads running through the Commission's findings was how diffuse responsibility had become. Warnings were not escalated, advice was not acted on, and it was rarely clear who owned the decision to proceed.
The discipline: every consequential system needs a named owner who is accountable for it, plus a record of who decided what, when, and on what evidence. Accountability that belongs to everyone belongs to no one.
Why this matters more now, not less
Robodebt was rules-based automation. AI is about to put the same temptations on steroids: decisions made faster, at greater scale, by models whose reasoning is harder to inspect and easier to trust blindly. Every failure mode above gets worse when the system is a model rather than a formula.
The lesson is not "don't automate." Automation done well frees people for higher-value work, and it can make services fairer and faster. The lesson is that the disciplines have to scale with the stakes. Fast where it is safe. Controlled where it is risky. Provable to the people it affects.
These four disciplines are not exotic. They are what proportionate, accountable governance looks like in practice, and they are exactly the disciplines I try to build into the way regulated organisations adopt AI. Not as a brake on it, but as the thing that lets them move without harming the people they serve.
Robodebt didn't fail for lack of a tool. It failed for lack of these. The organisations that get AI right will be the ones that treat them as non-negotiable, before anything goes live.
Sources: Report of the Royal Commission into the Robodebt Scheme (Commissioner Catherine Holmes AC SC, 7 July 2023); Gordon Legal class action settlement (2020), approved by the Federal Court (2021). Written respectfully, with no intent to revisit harm done to the people affected.
Krish Mootoosamy is the founder of GatedFlow.