By Nathan Donaldson
Imagine a debt notice arrives in the post. It says you owe Centrelink several thousand dollars. The number on the page comes from a calculation a computer did. Nobody at the agency picked your case off a desk and looked at it. Nobody can tell you which rule you broke, only that the rule found you. If you want to argue with the number, the burden is on you to prove it wrong.
That is roughly what happened to a lot of Australians between 2015 and 2019. A system called the Online Compliance Intervention, which most people now know as Robodebt, was raising debts straight from averaged tax-office data. The Federal Court found in 2019 that raising a debt that way was unlawful. The class action settled for one-and-a-bit billion dollars. The Royal Commission report landed in 2023 and called the scheme “a crude and cruel mechanism, neither fair nor legal.” Two mothers gave evidence about their sons’ suicides. The Commission noted a third tragic death linked to a 2017 discrepancy letter.
The thing I want to point at is not the automation. It is what was missing around the automation.
There was no published rule that a citizen could read to see how their debt was calculated. There was no individual explanation in the letter that arrived. There was no real human gate before recovery action started. There was no audit record an outsider could inspect to ask whether the rule the computer ran actually matched the law. The technical capacity to do the match arrived years before the oversight scaffolding that would have made the resulting decisions legitimate.
That gap is what I want to talk about today.
At Boost we have been thinking about Agentic Government as a 5-layer model. Layer 1 is the foundation, the identity, data exchange and compute that everything else sits on. Layers 2, 3 and 4 are where agents do most of the visible work: moving cases between agencies, talking to citizens, doing the work itself. Layer 5 is the one that hardly anyone leads with.
Layer 5 is oversight and governance. Audit logs. Agent registers. The rule written in a way a computer can run, which is also the way an auditor or a tribunal can read it back. A human-in-the-loop where the consequences earn it. A real right of appeal. An explanation a normal person can understand.
The model is architectural, not a maturity ladder. A serious Agentic Government build needs all five. The reason layer 5 is the binding constraint, I think, is not that automation is dangerous. It is that the failure cases in this field, the ones with names and royal commissions attached, are layer-5 failures. The automation was the easy part. The audit trail showed up late.
Robodebt is the cleanest case to point at because the rule it was running was a set rule, not a learning model. A simple averaging calculation, the kind any junior developer could write. The catastrophe was not the rule. The catastrophe was that the rule had no statutory basis, no published explanation, no inspectable trail, no real contest path, and an Ombudsman who was actively obstructed when she went looking.
The Netherlands has a related story from the same decade. Their childcare-benefits affair, the toeslagenaffaire, ran from 2005 to 2019. The tax administration used a risk-scoring model to flag families for fraud investigation. Nationality was one of the inputs. The model’s design was not published. There was no individual explanation for the families flagged. About 26,000 families were wrongly accused. The third Rutte cabinet resigned in full over it in January 2021.
Different country, different decade, different kind of model. A learning model in the Dutch case, a set-rules calculator in the Australian one. Same shape of failure. No inspectable rule-trail. No individual explanation. No proportionate contest path. No real human gate before recovery action started. The kind of automation matters for the design of layer 5, but not for whether you need it.
The other half of the story is the one that does not get told as often. Canada has had a Directive on Automated Decision-Making since 2019. Before any federal department deploys an automated decision system, it has to fill out an Algorithmic Impact Assessment. The form scores the system into four impact tiers. The higher the tier, the more the department has to do: published plain-language description of the decision, public posting of the assessment, mandatory notice to the affected person, access to human review, peer review of the system design, head-of-agency sign-off at the top tier. Seven years in, I haven’t found a Canadian-federal Robodebt-class case on the public record.
The UK has gone two ways at once on the same problem. On disclosure it has tightened. By 2025, the UK had made ATRS publication mandatory for central government departments and specified arm’s-length bodies. The Algorithmic Transparency Recording Standard sits on a public hub on gov.uk. The record names the tool, the owner, the data it uses, how humans oversee it, and how to contest it. More than 125 records have been published so far. The OECD points to it as world-leading.
On the permission underneath, it has loosened. On 5 February 2026, section 80 of the UK’s Data (Use and Access) Act 2025 came into force. It took Article 22 out of the UK GDPR and put Articles 22A to 22D in its place. Article 22 had held back decisions made entirely by a computer where the consequences for the person are serious. Under the new articles that kind of decision is allowed, as long as safeguards are in place, where the data is not special-category data. For special-category data the tighter restriction stays.
The bill that would have made the duties binding went nowhere. The Public Authority Algorithmic and Automated Decision-Making Systems Bill passed every stage in the House of Lords by 7 February 2025. Then it stopped. It never got its first reading in the Commons, and it did not become law. So the transparency register moved forward in the same period that the binding-duties bill was left to expire.
Estonia is the example at the foundation layer. X-Road has been running since 2001. Every exchange of personal data between Estonian agencies is authenticated, encrypted, timestamped, signed and logged. Citizens can see in the e-Estonia portal which officials looked at their records and why. Unauthorised access is a criminal offence. Twenty-five years on, no comparable accountability blow-up has surfaced at the data-exchange layer that I can find. The lesson there is structural. Layer 5 is cheap when it is built into the foundation. It is expensive and brittle when it is bolted on after.
The common thread across these three is that the oversight design was specified before, or in lockstep with, the automation. Not after.
Here is the constructive bit. The rules-based cases that failed, Robodebt, Michigan’s unemployment-fraud system, the Arkansas Medicaid algorithm, all failed in the same way: the rule was kept beyond inspection. A 24-line eligibility rule expressed in code is more inspectable than a discretionary human queue with no record of which criteria each officer applied. Sweden’s Trelleborg municipality automated simple-case social-assistance decisions in 2017, and turnaround on those decisions improved materially.
That is the constructive limb of the layer-5 argument. Rules-based automated decisions are the easy case to get right first, because set rules produce an explicit, publishable rule-trail. The same automation that critics worry about can be more transparent than the human queue it replaces. But only if layer 5 turns that latent auditability into something a citizen, an auditor or a review tribunal can actually see and contest.
The harder case is the learning-model side, the kind of system the Netherlands had, the kind Denmark’s Udbetaling Danmark agency is currently running. Amnesty International reported in November 2024 that the agency operates more than sixty machine-learning models to flag suspected benefit fraud. The agency says human caseworkers always review flagged cases. Amnesty’s point is that the human-review claim is not externally auditable. That argument, the one over whether the human-in-the-loop is real or asserted, is the layer-5 argument that the next decade of this work will keep landing on.
The EU has legislated for the learning-model side. The AI Act classifies systems that decide eligibility for essential public benefits as high risk. Those requirements were first due to apply from 2 August 2026. That date has since moved: Regulation (EU) 2026/1744, adopted on 8 July 2026 and in force from 27 July 2026, amended the Act, and for systems like these the requirements now apply from 2 December 2027. Either way, the structure of what the Act requires for those systems, the documentation, the logging, the human oversight, the conformity assessment, the EU register, is recognisably a layer-5 design pattern at supranational scale.
New Zealand has had this law for three years. Subpart 5A of the Social Security Act 2018 came into force on 1 July 2023, and that is where the authority to make automated social-security decisions comes from. The Social Security (Modernisation) Amendment Act 2026 passed on 28 May 2026, and it widens that existing power rather than creating a new one: it extends the scope to an “administrative programme”, which means social assistance that does not sit in or under legislation. MSD has said the automation it uses is rules-based, not generative AI. That is MSD’s own choice rather than something the statute requires, because nothing in Subpart 5A limits the technology. The Minister has been clear that human judgement remains where it is needed.
The safeguards came with the power, in the same subpart. A person keeps the right to appeal an automated decision and to have it reviewed (section 363D(1)). A human has to be available as an alternative before MSD can approve an automated system at all (section 363A(4)(d)). And the automated decision takes legal effect as the decision of a named official (section 363A(7)), so someone is accountable for it by name. That is binding law, not guidance.
The Algorithm Charter sits alongside that law rather than above it. The Charter was published in 2020 and it is voluntary. It asks signatory agencies to document the algorithm in plain English, to partner with Māori, to focus attention on high and critical-risk decisions, to keep humans in the oversight loop, to publish a clear contact point for review.
The opportunity is structural. The law is already in place, and the rules-based decisions MSD has said it will automate are the easy case to get layer 5 right on. What is left is design work rather than legislation: an inspectable rule, a human-failsafe gate proportionate to the consequence, a published register of which decisions are automated and under which rule, a contestable explanation, an audit log. The toolkit other jurisdictions have published, the Canadian impact assessment, the UK transparency register, the Estonian audit-trail model, the EU documentation standard, is a fair place to look for how to build those.
Layer 5 is the binding constraint on Agentic Government, and rules-based decisions of the kind MSD has chosen are the easier case to get right first.
Here is what would change that, measured from 29 July 2026.
I would be wrong if agents start making the decisions that go against people, on their own, at real scale rather than in a pilot. Those are the ones that count here: refusing a benefit, stopping one, turning down a visa, raising a tax debt. For that to prove me wrong it has to hold in two or more comparable jurisdictions, not one, run for at least eighteen months, and get through that time without a serious accountability failure. If it does, I am wrong about layer 5 and I will say so and revise this.
There are four things I would count as a serious accountability failure, and I am naming them now rather than when the evidence arrives, because otherwise I can move the line to suit myself. A senior court finds the system unlawful or invalid. A statutory oversight office, an ombudsman, an auditor-general, a privacy or information commissioner, finds something wrong with the system itself rather than with one case. A minister or the agency switches it off or pulls it back. Someone sets up a programme to pay a whole group of people back, or to otherwise put right what happened to them. Press criticism does not count. Nor does academic criticism, a single successful appeal, or my own opinion of it.
One jurisdiction getting that far would already be enough to change how I talk about this. It would not prove me wrong on its own. But from that point I would stop calling this a position I hold and start calling it contested, whether or not anything had gone wrong.
There is a second half to the binding-constraint claim, and it has its own test. Where oversight is mature, I think it narrows what the machine is allowed to decide rather than speeding it up. So if a jurisdiction that has built mature layer-5 oversight lets an agent decide against people on its own, at any scale at all, that half is wrong. It would not take a big deployment to show it, because the claim is that mature oversight would not permit it at all.
Two more things I will treat as signals rather than proof. If a jurisdiction changes its law to allow these decisions, and an agent is making them on its own within two years of that law starting, I will test the whole claim again. And if an agent is deciding at real scale on its own but only ever in the person’s favour, saying yes and never no, that does not prove me wrong. It does weaken what I am claiming, and I will say so here if it happens.
My best guess, on New Zealand’s side of this, is that the next two years will tell us whether the implementation joins the constructive set or the failed-in-the-dark set. The variable that decides which set it joins is layer 5. The audit trail, the inspectable rule, the contest path, the human gate where the consequences earn it. Not whether to automate. How to make the automated decision legible to the person it affects and to the auditor who comes later.
If you’ve worked on any of this, in New Zealand, Australia, Canada, or anywhere else, send me a note.
2026-07-29. Corrected: this post dated New Zealand’s power to make automated social-security decisions to the Amendment Act of May 2026, when Subpart 5A of the Social Security Act 2018 has carried that power since 1 July 2023 along with the safeguards described above; and it said the UK had tightened its oversight without saying that the permission underneath loosened on 5 February 2026.
2026-07-30. Also updated: the test above for what would prove me wrong. The old test, measured from 22 May 2026, counted any decision affecting a citizen at material scale. The new one, measured from 29 July 2026, counts only decisions that go against the person, and it says up front what I would count as a serious accountability failure. That change makes my position harder to disprove. So I am noting it here rather than making it quietly.
