AI fraud detection by government: the lessons after SyRI

Short answer: AI fraud detection by government is high-risk under the AI Act and also falls under the GDPR and administrative law. Dutch history — the SyRI ruling of the District Court of The Hague in 2020 and the childcare-benefits scandal — shows what goes wrong without transparency and proportionality. The lesson is not "no AI", but: no opaque risk scores, no proxy discrimination, and instead explainability, proportionality and a fundamental rights assessment up front.

What SyRI made clear

In the SyRI case the District Court of The Hague ruled in 2020 that the System Risk Indication — a tool that linked databases to score citizens for fraud — breached Article 8 ECHR (right to private life). The court found the interference insufficiently transparent and not proportionate: citizens could not know whether or why they were analysed, and the safeguards fell short. SyRI was struck down. That is the anchor: opacity is in itself a fundamental rights problem.

The benefits scandal as a second lesson

The Dutch childcare-benefits scandal showed the human consequences of risk selection without proper safeguards: wrongful fraud accusations, a reversed burden of proof and discriminatory risk indicators (including nationality). The pattern recurs: a model flags, an implementing agency follows blindly, and the citizen bears the burden of an inscrutable judgment. Automation bias and the absence of meaningful human review made it structural.

Why the AI Act now regulates this

Under the AI Act, AI that assesses access to public benefits and essential services (Annex III, point 5) or that assesses risks of individuals in law enforcement (point 6) is high-risk — the categories government fraud detection usually falls under — bringing the full high-risk obligations: risk management, data quality and bias mitigation, logging, technical documentation and human oversight. Some variants also touch the prohibited AI practices, such as general social scoring. The AI Act thereby codifies precisely the safeguards that SyRI lacked.

Proxy discrimination is the core risk

The subtlest danger is indirect discrimination through proxies: an attribute such as postcode, language background or nationality that functions as a prohibited ground without naming it. Bias testing on training data and on outcomes is not a nice-to-have but the core of lawful fraud detection. Data quality and representativeness (AI Act Art. 10) sit in direct line with the prohibition of discrimination.

What to do

• Test proportionality up front: does the fraud interest outweigh the interference, and can it be done less intrusively? Run a FRIA.
• Eliminate proxy discrimination: analyse training data and outcomes for indirect discrimination, repeated over time.
• Make risk scores explainable: no black box; the citizen must be able to understand and contest — see explainability of government algorithms.
• Secure meaningful human oversight: no blind following of a flag; keep the burden of proof with the government.
• Register the system in the algorithm register and see AI in the public sector.

SyRI and the benefits scandal are not a footnote but the benchmark. Whoever deploys AI against fraud must first show that the tool itself does the citizen no injustice.