An artificial intelligence system used by the Department for Work and Pensions (DWP) to detect welfare fraud is showing bias according to people’s age and disability, it has been revealed by the Guardian newspaper.
An internal assessment of a machine-learning programme used to vet thousands of claims across England found it incorrectly selected people from some groups more than others when recommending whom to investigate for possible fraud.
The admission was made in documents released under the Freedom of Information Act by the Department for Work and Pensions (DWP). The “statistically significant outcome disparity” emerged in a “fairness analysis” of the automated system for universal credit advances carried out this year.
The emergence of the bias comes after the DWP this summer claimed the AI system “does not present any immediate concerns of discrimination, unfair treatment or detrimental impact on customers”.
Campaigners responded by accusing the government of a “hurt first, fix later” policy and called on ministers to be more open about which groups were likely to be wrongly suspected by the algorithm of trying to cheat the system.
“It is clear that in a vast majority of cases the DWP did not assess whether their automated processes risked unfairly targeting marginalised groups,” said Caroline Selman, senior research fellow at the Public Law Project, which first obtained the analysis.
“DWP must put an end to this ‘hurt first, fix later’ approach and stop rolling out tools when it is not able to properly understand the risk of harm they represent.”
By one independent count, there are at least 55 automated tools being used by public authorities in the UK potentially affecting decisions about millions of people, although the government’s own register includes only nine.
Description of cartoon for those using screen reading software
Alongside of a sign that reads ‘Department for Work and Pensions (DWP) – Disabled Claimants AI Scrutiny’, a large robot is sat at a table working through a pile of PIP claim forms from disabled people. It has marked every form with a large red ‘Rejected’ stamp. The robot has a face mask bearing the likeness of Iain Duncan Smith, former Tory minister for the DWP. A voice from the side is saying: “It’s been a lot happier since we gave it the personality of Duncan Smith!”
Crippen Cartoons 