New from me @gabrielgeiger.bsky.social + Justin-Casimir Braun: Amsterdam believed that it could build a #predictiveAI for welfare fraud that would ALSO be fair, unbiased, & a positive case study for #ResponsibleAI. It didn't work. Our deep dive why: www.technologyreview.com/2025/06/11/1... - ThreadSky

eileenguo.bsky.social • 4 days ago

New from me @gabrielgeiger.bsky.social + Justin-Casimir Braun:

Amsterdam believed that it could build a #predictiveAI for welfare fraud that would ALSO be fair, unbiased, & a positive case study for #ResponsibleAI. It didn't work.

Our deep dive why: https://www.technologyreview.com/2025/06/11/1118233/amsterdam-fair-welfare-ai-discriminatory-algorithms-failure/

Comments

eileenguo.bsky.social•4 days ago

Thread from our reporting partner @lighthousereports.com w/ more on the story:

https://bsky.app/profile/lighthousereports.com/post/3lrdjsudvtc2m

kennypowers55.bsky.social•3 days ago

Ai is a glorified google search engine.

mikeyearworth.bsky.social•3 days ago

Admirable openness but not exactly surprising - the problem is believing that technology has a way of avoiding with the hard problem of deciding what is fair. You can’t delegate that to AI.

deblavoy.bsky.social•4 days ago

“it represented a substantial risk to welfare recipients’ rights that no amount of technical tweaking could fix. As this algorithmic experiment unfolded over several years, it called into question the… central premise: that responsible AI can…actually make algorithmic systems fair in the real world.

joeyvazquez.bsky.social•3 days ago

Anti-AI folks love to put up quotes about AI models failing to meet extreme standards - such as perfect fairness in benefits administration - but they never put up the quotes about how AI systems did in comparison to the human systems they were trying to replace.

cemhend.bsky.social•3 days ago

Ok, here’s one: “While it had been designed to reduce the number of welfare applicants flagged for investigation, it was flagging more. And it proved no better than a human caseworker at identifying those that actually warranted extra scrutiny.”

deblavoy.bsky.social•3 days ago

Algorithms are institutionalized narratives. Garbage in, garbage out. But humans can evolve the narrative (criteria) and be held accountable. We don’t know how to do that with machines in the same way. In other words what’s interesting here is that we don’t know what the right answer looks like.

deblavoy.bsky.social•3 days ago

https://www.stonybrook.edu/commcms/wicked-problem/about/What-is-a-wicked-problem
And wicked problems are not the same as hard problems (cure cancer) where you know what the solution is.

deblavoy.bsky.social•3 days ago

It’s really exciting to think that LLMs can solve problems non-algorithmically, but in the case where we cannot even describe the problem let alone the answer, it may not be wise to give it over to statistics. At least not without ample evidence that it at least helps.

deblavoy.bsky.social•3 days ago

It’s a “wicked” problem - not just a hard problem.

joeyvazquez.bsky.social•3 days ago

Now that's a narrative.

joeyvazquez.bsky.social•3 days ago

Again, your judgment criteria is fundamentally flawed in that it creates bars that no technology should even try to meet.

AI models used in a system like welfare should never be about 100% removing human accountability but rather about enhancing outcomes in tandem with humans.

deblavoy.bsky.social•3 days ago

Yes - but that’s not the issue. The judgement criteria is the problem. The acknowledged problem. We don’t agree on what they should be. We do not know how to solve THAT problem. Until we do, the machine isn’t helping except to the extent that it can help on that Q.

deblavoy.bsky.social•3 days ago

Sort of. Im saying that ‘wicked’ problems are inherently different from ‘hard’ problems and as such make it nearly impossible for us to use tech to solve them - because we don’t know what ‘solving’ means.

joeyvazquez.bsky.social•3 days ago

Well, the first thing that jumps out is poor project design or poor execution.

Seems unwise to predetermine a benchmark for how many applicants should or should not be flagged for investigation.

But assuming they had a valid reason to drive towards that goal, then their system design was flawed.

joeyvazquez.bsky.social•3 days ago

See, what I find interesting about this is everybody's just assuming that those folks knew what they were doing.

Fact is, the technology is too new and complicated to assume anybody knows precisely what they're doing with it - Even the AI companies admit they don't fully understand it yet.

joeyvazquez.bsky.social•3 days ago

So just saying the system flagged more applicants than they wanted doesn't inherently mean anything about the system or tech. Did it rightfully flag those applicants? Was the benchmark erroneous? Did the AI fail it's design or was the design always flawed due to chasing a predetermined benchmark?

joeyvazquez.bsky.social•3 days ago

"In abandoning the model, the city has returned to an analog process that its own analysis concluded was biased against women and Dutch nationals... (and displayed) many of the concerns raised about the complex, layered biases within the AI model also apply to the caseworker-led process."

jeremykun.com•3 days ago

Yes but the humans can be held responsible for bad outcomes.

joeyvazquez.bsky.social•3 days ago

I don't see the "but" here unless you're perceived goal for the system was to remove all accountability for welfare application decisions from humans.

If that was the goal, then the project was doomed from the jump, right?

You can't remove a human accountability from a system like welfare.

jeremykun.com•3 days ago

I can't assume that it was a goal or not, but it is a difference from an equivalent-outcome system with humans. Any automated decision making system naturally encourages laundering responsibility.

joeyvazquez.bsky.social•3 days ago

*your 😑 frickin autocorrect

joeyvazquez.bsky.social•3 days ago

But, I think you're right that removing accountability from the politicians and bureaucrats was a monumental driving force behind this project.

And if that's how that AI system was designed, then it set up for failure.

So it being scrapped isn't the big W anti-AI zealots want it to be.

rosamundevb.bsky.social•4 days ago

great investigation 👏

eileenguo.bsky.social•4 days ago

Thanks so much for reading and sharing!

joeyvazquez.bsky.social•3 days ago

Really is sad watching self-righteous creators wage myopic anti-AI campaigns to destroy a technology that holds so much promise - and has already fulfilled much of that promise - for so many in vulnerable communities.

joeyvazquez.bsky.social•3 days ago

Of course, this is what happens every time a disruptive technology comes along - those who fear it will manipulate early application mistakes into entire campaigns to abolish the technology.

They're really just as bad as the deceitful clowns who manipulatively oversell the technology.

Comments

Posting Rules

Reply