To measure AI's impact, ask who is better or worse off
Usage figures show if an AI tool is used, not if it helps the user. At our 29 October 2026 Open Track webinar, four Fellows of the Emerging Leaders in Ethical AI Governance Fellowship argued that AI's impact should be judged in terms of outcomes for the people affected, including those that the system leaves out. Their advice: involve communities early, measure harms as well as benefits, and name who is accountable when things go wrong.
In 2021, the Dutch cabinet resigned after a scandal involving an automated risk model.
From 2013, the tax authority used the model to select childcare benefit claims for checks. It was judged on whether it detected fraud. The families wrongly flagged were not measured. Tens of thousands of parents were affected.
Joel Christoph, founder of 10Billion.org, draws a simple lesson from this case: if only the benefit is measured, the harm can go unseen.
Usage is not impact
Billions of dollars have gone into "AI for Good". Yet the figures most often reported, such as usage and time spent, are simply the easiest to collect. They do not show whether people are better or worse off.
Joel suggests asking affected communities which mistakes would hurt them most, then testing for those before launching an AI tool. He also recommends tracking how long it takes to correct a mistake. A fix in a day is very different from a fix in a year.
Good performance can hide uneven results
Nancy Muriithi, a PhD candidate in AI ethics in medicine at the University of Patras, notes that a system can work well technically and still fall short socially.
Take an AI hiring tool that screens out people with informal work experience. If only the candidates who pass are evaluated, the results look excellent. The people rejected never become part of the evidence.
Her test for any claim of social benefit: whose life is improved, compared to what, and at whose expense?
Easy-to-measure AI impact figures need a companion question:
What is easy to measure + What to ask as well
Usage and time spent + Are affected people better or worse off?
Candidates who pass a screen + What happened to those who were rejected?
Fraud detected + How many people were wrongly flagged?
Average accuracy + Which groups are missing from the data?
Time saved + Who bears the cost of errors, and for how long?
Numbers help, but they can mislead
Meem Arafat Manab, a doctoral researcher at Universidad Politécnica de Madrid, warns that any single metric invites gaming. A fairness threshold, such as a minimum selection rate in hiring, is easy to check. But a number can hide differences within groups, and large figures can drown out small communities. A group of five people still matters.
Accountability needs names and timeframes
A human in the loop is not enough on its own, Joel argues. It must be clear which person or office is responsible for fixing a problem, and by when.
Nancy adds a question to ask before launching any AI tool: what evidence would make us reconsider deploying this system? If nobody can answer, monitoring is just a tick box exercise.
The takeaway
Efficiency matters, but within limits. As Meem asked, if we accept speed limits on cars, why not on AI? Counting AI use is easy. The real measure of impact is who is helped, and who is harmed.
About this post: Based on the Open Track webinar "Impact measurement in the age of AI", moderated by Addie Achan, founder of The Kaleidoscope Project. Learn more about the Open Track webinar series. Watch the full webinar on YouTube.