Throughout the organizations the place this method has emerged and began to be utilized, step one is shifting the unit of research.
For instance, in a single UK hospital system within the interval 2021–2024, the query expanded from whether or not a medical AI utility improves diagnostic accuracy to how the presence of AI inside the hospital’s multidisciplinary groups impacts not solely accuracy but in addition coordination and deliberation. The hospital particularly assessed coordination and deliberation in human groups utilizing and never utilizing AI. A number of stakeholders (inside and out of doors the hospital) selected metrics like how AI influences collective reasoning, whether or not it surfaces missed issues, whether or not it strengthens or weakens coordination, and whether or not it adjustments established danger and compliance practices.
This shift is key. It issues lots in high-stakes contexts the place system-level results matter greater than task-level accuracy. It additionally issues for the economic system. It could assist recalibrate inflated expectations of sweeping productiveness good points which can be to this point predicated largely on the promise of enhancing particular person process efficiency.
As soon as that basis is about, HAIC benchmarking can start to tackle the ingredient of time.
At this time’s benchmarks resemble college exams—one-off, standardized checks of accuracy. However actual skilled competence is assessed otherwise. Junior docs and attorneys are evaluated constantly inside actual workflows, below supervision, with suggestions loops and accountability constructions. Efficiency is judged over time and in a particular context, as a result of competence is relational. If AI programs are supposed to function alongside professionals, their impression needs to be judged longitudinally, reflecting how efficiency unfolds over repeated interactions.
I noticed this side of HAIC utilized in one in all my humanitarian-sector case research. Over 18 months, an AI system was evaluated inside actual workflows, with explicit consideration to how detectable its errors had been—that’s, how simply human groups might determine and proper them. This long-term “report of error detectability” meant the organizations concerned might design and check context-specific guardrails to advertise belief within the system, regardless of the inevitability of occasional AI errors.
An extended time horizon additionally makes seen the system-level penalties that short-term benchmarks miss. An AI utility might outperform a single physician on a slim diagnostic process but fail to enhance multidisciplinary decision-making. Worse, it might introduce systemic distortions: anchoring groups too early in believable however incomplete solutions, including to folks’s cognitive workloads, or producing downstream inefficiencies that offset any velocity or effectivity good points on the level of the AI’s use. These knock-on results—typically invisible to present benchmarks—are central to understanding actual impression.
The HAIC method, admittedly guarantees to make benchmarking extra advanced, resource-intensive, and tougher to standardize. However persevering with to judge AI in sanitized situations indifferent from the world of labor will depart us misunderstanding what it actually can and can’t do for us. To deploy AI responsibly in real-world settings, we should measure what really issues: not simply what a mannequin can do alone, however what it permits—or undermines—when people and groups in the actual world work with it.
Angela Aristidou is a professor at College Faculty London and a college fellow on the Stanford Digital Economic system Lab and the Stanford Human-Centered AI Institute. She speaks, writes, and advises concerning the real-life deployment of artificial-intelligence instruments for public good.
