An algorithm used for 200 million patients was producing systematic racial bias. The governance infrastructure to detect it did not exist. Seven years later, most US hospitals still don't have that governance infrastructure.
Obermeyer et al. (2019) published in Science the defining case study in clinical AI governance: a commercial algorithm used for approximately 200 million US patients to identify high-risk candidates for care management used healthcare cost as a proxy for health need. Because structural barriers reduce Black patients' healthcare spending relative to white patients with equivalent illness, the algorithm systematically underestimated Black patients' needs. The verbatim governance finding: remedying the disparity would raise Black patients receiving extra help from 17.7% to 46.5%. The ASTP/ONC Data Brief No. 80 (Chang et al., 2025) documents that 71% of US hospitals now use EHR-integrated predictive AI while fewer than half evaluate all models for demographic bias.
~200 million patients. Remedying disparity: 17.7% → 46.5% Black patients receiving help. Algorithm used cost as proxy for need.
71% of hospitals use EHR-integrated predictive AI. Fewer than half evaluate all models for bias. 74% have distributed accountability.
AI dermatology tools underperform on darker-skin tones. Demographic performance disparities are systematic and recurring.
The prototype organizes reflection across six proposed governance dimensions. It does not produce a validated institutional score, audit, accreditation, or certification.
Complete AI tool inventory publicly available; internal evaluation standard independent of vendor and regulatory clearance; demographic impact assessment before adoption and annually post-adoption, stratified for the specific patient population; override protocol with no coercive burden; patient notification standard; monitoring plan with defined disparity thresholds triggering suspension.
The full tier operationalization — PIONEERING through EARLY STAGE, with specific observable criteria for all six dimensions — is documented in the Framework Foundation.
Health system CMOs and CIOs · Clinical AI governance committees · Healthcare compliance officers · EHR implementation teams · Any healthcare institution using predictive AI in clinical decisions
And any patient whose care is shaped by an algorithm they will never see.
Obermeyer et al. (2019) demonstrated that an AI algorithm used for 200 million patients was systematically producing racial bias because it used healthcare cost as a proxy for health need. The governance failure was post-adoption: no monitoring, no demographic impact assessment, no override protocol. In deep space, where communication delays make Earth-based clinical consultation impractical or impossible, the AI tools making autonomous clinical decision support recommendations become life-critical. The M11 governance dimensions — demographic impact assessment, human override protocols, post-adoption monitoring with suspension thresholds — function in space as mission safety infrastructure.
Review the module scope and proposed dimensions. The assessment remains an educational prototype; no email report or institutional finding is active.