Your hospital is using a risk-stratification algorithm that has never been independently assessed for your patient population.
71% of US hospitals use EHR-integrated predictive AI. Only 57% evaluate all or most models for demographic bias. (Chang et al., 2025 — ASTP/ONC Data Brief No. 80)
In 2019, Obermeyer et al. published in Science what is now the definitive case study in clinical AI governance failure: a commercial algorithm used for approximately 200 million patients to identify high-risk candidates for care management was using healthcare cost as a proxy for health need. Because structural barriers reduce Black patients' healthcare spending relative to white patients with equivalent illness severity, the algorithm systematically underestimated Black patients' needs. The verbatim finding: remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7% to 46.5%.
The algorithm was not designed to discriminate. It was commercially deployed, widely used, and embedded in the EHR infrastructure of health systems across the country. Its bias was detectable — through demographic stratification of outcomes data — and was not detected because the governance infrastructure to detect it did not exist.
Six years later, the ONC's 2025 hospital AI governance survey found that 71% of hospitals now use EHR-integrated predictive AI, while only 57% evaluate all or most models for demographic bias. The governance gap that allowed the Obermeyer algorithm to operate at scale is the governance gap that most US hospitals still have today.
The governance question goes beyond whether your AI tools have been FDA-cleared or commercially validated. What matters is whether your institution has established its own demographic impact assessment standard, specific to your patient population, applied before and after deployment, independent of what any vendor or regulator reports. Most have not.
M11 governance requires: documented AI tool inventory, independent demographic impact assessment before adoption and annually post-adoption, human override protocols without coercive documentation burden, and defined disparity thresholds that trigger governance response.