AI Governance in Healthcare: The Risk Isn’t Just the Algorithm

Why Healthcare Leaders Need to Govern the Model, the Data, and the Organization Around It

When healthcare organizations evaluate artificial intelligence, one of the first questions is often, “How accurate is it?” That is an important question, but after more than 40 years in healthcare compliance and as I continue my work in AI leadership and governance, I believe it is only the beginning. Some of the most significant risks associated with AI may not originate in the model itself. They can arise from the data that feeds the system, the vendor that provides it, the employees who use it, the decisions it influences, the systems it connects to, or the absence of clear accountability when something goes wrong.

This is familiar territory for healthcare compliance professionals. We have never evaluated compliance risk by examining only one component of a process. We look at the broader control environment: the people, processes, systems, data, policies, controls, oversight, and accountability that collectively determine whether an activity is operating appropriately. AI should be evaluated with the same discipline. Looking only at model performance can create a false sense of security when the larger environment surrounding the model has not been adequately assessed.

I believe healthcare leaders should think about AI risk across three interconnected areas: the model and its outputs, the data and security environment, and the organization surrounding the technology. Those areas cannot be evaluated independently because weaknesses in one can change the risk associated with the others. A highly accurate model can still create significant exposure if it is trained or supplied with inappropriate information, given excessive access, deployed into a poorly controlled workflow, or used by people who do not understand its limitations.

The question, therefore, is not simply whether the AI works. The question is whether the entire environment in which the AI operates is governed appropriately.

AI Model Risk in Healthcare: Can We Trust What AI Is Telling Us?

The model is where most AI risk discussions begin, and for good reason. Healthcare organizations need to understand whether an AI system produces accurate and reliable outputs, whether it can hallucinate or fabricate information, whether bias may affect its results, and whether consequential recommendations can be sufficiently explained or validated. Those questions become more important as AI moves into workflows where an inaccurate output can affect coding, reimbursement, compliance, patient communication, clinical decisions, or other high-consequence activities.

But I believe healthcare leaders need to ask another question that is less technical and potentially just as important: Are people becoming too comfortable trusting the technology?

AI can produce an incorrect answer that looks remarkably authoritative. A polished response, sophisticated terminology, or confident recommendation does not make the underlying information correct. This creates a particular challenge in healthcare because professionals are accustomed to working with authoritative sources and established reference materials. When an AI system presents an answer in a highly fluent manner, users may unconsciously treat the presentation of the answer as evidence of its validity.

Consider a compliance professional asking an AI system whether a particular service meets Medicare requirements. The response may appear complete and convincing, yet the underlying guidance could be outdated, applicable to a different payer or provider type, or irrelevant to the jurisdiction or date of service in question. The problem is not necessarily that the model intentionally produced misinformation. The problem is that the user may not recognize that the answer requires independent verification.

That is why one of my most basic rules remains: Show me the source.

For consequential healthcare decisions, accuracy and explainability need to work together. We need to know what supports the recommendation, whether the source is authoritative and current, whether it applies to the circumstances being evaluated, and whether a qualified human can verify the conclusion. The more consequential the decision, the less acceptable it becomes to rely on an answer simply because the AI delivered it confidently.

Healthcare AI Data Security: What Is the AI Accessing?

Model performance tells us only part of the story. Healthcare AI may interact with some of the most sensitive information an organization possesses, including patient information, claims, clinical documentation, employee information, financial data, contracts, internal investigations, and proprietary information. The sensitivity of that information means that healthcare organizations need to understand not only what the AI produces, but what information the system can access and what happens to that information throughout the AI lifecycle.

The questions need to extend beyond traditional access controls. What information enters the system? Where does that information go? Who can access it? How long is it retained? Is it used to train or improve a model? Can it leave the organization's controlled environment? What happens when the AI connects to another system? What information can be retrieved, transferred, or acted upon as a result of that connection?

These questions become even more significant as organizations move toward agentic AI. There is a meaningful difference between an AI assistant that reads information and provides a response and an AI agent capable of accessing multiple systems, retrieving information, moving data, and taking actions within a workflow. The second environment creates additional opportunities for both value and risk because the system has a greater ability to affect the surrounding environment.

More capability creates a different control environment.

That does not mean organizations should avoid more capable AI. It means the controls need to reflect what the technology is actually capable of doing. An organization cannot govern an AI agent effectively using the same assumptions it applied to a simple information-retrieval tool. As access, connectivity, and autonomy increase, the organization needs corresponding clarity around authorization, data handling, monitoring, human intervention, and accountability.

Healthcare AI Governance: Who Owns the System and the Decisions?

Ultimately, I believe this may be the most important part of the conversation.

AI does not operate in isolation. People choose it, deploy it, configure it, determine what authority it receives, decide where it can be used, establish whether its recommendations require review, and determine what happens when it fails. The technology may generate an output, but the organization remains responsible for the environment in which that output is used.

Every healthcare organization implementing AI should therefore be able to answer some basic governance questions. Who owns the AI system? Who owns the decision it influences? Who monitors its performance? Who approves changes? Who can override it? Who can restrict its authority? Who can stop it? And who is accountable when something goes wrong?

Those questions become increasingly important as responsibility moves from human decision-making toward AI-supported or AI-initiated decisions. Ownership of the technology and ownership of the decision are not necessarily the same thing. A technology department may own the application, while a clinical, coding, revenue cycle, compliance, or operational leader owns the underlying business decision. If those responsibilities are not clearly defined, the organization can end up with technology ownership without meaningful decision accountability.

If the organization cannot answer these questions clearly, it has a governance problem regardless of how sophisticated the technology may be.

AI Vendor Risk Management: The Vendor Is Part of Your Control Environment

Healthcare organizations also need to look beyond their own organizational boundaries. Increasingly, AI will enter healthcare through third-party vendors, existing enterprise applications, and technology platforms that introduce or expand AI capabilities over time. That creates an important governance consideration because an organization may be relying on technology whose underlying model, data practices, integrations, or capabilities it does not directly control.

A vendor may update its model, change how data are processed, introduce new AI functionality, add integrations, or expand what an AI agent can do. The healthcare organization may not have intentionally changed its use case, but the risk profile may nevertheless have changed. A system that was originally evaluated as an assistive tool could acquire capabilities that allow it to make recommendations, access additional information, or initiate actions.

Vendor governance must become part of AI governance.

For higher-risk applications, the contract itself may become one of the organization's most important controls. Healthcare organizations need to understand what the technology does, what data are used, how changes are communicated, what audit rights exist, how incidents are handled, what happens when the relationship ends, and how responsibility is allocated when the technology produces a consequential problem.

Vendor due diligence should therefore not be treated as a one-time procurement exercise. If the technology changes, the organization needs a mechanism for determining whether those changes require a new risk assessment, additional controls, or changes to the governance structure.

The organization may not control the vendor's technology. It does control whether and how that technology remains authorized within its own environment.

Workforce AI Risk: Do Not Forget the People Using the Technology

There is another dimension of AI risk that technical assessments can easily overlook: the workforce.

Employees can become overly dependent on AI. Professional skills can erode when people routinely defer to automated recommendations. Staff may not understand the limitations of a system or recognize when an output requires independent validation. Employees may use unauthorized AI applications when approved tools do not meet their needs. At the same time, other employees may distrust AI so strongly that they avoid appropriate uses of the technology or create workarounds that introduce their own risks.

There is also a human dimension to organizational change. Some employees may fear that AI is being introduced primarily to replace their roles. Others may feel pressure to accept AI-generated recommendations because the organization expects productivity improvements. Those concerns can affect how people interact with the technology and whether they report problems when they encounter them.

These are not simply workforce issues. They are governance issues.

AI literacy therefore needs to become part of the control environment. Employees need to understand what the technology can do, where it can fail, what information may appropriately be entered into a system, when outputs require validation, and when concerns should be escalated. Training should not simply explain how to use an AI tool. It should establish the boundaries within which the tool is authorized to be used.

A workforce that understands how to challenge AI is an important component of a controlled AI environment.

Human-in-the-Loop AI Is Not Enough

We hear frequently that high-risk AI needs a human in the loop. I agree that human oversight is essential, but I think the phrase can create a false sense of security if we do not examine what that human is actually expected and authorized to do.

A person clicking “approve” on hundreds of AI-generated recommendations is technically in the loop. That does not necessarily constitute meaningful oversight. If the reviewer lacks the expertise to identify an error, cannot see the information supporting the recommendation, does not understand the system's limitations, or has no practical ability to disagree with the output, the existence of an approval step does not create effective governance.

The more useful questions are: Which human is responsible? What expertise do they have? What information can they see? Can they challenge the output? Can they override it? When must they escalate? Do they have authority to stop the workflow? What happens when the system produces a pattern of questionable results?

This is why I prefer to think more broadly in terms of Human-in-Governance™.

Human judgment should not simply appear at one point in an automated workflow. Human authority and accountability should surround the system. People should establish the boundaries within which AI operates, determine when intervention is required, monitor whether the technology remains within those boundaries, and retain the authority to intervene when the system behaves outside expectations.

That becomes increasingly important as AI moves from Assist → Recommend → Decide → Act. The greater the autonomy, the greater the need for meaningful human governance.

AI Accuracy Does Not Equal Responsible Deployment

This may be one of the most important lessons for healthcare leadership. Imagine an AI system that is highly accurate under the conditions in which it was tested. That sounds reassuring, but technical performance alone does not tell us whether the organization has deployed the technology responsibly.

What happens if nobody monitors the errors? What happens if employees begin assuming the system is always right? What happens if errors disproportionately affect one patient population? What happens if the system accesses information it should not access? What happens if the vendor changes the model without an adequate reassessment? What happens if there is no meaningful audit trail? What happens if nobody knows who owns the decision?

And perhaps most importantly, what happens if one error is replicated across thousands of claims, records, communications, or other transactions before someone recognizes the pattern?

These questions demonstrate why technical performance does not equal responsible deployment. The model can function as designed while the organization fails to establish the surrounding controls necessary to manage its use.

AI governance therefore cannot belong exclusively to IT. Depending on the use case, the governance structure may need representation from compliance, privacy, security, legal, clinical leadership, revenue cycle, operations, IT, HR, procurement, and executive leadership. The appropriate participants will vary according to the risk, but the principle remains the same: governance should reflect the actual environment in which the technology operates.

Healthcare Compliance and AI Governance: Applying Proven Controls

The deeper I get into AI governance, the more familiar the underlying discipline becomes. Healthcare compliance professionals already know how to identify risk, assess likelihood and impact, establish ownership, develop policies, implement controls, educate staff, audit and monitor, investigate problems, perform root-cause analysis, implement corrective action, and validate whether those actions actually worked.

AI does not replace those disciplines. It makes them more important.

What changes is the speed, scale, and potential autonomy of the technology. A human may make one incorrect decision. An automated system can potentially replicate an incorrect decision across thousands of transactions before anyone identifies that a systemic problem exists. A vendor update can change the behavior of a system without the organization intentionally changing its workflow. An AI agent can potentially move information and initiate actions across systems at a speed that makes traditional retrospective review insufficient.

That means our compliance disciplines need to operate earlier and more continuously. We need to understand the technology before deployment, establish appropriate boundaries, define decision rights, monitor performance and exceptions, and have a clear process for investigating and correcting problems.

I would summarize the approach this way: Know the Model → Know the Data → Know the People → Know the Vendor → Know the Controls → Monitor the Outcome And when something goes wrong, the response should be equally disciplined: Detect → Escalate → Investigate → RCA → CAP → Validate → Reassess. That is not fundamentally different from what healthcare compliance has been doing for years. It is the application of those disciplines to a technology capable of changing quickly and operating at significant scale.

AI Governance Requires More Than Model Accuracy

As healthcare organizations move deeper into AI, we need to broaden the conversation beyond model accuracy. Accuracy remains important, but it is one component of a much larger control environment.

Healthcare leaders should ask whether the system is fair, whether consequential recommendations can be explained, what information it accesses, how that information is protected, who owns the system, who owns the decisions it influences, what the vendor is responsible for, whether employees have been trained to challenge AI, whether changes in system behavior can be detected, whether qualified humans can intervene, and whether decisions can be reconstructed when questions arise.

Those questions move the organization from evaluating AI performance to evaluating AI governance. We need both.

A model can be accurate and still be deployed into an environment that is poorly governed. A system can perform well under testing conditions and create unacceptable exposure when its outputs are used in a different workflow. A vendor can provide a capable technology while the organization fails to establish clear ownership. Employees can follow the approved process while still misunderstanding when an AI output requires independent validation.

The risk, in other words, does not live exclusively inside the algorithm. It exists across the ecosystem we build around it.

That is why responsible AI governance requires us to look at the model, the data, the vendor, the workforce, the workflow, the controls, and the people accountable for the decisions. If we evaluate only the algorithm, we are evaluating only one part of the risk.

For healthcare leaders, the more important question is not simply whether the AI works. It is whether the organization has built an environment capable of using it responsibly, recognizing when it fails, and acting when the risk changes.

That is the standard we should be working toward.