AI Governance Is Becoming a Healthcare Compliance Responsibility

As AI moves deeper into healthcare operations, governance can no longer be treated as an IT issue. Compliance leaders need a seat at the table.

Artificial intelligence is moving rapidly from experimentation into everyday healthcare operations. AI is being used to summarize medical records, support documentation, assist coding, analyze claims, identify payment anomalies, communicate with patients, prioritize work, conduct research, and support clinical and administrative decisions.

We are also quickly moving beyond generative AI. Agentic AI introduces systems capable of monitoring information, coordinating multiple steps, determining what happens next, interacting with other systems, and taking defined actions with increasing levels of autonomy. That changes the risk equation.

For healthcare compliance professionals, the question is no longer simply, "Is our organization using AI?" We need to be asking, "How is AI being used, what decisions can it influence, what information can it access, what actions can it take—and how do we know appropriate controls are working?" These questions make one conclusion increasingly clear: AI governance in healthcare is rapidly becoming a compliance responsibility.

AI Governance Is Bigger Than an AI Policy

Developing an AI policy is important, but a policy is only one component of governance. An organization can have a well-written AI policy, make it electronically available to every employee, provide training on it, and still have significant AI risk.

Why? Because the real risks emerge when AI interacts with actual workflows, data, people, vendors, and decisions. Policy may establish expectations, but governance operationalizes those expectations.

Healthcare organizations therefore need operating mechanisms—not just written expectations—to determine which AI use cases are permitted, which require additional review, and which may be too risky. Those mechanisms must define what data a system may access, which decisions it may influence, what actions it may take, and when human approval is mandatory.

Operational governance must also establish how AI outputs are validated, how exceptions are escalated, how performance is monitored, who owns the risk, and what evidence is retained. Together, these mechanisms turn governance from an abstract principle into a working system. That is much closer to the work healthcare compliance professionals already perform every day.

Compliance Professionals Have Seen This Movie Before

Healthcare compliance has never been simply about writing policies. Effective compliance programs require risk assessment, education, auditing, monitoring, reporting, accountability, investigation, corrective action, and continuous improvement.

We establish and test controls, identify failures, investigate why they occurred, correct them, and document what we did. AI governance requires many of these same disciplines.

The technology may be different, but the governance questions are surprisingly familiar. Compliance teams already ask what could go wrong, how likely it is, what the impact would be, which controls should prevent it, and how the organization will know if a control fails. They also establish who is responsible and what should happen next. This is why I believe compliance leaders should be deeply involved in healthcare AI transformation.

Start With an AI Risk Inventory

Organizations cannot govern AI they do not know exists. One of the first steps should therefore be understanding where AI is already being used—and that may be harder than leadership expects.

AI capabilities are increasingly embedded inside applications employees already use, while staff may also be experimenting independently with generative AI tools.

An organization needs to know which AI systems are being used, for what purpose, by whom, and whether they were developed internally or supplied by a vendor. It should also understand what data each system can access, which decisions it can influence, whether it can take action, and whether its outputs are internal or external. Most importantly, leaders need to identify whether a use case could affect patients, claims, reimbursement, compliance, privacy, or other regulatory obligations.

This information becomes the foundation of an AI risk register. Not every use case belongs in the same risk category: using AI to help draft an internal meeting agenda is fundamentally different from using AI to recommend coding, determine medical necessity, prioritize claims for denial, or influence patient care. Governance needs to recognize those differences.

Risk Should Determine the Level of Control

I believe healthcare organizations need a tiered approach to AI risk. A lower-risk use case may require basic safeguards and routine monitoring, while a moderate-risk use case may require source validation, testing, documented human review, and periodic quality monitoring.

A high-risk use case may require formal approval, clearly defined human decision authority, enhanced validation, audit trails, escalation requirements, ongoing monitoring, and potentially legal, privacy, security, clinical, or compliance review. Some uses may simply be inappropriate.

The goal is not to make every AI workflow cumbersome. It is to apply controls proportionate to risk, a principle healthcare compliance already understands very well.

Who Owns the AI Decision?

This becomes one of the most important governance questions as AI becomes more autonomous. Consider an AI system that reviews documentation and identifies a potential overpayment.

The AI may have performed excellent analysis, but it cannot resolve the organization’s accountability. A qualified person must still determine whether an overpayment actually exists, whether repayment obligations have been triggered, whether counsel should become involved, and who may approve communication with a payer or regulator.

Those decisions cannot become ambiguous simply because AI participated in the analysis. Organizations need an accountability matrix that clearly identifies who owns the workflow and who has authority at consequential decision points.

For higher-risk AI workflows, the accountability matrix may assign the operational process to a business owner and system performance and integration to a technical owner. Compliance, privacy, and security leaders may oversee applicable controls, while a subject-matter expert validates domain-specific conclusions. An executive sponsor owns strategic accountability, and a designated human decision-maker retains authority over defined consequential decisions.

Without that clarity, AI can create an accountability gap where everyone participated but no one truly owns the outcome.

Human-in-Governance™ Must Be Designed Into the System

This is where I believe healthcare needs to move beyond the traditional concept of "human-in-the-loop." Having a person review an AI-generated output is important, but governance requires more than review.

My concept of Human-in-Governance™ begins with the premise that humans establish the rules under which AI operates.

Qualified professionals determine what AI is authorized and prohibited from doing, which sources it may use, and which decisions remain human-owned. They define the risk thresholds that trigger escalation, the evidence required to support an output, the actions that require approval, and the circumstances in which the workflow must stop. They also establish who remains accountable for the outcome.

The human does not simply review the AI. The human governs the environment in which AI operates.

Some Controls Should Monitor. Others Should Block.

This distinction will become increasingly important with agentic AI. Traditional compliance monitoring frequently identifies something after it occurs, but agentic systems give us an opportunity to build some governance directly into the workflow.

A monitoring control might flag unusual activity, declining accuracy, repeated overrides, or emerging patterns for later review. A blocking control, by contrast, prevents the workflow from proceeding.

For example, an AI-enabled compliance system might be prohibited from moving forward when an authoritative source cannot be verified, required information is missing, or two authoritative sources conflict. The same blocking control should apply when confidence falls below an established threshold, a recommendation lacks supporting evidence, the requested action exceeds the agent’s authority, or required human approval has not occurred.

In those situations, the appropriate response is stop, escalate, and require review. That is not AI failing; that is governance functioning as designed.

The Guardian Agent: Compliance Built Into the Workflow

Agentic AI also introduces an interesting opportunity for compliance professionals: the concept of a Guardian Agent. A Guardian Agent acts as a quality and governance layer across an AI-enabled workflow by verifying that authoritative sources are traceable, required information is present, and conclusions are supported by evidence. It can also confirm that human approvals occurred, the system remained within its authorized scope, exceptions were properly escalated, and required records were retained.

The Guardian does not replace compliance oversight. It helps operationalize the controls compliance has established. This is an important distinction: AI should not govern itself. Humans establish the standards; technology can help enforce them.

Vendor Governance Will Become Critical

Many healthcare organizations will not build their own AI systems; they will buy them. Increasingly, AI will also appear as a feature inside existing technology, making vendor governance critical.

Healthcare organizations should understand much more than whether a product simply "uses AI." Their due diligence should identify the model being used, the data being processed, whether organizational data trains the model, where information is stored, and which subcontractors or third parties are involved. Leaders should also understand how access is controlled, how outputs are validated, how frequently the underlying model changes, and how material changes are communicated.

The review must go further by examining what monitoring the vendor performs, whether the organization can obtain sufficient logs and evidence, and what happens when the system fails. The organization should know whether it can override or disable AI functionality and, ultimately, whether it can explain and defend decisions made with the technology. Answering these questions requires AI vendor due diligence to evolve beyond traditional technology procurement.

Auditability Will Matter

Healthcare organizations should assume that consequential AI-assisted decisions may eventually be questioned—perhaps by an internal auditor, payer, regulator, attorney, patient, board, or client. The organization should be able to reconstruct what occurred.

For significant AI-assisted activities, I believe we should strive to maintain a traceable chain:

Source → Data → AI Analysis → Controls → Human Review → Decision → Action

This is particularly important in compliance. If AI contributed to a coding recommendation, audit finding, reimbursement decision, regulatory interpretation, or corrective action, the organization may need to explain how that conclusion was reached.

The standard must be higher because "That's what the AI said" will never be an acceptable compliance defense.

AI Governance Needs Continuous Monitoring

AI governance cannot be a one-time implementation exercise. Models, vendors, regulations, workflows, employee use cases, and organizational risks all change, while AI itself continues to evolve at extraordinary speed.

Organizations therefore need a monitoring cadence that reviews AI incidents, human overrides, escalations, accuracy and quality measures, unsupported outputs, and privacy or security events. That review should also capture new use cases, vendor changes, user adoption, complaints, control failures, and changes in applicable regulation. This information must feed back into the program because governance must evolve as the technology evolves.

Compliance Needs a Seat at the AI Table Now

Healthcare compliance professionals should not wait until AI systems have already been purchased and implemented to become involved. By then, some of the most important decisions may already have been made.

Compliance should participate in AI strategy, use-case evaluation, risk classification, vendor assessment, and workflow and control design. Its role should also extend to human oversight requirements, monitoring, incident response, training, and board reporting. That does not mean compliance should own AI transformation; it means compliance must be one of the disciplines shaping it.

AI transformation requires collaboration among technology, cybersecurity, privacy, legal, clinical, operational, financial, data, and compliance leaders. No single department can govern this alone.

The ProCode Perspective

Healthcare compliance has always been about more than identifying what the regulations say. Our responsibility is helping organizations translate requirements and risks into systems that work in the real world, and AI presents the same challenge at a much faster pace.

The organizations that succeed will not simply have AI policies. They will know where AI is operating, what risks it creates, who owns those risks, and which decisions remain human-owned. They will understand what controls are operating, how exceptions are escalated, and how performance is monitored. Most importantly, they will be able to produce evidence that their governance actually works.