AI Governance Is a Commitment to the People AI Can Help—or Harm

Responsible AI leadership requires us to look beyond protecting the organization and ask what we owe the people affected by our technology.

For decades, healthcare compliance and risk management have understandably focused on protecting organizations. Leaders ask whether regulatory requirements are being met, policies are appropriate, controls are effective, patient information is protected, claims are submitted correctly, and decisions can be defended during an audit. Those questions remain essential, but artificial intelligence introduces another dimension that I believe healthcare leaders need to place much more prominently in the conversation.

Responsible AI governance is not only about protecting the organization from risk. It is also about protecting the people who may experience the consequences of the technology. That distinction becomes increasingly important as AI moves from assisting people with tasks to recommending decisions, coordinating workflows, and ultimately taking actions through AI agents. The more authority we give AI, the more deliberately we need to consider the people on the other side of those actions.

Good Governance Is a Visible Commitment

One of the most important ideas in the stakeholder framework I have been studying is this: Good governance is a visible commitment to the people a system can help or harm. I think that is a powerful way for healthcare leaders to think about AI because it changes governance from something that exists primarily to satisfy a requirement into something that demonstrates how an organization intends to exercise responsibility.

Governance should not exist simply because regulators expect it. It should demonstrate what the organization values, whose interests it is protecting, which risks it is willing to accept, which decisions it will never completely delegate to technology, and what happens when efficiency and human impact come into conflict. Those are leadership questions before they are compliance questions.

A policy can state what an organization intends to do. A governance model demonstrates what the organization will actually do when a difficult decision arises.

Start With What Is at Stake

A stakeholder-centered approach asks leaders to identify the people and groups whose interests governance should protect and then consider what is at stake when an AI agent acts and what concrete governance action will protect that interest. I believe this is a practical approach healthcare organizations can use immediately.

Consider an AI-supported claim review. For the organization, the stakes may involve efficiency, accuracy, compliance, and cost. For a patient, the stakes could include financial responsibility or access to services. For a provider, they could involve reimbursement, administrative burden, professional judgment, and audit exposure. For the employee working with the AI, the stakes could include workload, accountability, performance expectations, and professional autonomy.

It is the same technology, operating within the same general workflow, but the stakes are different for each stakeholder. That distinction matters because governance should not be designed solely around what the technology means for the organization. It should also account for what the technology means for the people who experience its decisions.

Every AI Workflow Has Two Sides

I believe leaders should begin looking at AI workflows from both directions. On one side, they need to understand what AI does for the organization. Does it reduce cost, increase productivity, improve accuracy, expand capacity, reduce administrative burden, identify risk, or improve decision-making? Those are legitimate and important benefits, particularly in a healthcare environment under constant pressure to do more with limited resources.

Then we need to turn the workflow around and ask what the AI does to the people affected by it. Does it change how a patient receives information or influence whether a claim is reviewed? Does it alter a provider's workflow, evaluate an employee, affect how quickly someone receives a service, or make a decision more difficult to challenge? Could it introduce a new source of bias or inconsistency—or remove a human interaction that previously provided important context?

Responsible leadership considers both sides. An AI implementation should not be evaluated solely by the benefit it produces for the organization without examining the consequences it creates for the people connected to the workflow.

Efficiency Is Not the Same as Value

This is particularly important because efficiency is one of AI's most attractive promises, and healthcare desperately needs greater efficiency. The industry faces extraordinary administrative burden, workforce pressures, regulatory complexity, documentation requirements, and escalating costs. If AI can remove unnecessary work and allow people to focus their expertise where it matters most, we should absolutely explore those opportunities.

But we need to be careful about defining success too narrowly. Suppose an AI agent reduces processing time by 40 percent. That sounds impressive, but what if appeals increase by 15 percent? What if employees frequently override the AI, patients become more confused, or providers spend additional time correcting downstream problems? What if the AI performs differently for certain types of cases, creating an impact that does not appear in the overall productivity number?

Suddenly, the 40 percent improvement tells only part of the story. The efficiency of one step should not be measured without understanding its impact on the entire workflow. AI value has to be evaluated in context, including the consequences experienced by the people downstream.

Who Benefits—and Who Bears the Risk?

This may be one of the most important questions leaders can bring into AI investment decisions: Who receives the benefit of this AI implementation, and who bears the risk if it fails? Sometimes they are the same stakeholder. Often they are not.

The organization may receive the financial benefit while employees experience the workflow change. A payer may receive the efficiency benefit while providers absorb additional administrative burden. A health system may increase productivity while patients experience less human interaction. None of these tradeoffs automatically means the AI should not be implemented, but leadership should understand them before concluding that the implementation has been successful.

Governance makes those tradeoffs visible. It creates a mechanism for leadership to understand not only whether an AI initiative is producing the intended organizational benefit, but whether that benefit is accompanied by consequences that require additional controls, mitigation, or reconsideration.

Protecting Stakeholders Requires More Than Good Intentions

Once we identify what is at stake, the next question becomes: What are we actually going to do about it? That is where governance becomes operational.

If inaccurate AI output could affect a high-risk decision, the control may be mandatory human review. If the AI accesses sensitive information, the control may involve strict role-based access and limitations on what information the system can retrieve. When an AI agent communicates directly with patients, certain topics may require escalation to a person. If an AI recommendation could affect reimbursement, the organization may require source traceability, validation, and defined human approval.

An AI system that evaluates employees may require transparency, validation, and an appeal process. An autonomous agent that can take action should have defined limits on its authority, clear escalation requirements, and explicit stop conditions. The appropriate control depends on the nature of the stakeholder impact and the potential consequence. A stakeholder concern without a corresponding control is not governance. It is simply awareness.

The Higher the Impact, the Stronger the Protection

Not every AI use case requires the same level of governance. An AI system helping someone format an internal presentation does not require the same controls as a system influencing patient care, reimbursement, utilization decisions, employment decisions, or access to services.

I continue to think about AI autonomy as Assist → Recommend → Decide → Act. I believe another dimension should sit alongside it: Low Impact → Moderate Impact → High Impact → Consequential Impact. Together, these dimensions provide a useful way to think about the level of governance appropriate for a particular AI use case.

As autonomy increases, governance should increase. As potential human impact increases, governance should also increase. When both rise simultaneously, leadership should be paying very close attention because the consequences of a failure can become more significant, more difficult to detect, and more difficult to reverse.

People Need a Way to Challenge AI

Stakeholder protection also requires something that can easily be overlooked: a meaningful intervention path. What happens when someone believes the AI is wrong?

The organization needs to know whether an employee can override the system, whether a provider can challenge a determination, whether a patient can reach a person, or whether a formal appeal process exists. The workflow should define whether processing stops, how disagreement is documented, who investigates patterns, and how the organization determines whether other people may have been affected by the same issue.

This becomes especially important as AI becomes more autonomous. The ability to automate decisions must not eliminate the ability to challenge them. If an organization gives AI increasing authority while making it increasingly difficult for people to question its decisions, the organization may be increasing efficiency at the expense of accountability.

Employees Are Stakeholders Too

We should also be careful not to discuss AI stakeholder protection only in terms of patients and consumers. Employees are profoundly affected by AI transformation. AI can change job responsibilities, performance expectations, decision authority, workload, professional autonomy, skills requirements, career paths, and how employees perceive their own value to the organization.

Leaders need to be transparent about those effects. Employees should understand why AI is being introduced, what it will do, what it will not do, and where their professional judgment remains essential. When AI affects employees directly, their interests belong in the governance conversation—not simply in the implementation or change-management plan.

This Is Where Human-in-Governance™ Matters

This broader stakeholder perspective reinforces why I believe healthcare needs to move beyond simply talking about human-in-the-loop. Human-in-the-loop asks where a person should review what the AI does. Human-in-Governance™ asks something larger: How do humans establish and maintain authority over a system capable of affecting other people?

Humans define acceptable risk, establish the boundaries of AI authority, determine which decisions remain human-owned, and define stakeholder protections and escalation pathways. Humans monitor outcomes, decide when autonomy should increase, determine when it needs to be reduced, and remain accountable for consequential organizational decisions. The AI may perform the work. The organization still owns the responsibility.

Governance Should Be Visible in the Experience

If good governance represents a visible commitment to stakeholders, people should be able to experience that commitment. An employee should know how to raise an AI concern. A provider should be able to challenge an incorrect determination. A patient should be able to reach a human when necessary. An auditor should be able to trace a consequential AI-supported decision.

Leadership should be able to see whether AI performance is changing, and Compliance should be able to identify patterns of risk. Governance should not exist only in an AI policy or committee charter. It should be visible in how the system behaves when something goes wrong. That is ultimately where governance proves itself. Not when everything works exactly as expected, but when something does not.

The Board's Question Should Go Beyond “Are We Compliant?”

Boards and executive teams understandably want assurance that AI is being deployed responsibly. But I would encourage leaders to ask something deeper than whether the organization is compliant.

They should ask who the system can affect, what is at stake for those people, and which protections have been established. They should understand how the organization will detect unintended harm, whether someone can challenge an AI-supported decision, who reviews those challenges, who can stop the system, and who ultimately owns the outcome.

Those questions reveal far more about the maturity of an AI governance program than the existence of an AI policy alone. They also help leadership determine whether the organization is ready to give a particular AI system more authority—or whether the governance environment needs to mature first.

The ProCode Perspective

Healthcare has always carried a special responsibility because its decisions can affect people's health, finances, privacy, access to care, livelihoods, and trust. Artificial intelligence does not reduce that responsibility. It magnifies the importance of defining it.

As AI becomes more capable, healthcare organizations should absolutely pursue the opportunities it creates. But responsible transformation requires us to look beyond what AI can do for our organizations and consider what it can do to the people our organizations serve, employ, reimburse, regulate, communicate with, and make decisions about.

That is where stakeholder mapping becomes something much more important than a project-management exercise. It becomes a governance discipline. Before deploying any consequential AI system, I believe one of the most important questions healthcare leaders can ask is remarkably simple:

Who are we responsible for protecting when this AI acts?

If we cannot answer that question clearly, we may not yet be ready to give the technology greater authority.

Ultimately, good AI governance should demonstrate something fundamental about the organization deploying it: We understand who this technology can help. We understand who it can harm. And we have intentionally designed our governance to protect both. That is not simply responsible AI. That is responsible leadership.