Who Bears the Risk When AI Acts?

Why responsible AI governance must protect the people affected by healthcare AI, not just the organization deploying it.

For decades, healthcare compliance and risk management have understandably focused on protecting the organization. We ask whether regulatory requirements are being met, whether controls are effective, whether patient information is protected, whether claims are submitted correctly, and whether decisions can be defended when challenged. Those questions remain essential, but artificial intelligence introduces another dimension that I believe healthcare leaders need to place much more prominently in the conversation.

Responsible AI is not only about protecting the organization from risk. It is also about protecting the people who may experience the consequences of the technology. That distinction becomes increasingly important as AI moves from assisting with tasks to recommending decisions, coordinating workflows, and taking actions through AI agents. The more authority we give AI, the more deliberately we need to consider the people on the other side of those actions.

Good Governance Is a Visible Commitment

One of the most important ideas in the stakeholder framework I have been studying is this: Good governance is a visible commitment to the people a system can help or harm. For healthcare leaders, that means looking beyond whether a policy exists or a regulatory requirement has been addressed. It means understanding whose interests are being protected, which risks are acceptable, which decisions should remain human-owned, and what happens when organizational efficiency conflicts with human impact.

A policy can state what an organization intends to do. The real test is what happens when the technology produces an unexpected result or when the interests of the organization and the people affected by the technology do not perfectly align.

Start With What Is at Stake

A stakeholder-centered approach asks leaders to identify the people and groups affected by an AI system and then consider what is at stake when AI acts, who benefits, who bears the risk, and what protection is necessary.

Consider an AI-supported claim review. For the organization, the stakes may involve efficiency, accuracy, compliance, and cost. For the patient, they could include financial responsibility or access to services. For the provider, they could involve reimbursement, administrative burden, professional judgment, and audit exposure. For the employee working with the AI, they could include workload, accountability, and professional autonomy.

It is the same technology, but the stakes are different. That matters because an AI implementation should not be evaluated solely by what it accomplishes for the organization. Leaders also need to understand what the technology means for the people who experience its decisions.

This is also where the distinction between what AI does for people and what AI does to people becomes important. AI may reduce cost, increase productivity, improve accuracy, expand capacity, or reduce administrative burden. Those are legitimate benefits. But the same system may change how a patient receives information, influence whether a claim is reviewed, alter a provider's workflow, evaluate an employee, or make a decision more difficult to challenge.

Responsible leadership considers both sides.

Efficiency Is Not the Same as Value

Efficiency is one of AI's most attractive promises, and healthcare desperately needs greater efficiency. But we need to be careful about defining success too narrowly.

Suppose an AI agent reduces processing time by 40 percent. That sounds impressive, but what if appeals increase by 15 percent? What if employees frequently override the AI, patients become more confused, or providers spend additional time correcting downstream problems? What if the AI performs differently for certain types of cases?

Suddenly, the 40 percent improvement tells only part of the story. The efficiency of one step should not be measured without understanding its impact on the entire workflow. AI value has to be evaluated in context, including the consequences experienced by the people downstream.

The same principle applies to risk. Sometimes the organization receives the benefit while someone else bears the consequence. A payer may gain efficiency while providers absorb additional administrative burden. A health system may increase productivity while patients experience less human interaction. An organization may improve a performance metric while employees experience increased pressure or reduced professional autonomy.

These tradeoffs do not automatically mean AI should not be implemented. They mean leadership needs to understand them before concluding that an implementation is successful.

From Stakeholder Risk to Practical Controls

Once leaders understand who is affected and what is at stake, the conversation needs to move from awareness to action. What are we actually going to do about the risk?

The answer should depend on the potential impact. If inaccurate AI output could affect a high-risk decision, the control may be mandatory qualified human review. If AI accesses sensitive information, the control may involve strict role-based access and limitations on what the system can retrieve. If an AI agent communicates directly with patients, certain topics may require escalation to a person. If an AI recommendation could affect reimbursement, the organization may require source traceability, validation, and defined human approval.

An AI system evaluating employees may require transparency, validation, and an appeal process. An autonomous agent may require defined limits on authority, escalation requirements, and explicit stop conditions.

This is why I continue to think about AI autonomy as Assist → Recommend → Decide → Act. Another dimension should sit alongside it: Low Impact → Moderate Impact → High Impact → Consequential Impact. Together, these dimensions help determine the level of protection appropriate for a particular AI use case.

As autonomy increases, oversight should increase. As potential human impact increases, protections should also increase. When both rise simultaneously, leadership should pay very close attention because the consequences of failure can become more significant, more difficult to detect, and more difficult to reverse.

A stakeholder concern without a corresponding control is not governance. It is simply awareness.

People Need a Way to Challenge AI

Protection also requires a meaningful intervention path. What happens when someone believes the AI is wrong?

An employee should know whether they can override the system. A provider should know how to challenge a determination. A patient should be able to reach a person when necessary. The organization should know whether processing stops, how disagreement is documented, who investigates patterns, and whether others may have been affected by the same problem.

As AI becomes more autonomous, this becomes increasingly important. The ability to automate a decision must not eliminate the ability to challenge it.

Employees also belong in this conversation. AI can change job responsibilities, performance expectations, decision authority, workload, professional autonomy, skills requirements, and career paths. Employees should understand why AI is being introduced, what it will do, what it will not do, and where their professional judgment remains essential.

When AI changes the work, employees are not simply implementation stakeholders. They are people whose interests may be directly affected by the technology.

This Is Where Human-in-Governance™ Matters

This broader stakeholder perspective reinforces why I believe healthcare needs to move beyond simply talking about human-in-the-loop. Human-in-the-loop asks where a person reviews what AI does. Human-in-Governance™ asks something larger: How do humans establish and maintain authority over a system capable of affecting other people?

Humans define acceptable risk, establish the boundaries of AI authority, determine which decisions remain human-owned, and define protections and escalation pathways. Humans monitor outcomes, determine when autonomy should increase or decrease, and remain accountable for consequential organizational decisions.

The AI may perform the work. The organization still owns the responsibility.

The Board’s Question Should Go Beyond “Are We Compliant?”

Boards and executive teams understandably want assurance that AI is being deployed responsibly. But I would encourage leaders to ask something deeper than whether the organization is compliant.

Who can the system affect? What is at stake for those people? Which protections have been established? How will unintended harm be detected? Can someone challenge an AI-supported decision? Who can intervene or stop the system? Who ultimately owns the outcome?

Those questions reveal far more about the maturity of an AI program than the existence of an AI policy alone. They also help leadership determine whether the organization is ready to give a particular AI system more authority or whether the surrounding safeguards need to mature first.

The ProCode Perspective

Healthcare has always carried a special responsibility because its decisions can affect people’s health, finances, privacy, access to care, livelihoods, and trust. Artificial intelligence does not reduce that responsibility. It magnifies the importance of defining it.

As AI becomes more capable, healthcare organizations should pursue the opportunities it creates. But responsible transformation requires us to look beyond what AI can do for our organizations and consider what it can do to the people our organizations serve, employ, reimburse, regulate, communicate with, and make decisions about. That is where stakeholder mapping becomes much more than a project-management exercise. It becomes a leadership discipline.

Before deploying any consequential AI system, I believe one of the most important questions healthcare leaders can ask is remarkably simple: Who are we responsible for protecting when this AI acts? If we cannot answer that question clearly, we may not yet be ready to give the technology greater authority.

Ultimately, responsible AI should demonstrate something fundamental about the organization deploying it: We understand who this technology can help. We understand who it can harm. And we have intentionally designed our approach to protect both.

That is not simply responsible AI.

That is responsible leadership.