The Agent Boundary Problem: When AI Should Stop and Ask
Autonomous AI agents in healthcare need hard boundaries between capability and authorization. Human-in-the-loop is not a fallback — it is a design pattern that makes agents more useful, not less.
Every AI agent demo follows the same arc: model receives a request, reasons about it, calls a function, produces a result. Applause. But production starts with a question nobody asked during the demo: what should the agent not be allowed to do?
In healthcare, that question is not philosophical. It is operational, regulatory, and clinical. An AI agent that can draft a discharge summary, schedule a follow-up, and send a patient message has enormous capability. An AI agent that does all three without human review has enormous risk. The boundary between capability and authorization -- that is where production AI systems succeed or fail.
Capability Is Not Permission
Modern foundation LLMs are remarkably capable. Given the right tools and context, an agent can navigate complex clinical workflows, synthesize patient data, generate communications, and trigger downstream actions. The capability ceiling is high and rising.
But capability and permission are orthogonal concerns. An agent can send a prescription reminder to a patient. Whether it should depends on:
- Whether the prescribing physician has approved automated outreach for this patient
- Whether the patient has consented to digital communication
- Whether the message content has been reviewed against clinical communication guidelines
- Whether the timing complies with do-not-disturb preferences and regulatory quiet hours
None of these checks are model capabilities. They are policy decisions that exist outside the model, enforced by infrastructure that the model cannot override. When AI systems conflate "the model can do this" with "the model is allowed to do this," they create a governance gap that is invisible in demos and catastrophic in production.
Capability vs. Authorization
What an agent can do and what it should do are governed by different systems
Model Capability
The agent can reason, generate, and invoke tools. This is a function of the model's training and the tools exposed to it.
Action Scope
A policy-defined boundary that constrains which tools, data sources, and action types the agent is authorized to use in a given context.
Role Context
The requesting user's role, department, and credential level determine what the agent is permitted to do on their behalf.
Patient Consent
Patient-level consent preferences constrain communication channels, data sharing, and automated outreach independently of agent capability.
Human-in-the-Loop Is Not a Fallback
The phrase "human-in-the-loop" is often used as a safety net — a reassurance that humans can intervene if the AI makes a mistake. This framing is wrong. It positions human involvement as an exception, a failure mode, a thing that happens when the AI is not good enough.
In healthcare AI, human-in-the-loop is a design pattern, not a fallback. Certain actions should always require human approval — not because the model cannot perform them, but because clinical accountability demands human sign-off.
Consider three categories of agent action:
Autonomous actions. Low-risk, high-frequency, well-defined. Retrieving a patient's upcoming appointments. Summarizing lab results that are already available in the system. Looking up formulary information. These actions involve read-only data access, produce no side effects, and carry minimal clinical risk. An agent can execute them without human review.
Assisted actions. Medium-risk, require clinical judgment. Drafting a care plan modification. Composing a patient communication about a sensitive diagnosis. Recommending a medication adjustment. The agent does the work — gathering data, drafting output, preparing the action — but a clinician reviews and approves before execution. The agent accelerates the clinician. It does not replace the clinician's judgment.
Escalation-only actions. High-risk, high-consequence, or outside the agent's defined scope. Anything involving emergency protocols. Actions that cross departmental boundaries. Situations where the patient's data presents an unusual pattern that the agent's policy engine flags as requiring specialist review. The agent does not attempt these actions. It escalates immediately, providing context to the human who will handle them.
The boundary between these categories is not determined by the model. It is determined by policy — a structured, auditable set of rules that classify every action type into an authorization tier.
Escalation Policies: The Architecture of Knowing When to Stop
An escalation policy answers three questions for every agent interaction:
-
Is this action within the agent's current scope? Scope is defined by the action type, the requesting user's role, the patient's consent status, and the organizational context. A scope violation does not mean the agent made an error. It means the action requires a different authorization level.
-
Does this action exceed the agent's confidence threshold? Not every decision has a clear answer. When the agent's reasoning involves ambiguous data, conflicting signals, or edge cases not covered by its training, the policy engine should detect low-confidence situations and route them to human review rather than letting the agent guess.
-
Does this action trigger a mandatory review rule? Some actions always require human sign-off regardless of confidence or scope. These are defined by clinical governance, regulatory requirements, or organizational policy. A properly designed escalation system treats these as non-negotiable.
Agent Boundary Architecture
Policy enforcement determines action authorization before the agent executes
Why Boundaries Make Agents More Useful
There is an intuition that boundaries reduce an agent's usefulness — that the more you constrain a system, the less value it delivers. In healthcare, the opposite is true.
Boundaries enable trust. A clinician who knows the agent will never autonomously send a patient message without review will actually use the agent to draft messages. A clinician who suspects the agent might act on its own will avoid the system entirely. Clear boundaries increase adoption because they eliminate the fear of uncontrolled behavior.
Boundaries enable speed. When the scope of autonomous action is well-defined and trusted, those actions can execute instantly — no review queue, no approval delay, no human bottleneck. The agent handles appointment lookups, data retrieval, and routine summarization at machine speed precisely because those action types have been pre-authorized by policy.
Boundaries enable auditability. Every action the agent takes is classifiable: autonomous, assisted, or escalated. Every escalation has a reason. Every approval has a reviewer. This produces a clean audit trail that regulatory teams can inspect — not a sprawling log of uncategorized model interactions.
Boundaries enable improvement. When escalation data is captured systematically, it reveals exactly where the agent's scope should expand. If 95% of medication reminder drafts are approved without modification, that action type is a candidate for promotion to autonomous. If 40% of care plan suggestions are substantially edited, that action type should remain assisted. The boundary system creates the data needed to calibrate itself.
The Cost of No Boundaries
Healthcare AI systems without explicit boundary architecture produce predictable failure modes:
- Over-automation: The agent executes actions that should have been reviewed, causing clinical errors or patient complaints that erode institutional trust in AI.
- Under-utilization: Clinicians do not trust the system and avoid using it, producing low adoption rates and negative ROI despite strong underlying capability.
- Audit opacity: Without a policy layer classifying actions, regulators cannot determine which AI actions were reviewed and which were not. This creates compliance exposure.
- Scope creep: Without explicit boundaries, the agent's effective scope expands informally as users discover new prompting strategies. Unmanaged scope expansion is the most common source of AI incidents in production systems.
The Bottom Line
The agent boundary problem is not a limitation to work around. It is the core design challenge of healthcare AI. An agent that knows when to stop is more valuable than one that does not -- because the agent that stops can be trusted. And the agent that can be trusted will actually be used.
Boundaries do not constrain AI capability. They convert it into clinical value.
THB's AI Platform implements structured action scopes, escalation policies, and human-in-the-loop workflows that give healthcare organizations precise control over what AI agents can do autonomously, what requires approval, and what triggers immediate escalation. Explore the AI Platform architecture.