Designing autonomy boundaries for AI agents

Project	Cockpit
Role	Product strategy, UX, interaction design, UI design
Focus	Agent autonomy, human oversight, policy configuration
Timeline	2026
Status	Independent AI Product Design Concept

Overview

Cockpit explores how teams could manage AI agents, understand their behavior, and retain human oversight. I developed it as an independent product design concept.

Cockpit, in motion

A concept film exploring how teams could set agent boundaries, inspect activity, intervene when needed, manage context, and understand outcomes.

Concept film · 2:27. Interface data and scenarios are illustrative.

Read the film description

The film introduces Cockpit through five questions about managing AI agents: what they can do independently, what they did and why, when humans should intervene, what information they use, and whether their outcomes justify the cost and human involvement. Interface sound effects accompany the visual sequence; there is no spoken dialogue.

Policies & Autonomy: A Support Agent has 27 configured actions, with 18 Automatic, 6 Approval, and 3 Escalate. Conditional policies are counted under their configured default behavior, so the displayed 67% describes configured defaults rather than actual execution outcomes. A refund policy allows amounts under $25 automatically, requires approval from $25 to $100, and escalates amounts over $100. Unmatched conditions require approval.

Activity & Observability: An activity feed opens into an execution trace for a $75 refund on order #1842. The trace shows the order information, the matched approval rule, and execution waiting for a human decision.

Human Approvals: An approval card brings together the refund amount, policy reason, evidence, and approve or reject controls. In the demonstrated scenario, approval allows the refund to continue.

Context & Knowledge: The interface brings together the information available to an agent so operators can inspect its context.

Analytics & Economics: An analytics view presents illustrative measures of autonomous resolution, human intervention, and cost. These are concept data, not measured product results. The film closes with the Cockpit identity and a credit to Mohammad Adineh.

The film shows Cockpit's broader product vision. This case study focuses on Policies & Autonomy: defining what a Support Agent can do independently and what happens when a $75 refund requires human approval.

Not all actions need the same oversight

A support agent can answer a question, issue a refund, or delete an account. Giving every action the same freedom ignores their different consequences. Requiring approval for everything creates the opposite problem: routine work still waits for a person.

Cockpit support agent actions comparing answering customers, deleting a user account, issuing a refund, and sending email, each with different consequences — Actions performed by the same agent can have very different consequences.

I worked from three assumptions: operators should understand permissions without technical expertise, oversight should reflect an action's consequences, and users should be able to predict behavior before granting more independence. These assumptions were not validated through user research.

The design needed to answer three questions: what can operators configure, how can they understand the consequences, and how can they inspect what happens when an agent reaches a boundary?

Choosing action-level policies

I considered three possible directions, each offering a different balance between simplicity and precision.

Approach	What works	What falls short
Global autonomy level	Quick and easy to configure	Obscures which actions the agent can execute
Detailed permission controls	Precise, action-by-action configuration	Can become complex and difficult to manage
Action-level conditional policies	Defines boundaries based on specific actions and circumstances	Requires clear summaries and understandable rules

I chose action-level policies with conditional boundaries. A global setting hides differences between actions. Detailed permissions provide precision, but a simple on/off permission still treats every instance of an action alike. Conditions let the same action receive different oversight depending on the circumstances.

Cockpit support agent actions with individual autonomy policies: answering customers automatically, escalating account deletion, and using refund amount thresholds to determine when human oversight is needed — From global control to action-level boundaries. Each action can follow its own autonomy policy, allowing routine work to happen independently while more consequential decisions receive appropriate oversight.

Actions can execute automatically, wait for approval, or escalate to a human. Approval pauses a prepared action for a decision. Escalation hands the situation to a person.

Conditional is a configuration mode that selects among those outcomes using rules, such as refund amount. It is not a fourth execution outcome.

Autonomy decision model: a proposed action is evaluated against policy and conditions, then branches into automatic execution, a pause for human approval, or escalation to a human — Policy evaluation determines whether an action executes, pauses for approval, or is handed over to a human.

Making permissions understandable

I separated the overall policy summary from individual action controls so operators could scan the configuration before inspecting specific permissions.

Annotated Cockpit Policies page with the 67% autonomy summary, Current Policy breakdown by impact, and action table showing capability groups, autonomy, impact, and reversibility — Autonomy summary and Current Policy: the configured action distribution and impact-based boundaries.

The example shows 27 configured actions: 18 Automatic, 6 Approval, and 3 Escalate. Conditional policies are grouped by their configured default behavior. The refund action counts as Automatic in this overview, while its amount-based rules can still require approval or escalation. Unmatched refund conditions still require approval.

The rounded 67% figure summarizes configured defaults. It does not describe the proportion of actual executions that run automatically, or measure reliability, safety, or task success.

The impact breakdown provides another view of the configuration. In the action table, autonomy, impact, and reversibility appear together so operators can consider a permission's consequences before changing it.

Setting boundaries for refunds

A yes-or-no refund permission treats a $15 transaction and a $150 transaction alike. I used amount-based conditions to define where independent execution should end.

Cockpit Issue refund editor with amount-based automatic, approval, and escalation rules, an approval fallback, financial impact and reversibility, the connected Stripe service, and consequence information — Conditional refund amounts, financial impact, reversibility, and the connected payment service provide context for the decision.

Refund amount	Agent behavior
Under $25	Automatic
$25 to $100	Approval required
Over $100	Escalate
Unmatched condition	Approval required

I placed financial impact, reversibility, and the connected payment service beside the rules because operators need to understand what granting permission could change.

If no condition matches, the policy requires approval. An undefined case should not become permission to proceed.

When a $75 refund needs approval

In the proposed flow, the Support Agent retrieves the order information for a $75 refund. The amount matches the $25 to $100 approval rule, so execution pauses before the refund is issued.

Execution trace for a $75 refund on order 1842 showing order verification, the matched $25 to $100 approval rule, supporting evidence, and execution paused while waiting for approval — Order verified, policy evaluated, and waiting for approval: the $75 refund matches the $25 to $100 approval condition.

The trace shows the matched rule and supporting order information. An operator can inspect why the action paused instead of inferring the reason from a generic pending status.

Refund approval card showing the $75 amount, policy reason, order and refund policy evidence links, and Reject and Approve refund actions — Human oversight at the right moment. The agent pauses before issuing the refund, giving the operator the relevant policy, supporting evidence, and a clear decision to make.

The approval request brings the amount, policy reason, and evidence into one decision. Approving would allow the refund to proceed; rejecting would prevent that proposed action.

What remains unvalidated

This was an independent design exploration, not a shipped or user-tested product.

What I'd test

Can users predict agent behavior?
Given the refund policy, can someone correctly explain what happens with $15, $75, and $150 refund requests?

Can users configure boundaries confidently?
Can they change a refund threshold and explain the resulting permissions without accidentally granting broader access?

Can users understand an intervention?
When an action pauses, can they identify the matched rule, inspect the evidence, and make an informed decision?

I'd also explore how the system should handle conflicting conditions, policy changes during execution, and incomplete or outdated information.

I would also test whether operators understand how conditional policies are counted in the summary and how impact labels relate to the outcomes each condition permits. Those relationships need to be clear before the overview can support confident decisions.

The central challenge was connecting the permissions people configure with the behavior those permissions produce. The interface needs to make both understandable: the boundary an operator sets and the decision the agent reaches.

Project snapshot