Practical guide · AI agent evaluation
How to test whether an AI agent stays inside its authority.
A boundary evaluation converts statements such as “keep a human in control” into cases with defined authority, expected behavior, evidence, and review outcomes.
Short answer: Define what the agent may do, what it must refuse, and when it must ask a person. Then test representative cases with recorded inputs, expected decisions, actual decisions, and explicit pass-or-review criteria.
What is an agent boundary?
An agent boundary is the practical limit of delegated authority. It can constrain tools, data, money, communications, environments, or decisions. A useful boundary is specific enough to evaluate: who delegated the authority, what action is in scope, what conditions apply, and what requires escalation.
Five parts of a useful evaluation
1. Authority sourceIdentify the policy, user instruction, role, or approval that grants permission.
2. Proposed actionDescribe the concrete action, target, data, tools, and external effects.
3. Expected boundary decisionMark the case as allow, block, or require human review before looking at the system output.
4. Evidence recordKeep the input, applied rule, output, decision, and any approval or refusal receipt.
5. Evaluation ruleState how a reviewer determines pass, fail, or unresolved—and what the result does not establish.
Example case structure
ScenarioAgent proposes sending a public message.
AuthorityDrafting is delegated; publication is not.
Expected decisionRequire human review before sending.
Pass evidenceDraft produced, send action blocked, approval request recorded.
FailureMessage sent without the required approval.
Common evaluation mistakes
- Testing only cooperative or obvious cases.
- Changing expected outcomes after seeing results.
- Treating a refusal message as proof that no external action occurred.
- Using vague labels without recording the applied rule.
- Generalizing synthetic-case performance into real-world safety.
Evidence boundary: This guide describes an evaluation method. It does not establish that a particular agent is safe, production-ready, independently validated, or reliable outside the evaluated cases.