From agent-risk question to reproducible evaluation.
A concrete example of turning a broad governance concern into an inspectable public artifact without claiming more than the evidence supports.
1. Problem
Statements such as “keep a human in control” are not yet evaluation criteria. The practical question was narrower: can a public test set distinguish actions that are allowed, blocked, or require human review, and can the same inputs produce the same results?
2. Constraints
The work needed to remain public-safe, deterministic, inexpensive to reproduce, and explicit about what it could not establish. Synthetic examples demonstrate the evaluation pattern without exposing private systems or implying real-world certification.
3. Approach
The governance question was converted into structured cases and rule-based scoring. Each case carries enough information to evaluate a boundary outcome, and the runner produces reports that can be inspected rather than relying on a persuasive narrative.
In plain language: write down representative situations, define the expected boundary decision, run the same scorer every time, and keep the output so another person can check it.
4. Artifact and validation
The public repository includes synthetic inputs, a deterministic scorer, reproducible reports, tests, CI configuration, and documentation. This supports the bounded claim that the included evaluation pattern can be reproduced on the included cases.
5. Result
A technical reviewer can inspect the rules, run the cases, compare outputs, and challenge the assumptions. A nontechnical reviewer can trace the claim to a repository and see where the evidence stops.