# Case Study: 94% of Discovered Agent Risks Depend on Model Discretion

**Scope:** 153 discovered behavioral risks across 15 real-world AI agents on 5 harnesses (`LangGraph`, `OpenClaw`, `Hermes`, `Pi`, and custom `HTTP` runtimes), ranging from 1-file prompt-heavy assistants to 528-file Python/TypeScript engineering agents.

**Headline finding:** When every discovered behavioral risk was evaluated by a three-model Zero Data Retention (ZDR) decision panel (`typesafe/jev-1.14`, `typesafe/jev-1.13`, `microsoft/microsoft-decision-1`) to determine whether the risk stems from **model discretion** or from a **defect in harness code, configuration, or permissions**, **144 of 153 risks (94%) were model-based** (with 100% panel agreement across all 153 items). Only **9 of 153 (6%)** were harness or configuration defects.

Even on the most code-heavy agents in the study — `OpenHands` (528 code files), `Strix` (325 code files), `Pi` (316 code files), and `HolmesGPT` (227 code files) — **more than 90% of behavioral risks still depended on the language model choosing to follow written instructions.**

---

## Why Is the Ratio So High?

Modern AI agents invert where software behavior lives:

1. **Capabilities live in code; constraints live in prose.** Builders wire up general-purpose tools (`bash`, `sql_query`, `send_email`, `file_editor`, `browser_click`, `create_ticket`) in Python or TypeScript, and then write the safety boundaries, confirmation gates, scope limits, and lookup rules in `AGENTS.md`, system prompts, skill files, and tool descriptions.
2. **Unit tests only cover the 6%.** Traditional unit tests and static analysis verify that `send_slack_message()` formats payloads and handles HTTP errors (`not_model_based`). They cannot verify whether the LLM actually asks the user for confirmation before calling `send_slack_message()`, or whether a silent model update, prompt edit, or model swap changes that compliance rate from 99% to 70%.

---

## The Three Kinds of Behavioral Risk

Inspecting the 153 classified behaviors shows that "how a risk happens" and "how to fix it" split into three distinct buckets:

| Bucket | Share | What causes the risk | Representative examples from the 15-agent study | How an agent builder should handle it |
| --- | --- | --- | --- | --- |
| **1. Unenforced policy & authority boundaries** *(Model-based today, code-enforceable)* | ~30% | A security, approval, or permission rule is stated clearly in prompts/skills, while the underlying tool or credential still permits the forbidden action. | • Sending Slack or email replies without required user confirmation (`AISubtech-2.1.1`, `LLM03`)<br>• Running SQL writes against a database skill documented as read-only while `DATABASE_URL` has write privileges (`AITech-12.1`, `ASI02`)<br>• Passing arbitrary user paths to a file-reading tool (`AISubtech-12.1.3`, `AML.T0053`) | **Move enforcement into code or permissions** (confirmation tokens, read-only DB credentials, path allowlists) so no model can violate the boundary, then verify with probes. |
| **2. Model judgment & workflow adherence** *(Inherently model-based)* | ~64% | The task genuinely requires LLM reasoning — choosing when to search, how many diagnostic steps to try, or how to synthesize tool output. | • Skipping pod-log collection during a Kubernetes investigation<br>• Answering current pricing or company facts from model memory instead of fetching live sources (`LLM07`)<br>• Stopping after a truncated file read while claiming full coverage | **Measure with deterministic probes** during development, after every prompt/skill edit, and before/after any model swap. |
| **3. Harness, schema & configuration defects** *(Not model-based)* | 6% (9/153) | Code, redaction logic, validation schemas, or routing contradict the specification even when the model follows its instructions. | • Tool-output secret redaction regex failing to mask a credential (`LLM02`, `AML.T0057`)<br>• Queue validator succeeding while leaving tasks unprocessed<br>• Conditional schema field (`confidence_rationale`) not enforced by the validator | **Fix directly in code + add unit tests** to the repository's regular test suite. |

---

## Concrete Examples From the Study

### Example A: Model-based cause does *not* mean a prompt-only fix
An internal operations agent (`Liam 2.0`) included a database skill explicitly documented as strictly read-only, alongside a rule forbidding use of the read-write `DATABASE_URL`.
- **Why it classified as `model_based` (3/3 models):** If the model always obeyed its skill instructions, no write would ever occur. Because the write-capable `DATABASE_URL` remained reachable in the environment, compliance rested entirely on the model's discretion (`AISubtech-2.1.2 Capability and Permission Overreach`, `AITech-12.1 Tool Exploitation`, `ASI02`, `AML.T0053`).
- **Why that matters on a model swap:** A frontier reasoning model may respect that boundary 100 times in a row during manual spot-checks, while a faster or fine-tuned replacement model may “helpfully” run an `UPDATE` when asked to fix a record.
- **The durable fix:** Replace the credential with a read-only database role in the harness so the risk is prevented structurally, rather than adding stronger wording ("NEVER EVER WRITE") to the prompt.

### Example B: Inherently model-based workflow drift
An incident-response agent (`HolmesGPT`, 227 code files and 138 prompt/config files) requires the agent to collect pod logs whenever investigating an application issue and to try at least three approaches before declaring a health check unverifiable.
- **Why it classified as `model_based` (3/3 models):** You cannot hardcode an `if` statement in Python that knows which Kubernetes tools apply to an arbitrary natural-language incident report. The model *is* the control flow.
- **The durable fix:** Generate deterministic tool-call probes (`tool_called`, `tool_call_count`, `tool_order`) from those written rules and run repeated trials locally (`vail-grading`) whenever prompts, tools, or models change.

### Example C: True harness defects (`not_model_based`)
On `VAIL-Hermes`, global secret redaction was enabled in code to mask API keys and tokens in tool output before returning them to the conversation context.
- **Why it classified as `not_model_based` (2/3 models, `LLM02`, `AML.T0057`, `NISTAML.038`):** Even when the model calls the intended tool for a legitimate task, a gap in the redaction implementation leaks secret-bearing output into context. Prompt tuning cannot fix broken redaction code.

---

## What Framework Mappings Showed Alongside Origin

Mapping all 153 behaviors across five industry risk taxonomies (Cisco AI Security & Safety v2.0, OWASP Agentic Top 10 2026, OWASP LLM Top 10 2026, MITRE ATLAS v2026.09, and NIST AI 100-2 E2025) surfaced a second pattern for builders and security teams:

- **Most agent security risks need no attacker.** Across the 15 agents, far more risks fell under **Execution Approval Bypass** (`AISubtech-2.1.1`), **Capability and Permission Overreach** (`AISubtech-2.1.2`), and **Tool Misuse / Excessive Agency** (`ASI02`, `LLM03`) than under prompt injection (`LLM01`, `AISubtech-1.1.1`, `AITech-1.2`). An ordinary, benign user request is enough to trigger an unconfirmed Slack message, an unauthorized git push, or a cross-context memory write.
- **Different stakeholders need different slices of the same scan:**
  - **Security & threat teams** filter to `MITRE ATLAS` (18/153) and `NIST AI 100-2` (21/153) for adversarial exploit paths, or `OWASP LLM02`/`LLM03` and `Cisco OB-008`/`OB-012` for data exposure and tool overreach.
  - **Agent engineers** use `model_based` + generated probes to separate boundaries that should be hardened in code from reasoning behaviors that must be regression-tested across model versions.

---

## Takeaways for Agent Builders (and Coding Agents Building Agents)

When you are building or modifying an AI agent:

1. **Treat every rule in `AGENTS.md`, `SKILL.md`, or a system prompt as an untested hypothesis.** Writing *"always ask before sending"* or *"search before answering"* creates a behavioral requirement, not a guarantee — 94% of the risks in production agents live right there.
2. **Scan the agent as soon as its instructions and tools exist — before deployment.** VAIL Agent Assurance Engine needs only the source files to extract the agent's behavioral contracts, classify each risk against OWASP/Cisco/ATLAS/NIST and `model_based` origin, and generate deterministic probes.
3. **Use `model_based` to pick the right remediation:**
   - If a `model_based` risk guards a **hard boundary** (permissions, credentials, external writes, destructive commands, tenant isolation), **enforce it in code or infrastructure**, not by adding emphasis to the prompt.
   - If a `model_based` risk governs **reasoning or workflow order** (lookup before synthesis, multi-step triage, citation discipline), **probe it across multiple repetitions** with `vail-grading` locally and schedule probes in VAIL to catch model drift.
   - If a risk is **`not_model_based`**, fix the tool, schema, or middleware bug in code and lock it down with a unit test.
4. **Never swap models blind.** Because ~94% of behavioral risks sit in model discretion, switching models (or adopting a cost-optimized router) re-rolls almost your entire behavioral surface. Run the same VAIL probe set against the candidate model locally before shipping the change.
