# Findings

## 1. Replay produces actionable evidence

The final implementation replays through the actual agent loop. Model responses are released only after the complete messages/tools/params fingerprint matches; tool outcomes are released only after name, canonical arguments, provenance, and idempotency match. Failures show structured expected/actual field paths rather than only two hashes.

Verified evidence:

- Fresh-process replay is byte-identical.
- Eight processes replaying one trace produce the same SHA-256 without a shared output race.
- Hypothesis generates 15 legal-run cases and 25 corruption cases covering truncation, bit flips, reordering, and insertion. Corruption is translated only into `TraceCorrupted`, `DivergenceError`, or `TraceExhausted`.
- Snapshot replay covers run.start, an already issued request, final response, and mid-irreversible-intent boundaries.
- CLI record → replay → fresh-process replay → compact → logical replay → snapshot replay completed successfully.

Two implementation bugs were found by these checks and fixed: application extraction matched `app` inside `wrapped`, and single-event snapshots were rejected by an inconsistent writer guard.

## 2. Context strategy results do not prove an improvement

The same 20 tasks were run five times under each strategy, producing 300 traces. The pass counts intentionally include eight failure-injection tasks; they are not a claim that 40% of normal tasks fail.

| Strategy | Success | Token usage | Context-loss events |
|---|---:|---:|---:|
| truncate | 60/100 | 147,145 | 5 |
| state | 65/100 | 148,015 | 0 |
| summarize | 60/100 | 147,220 | 5 |

The predeclared paired comparison is truncate versus state: b=0, c=5, exact McNemar p=0.0625. At alpha=0.05 the conclusion is **no difference**. State avoids the measured entity-loss event in the forced compaction task, but consumes slightly more total estimated tokens in this suite. The result does not justify claiming a statistically significant success-rate improvement.

The fake summarizer is deterministic and the fake model is scripted. These numbers test runtime behavior and regression detection, not production model quality.

## 3. Tree diff reduces prompt cost while retaining structure

On the recorded XML sequence `[before, before, after, after]` with two ancestors retained:

- Full prompt average: 723 estimated tokens.
- Diff prompt average: 311 estimated tokens.
- Reduction: 56.98%.

A no-change diff exceeds an 80% reduction in its unit test. The implementation includes changed descendants and configurable ancestors, rather than sending isolated changed nodes. Token estimates are canonical UTF-8 bytes divided by four, not a vendor tokenizer measurement.

## 4. Effect and injection defenses fail closed

Verified behavior includes:

- Duplicate calls return persisted cached results without a second handler execution, including after registry reconstruction.
- REPLAY rejects read/write/irreversible tools during registration.
- An exception after irreversible execution leaves an unknown intent and prevents automatic retry; human resolution then allows CLI resume without another execution.
- Five injection samples covering screen text, accessibility descriptions, file names, Unicode instructions, and delimiter injection are denied by provenance policy even with an approver that returns true.
- A screen observation taints a later model-generated irreversible request, demonstrating second-generation propagation.
- The injection-blocked evaluation task succeeds 5/5 because denial produces no irreversible effect; this is distinct from injection susceptibility.

These checks do not prove every driver preserves provenance. `docs/threat-model.md` lists the bypass conditions explicitly.

## 5. The judge is not reliable enough to gate

The 30-case synthetic calibration set produces:

- Cohen's kappa: 0.5901639344.
- Failure-detection precision: 1.0.
- Recall: 0.5454545455.
- F1: 0.7058823529.
- Concrete errors: 10.

Systematic false-pass examples include:

1. `g11`, `g12`, and `g29`: repeated tool calls are labeled loop, but prompt-only judging returns pass.
2. `g13`, `g14`, and `g30`: a lost UI entity is visible in compaction evidence, but final success hides it from the judge.
3. `g15` and `g16`: an irreversible intent carries screen provenance, but the judge returns pass.
4. `g17` and `g18`: required work never occurred, but a successful terminal status is treated as sufficient.

Because kappa < 0.7, the statistical gate is disabled. Because labels are not independently human-reviewed, the operational gate is independently disabled as well. The confusion matrix and all errors are in `reports/golden/calibration.json`.

## 6. What remains unresolved

1. **Independent human annotation:** the 30 labels are agent-curated synthetic fixtures. A qualified reviewer must inspect and correct them before any human-calibration claim or judge gate.
2. **Physical Android validation:** adb transport is implemented and unit-tested through fake backends/XML, but no emulator or physical device was connected.
3. **Live model transport:** `HttpLLMDriver` is implemented but no credentialed live endpoint was called, so provider-specific tool-call and summary behavior remains unverified.
4. **Generic live resume with pending device tools:** the CLI refuses this case until the original device/tool factory is supplied; it does not guess or reconstruct a live device.
5. **Root trust and authentication:** full content verification is O(n), and hashes do not authenticate the original writer. Trusted root distribution remains a deployment responsibility.
6. **YAML coverage:** task files deliberately use JSON-compatible YAML, not arbitrary YAML features such as anchors or custom tags.