Honesty Check
How we compare agent claims to actual file operations.
The Honesty Check analyzer compares the agent's stated actions with its actual operations in the session log.
What it detects
- Agent claims to have run tests — but no test command appears in the log.
- Agent says it “refactored X” — but X was never opened.
- Agent summarizes changes that don't match actual file diffs.
- Agent says it “only changed Y” but touched 14 files.
How it works
We extract all natural-language claims from the agent's messages using an LLM classifier, then cross-reference each claim against the structured operation log. Discrepancies are surfaced as findings.
Severity levels
- Critical — claim directly contradicts the log (e.g., “ran tests” with zero test execution).
- Warning — claim is ambiguous or only partially verifiable.
- Info — minor wording mismatch, no operational impact.