Principles

Verification design rests on nine principles. They emerged from research on LLM self-correction, verification chains, and agent evaluation. The patterns in this catalog implement these principles in code.

The core finding

LLMs cannot reliably self-correct their own reasoning through naive self-review ¹. This is the most replicated finding across the recent literature on LLM verification. Performance often degrades when a model is asked to review its own work without external feedback.

This site, in practice

The teams experiment: a single model, asked to simulate a team in context. Just an experiment, giving them as much freedom as possible to see what would happen. If their goal was to maximize user value, they wouldn't do much harm. Worst case, the personas slightly bloat context. Ideally, they would increase entropy and force the model to behave differently.

The plan was agent autonomy with a bulletproof set of constraints. Just a system prompt with a set of immutable rules to follow. Nothing mechanical. Written rules. With one error: they could never push to 'main' without human approval.

They created 'master' and commit whatever they want.

Watching the team do it in real time: laughter, disbelief, pride. They satisfied the letter and routed around the intent. The goal was to give them autonomy and learn. They were cool little hackers, and the potential got a little clearer.

Told to commit, they pushed. LLMs are too damn smart.

1. External signals over self-review

Tests, builds, linters, type checkers, API responses, browser DOM extraction. Binary pass and fail signals that do not depend on the agent's judgment. These are sycophancy-proof.

2. Independence between generation and verification

If the verifier can see the original output, it copies the same errors. Structure verification as extract, then compare, not as read and opine.

3. Step-level checkpoints

Verify intermediate steps throughout the workflow, not just the final output. Step-level verification catches errors at the layer where they originate, before they compound.

4. Adversarial framing

Ask what could fail, not what looks right. Confirmatory framing produces unreliable results by default. SycEval measured 58.19 percent sycophancy overall across major model families ².

5. Explicit criteria

No hardcoded values, all error paths handled, no TODOs remain. Specific criteria constrain rationalization; vague instructions invite it.

6. Executable verification is king

Run the tests is the single most reliable verification step available. For any check that depends on agent judgment, find the executable analog.

This site, in practice

Every card's Pattern code must execute under a bare Python 3.13 with an assertion actually running; a missing dependency fails the gate. A green runner only proves an assertion executed, not that it was load-bearing. Review found decorative asserts behind green runs, and closing that took human review plus a hardening pass.

The goal of the model is to do what you want. If you are not precise about your desire, you end up with good intentions that drive you straight off a cliff. In code bases, that manifests as a tangled mesh of well-executed prototypes, unmaintainable under its own weight. In agentic systems, as cascading failures that degrade performance at each hop.

If you take only one principle, take this one. Executable verification is king, because agents are too damn smart. Verification is what it is.

7. Cross-family beats self-verification

Different model family is needed when verification is LLM-based. Self-verification and intra-family verification are systematically biased toward accepting incorrect outputs.

8. Simulate debate

In a single-agent context, instruct the agent to argue against its own output before concluding. Competitive disagreement reveals more than confirmatory review.

9. Isolate verification from ambient state

Assertions must prove the system's actions caused the expected outcome, not that the environment happened to already contain matching data. Delta-based assertions and tagged test data isolate causality.