TT Claude Memory

feedback_conftest_rego_dogfood_self_validation_pattern

Properties1
descriptionWhen authoring a schema + its validator concurrently, run the validator against the schema-defining doc FIRST. The validator catching its own author's miss is the cheapest empirical signal that the convention is workable. Caught at Wave C Del 5 2026-05-26 when frontmatter-conventions.md itself was missing the v1.5-REQUIRED `description` field.

4 min read

The rule (locked 2026-05-26): When authoring a schema + its mechanical validator concurrently (per the “PAIRS WITH” pattern in Wave C LP §2.5: Conftest+Rego pairs with frontmatter v1.5 spec), run the validator against the schema-defining doc as the FIRST validation target. The schema-validates-its-own-author paradigm catches authoring oversights at the cheapest point — before the convention is propagated anywhere downstream.

Why (Rich-elevation 2026-05-26 post-Wave-C-close after coordinator handoff §3.1 caught this as a NEW lesson worth durable capture):

Wave C Del 5 implementation: 4 Rego policies + wrapper script smoke-tested on 3 docs. First smoke-test target was frontmatter-conventions.md itself (the file that defines the v1.5 convention). Empirical result:

warning │ /home/richardd/testatetech/docs-strategy/frontmatter-conventions.md │ main │ required field missing: "description" (per frontmatter-conventions.md v1.5 §1)

The canonical spec file was missing the field it had just declared REQUIRED. The validator caught the gap before any backfill ran, before any cascade, before any consumer document was authored against the new convention. Fix landed in-commit (added 1-line description: to the file’s frontmatter at c930954).

How to apply (general pattern; not just frontmatter):

  • Schema + validator paired authoring (Conftest+Rego per Wave C; JSON Schema + ajv; pyshacl shape + RDF; LinkML schema + linkml-validate; OpenAPI spec + Schemathesis; Catala rule + Mopsa-c verifier):

    1. Author the schema (e.g., frontmatter-conventions.md v1.5)
    2. Author the validator (e.g., policies/frontmatter/*.rego)
    3. Run the validator against the schema-defining doc itself as smoke-test #1
    4. Any finding here is a HIGH-confidence authoring oversight — the convention’s first reader is the convention’s first violator
    5. Fix in-commit before propagating

  • Spec + reference implementation paired authoring (Cedar policy + sample data; Arazzo workflow + Schemathesis suite; CITATION.cff schema + cite-tool.sh):

    • Same pattern: run the reference impl against the spec doc; fix gaps in-commit

  • Cascade-Q + arch-state row paired authoring (TT-specific):

    • Run the new cascade-Q file’s phase_q_relevance: field through the pin-drift hook
    • Verify the arch-state row’s amendment-number is next-available, not duplicated

Strategic value:

  1. Dogfood = highest-confidence finding type — when the validator catches its own author, the convention’s mental model is genuinely incomplete. Validators that pass on their own author’s doc are minimally calibrated.
  2. Cheapest fix point — pre-commit, pre-cascade, pre-propagation. Catching at this stage prevents N downstream consumers from inheriting the gap.
  3. Acquirer-DD signal — shows the validator was designed against the actual convention, not against an idealised version. Methodology-quality evidence.

Anti-pattern this rule prevents:

  • “I’ll smoke-test on consumer docs first because they’re what the validator will mostly run on” → NO. Consumer docs may legitimately not yet have the field (backfill pending); the spec-defining doc is the one place the field MUST exist for the convention to be valid. Test there first.
  • “The schema doc is meta-recursive; testing the schema against itself is weird” → NO. The schema doc is a working example of its own convention. If it doesn’t conform, the convention is incomplete or the author missed a field.
  • “I’ll add the field to the spec doc after the cascade lands” → NO. Cascading a convention that the canonical spec violates means every consumer gets a gap by reference. Fix the spec first.

Pattern recurrence (items where this discipline should apply):

  • ✓ Wave C Del 5 Conftest+Rego — validator caught description missing on frontmatter-conventions.md itself; fixed in-commit
  • ⚠️ Wave H pin-drift OPA migration (Phase-1.5+) — Rego policies will validate frontmatter pins; first smoke-test target MUST be the pin-drift convention’s spec doc (the §3.5 part of frontmatter-conventions.md). Pre-validation spike scope should include the dogfood test.
  • ⚠️ Wave I arch-state RDF/Turtle (Phase-1.5+) — RDF schema + Turtle exemplar must validate against itself (every amendment registry row matches the SHACL shape that defines amendment registry rows)
  • ⚠️ Future Anthropic Skills SKILL.md schemas — when a SKILL.md frontmatter contract is defined, the schema’s defining file must conform to the schema (S9 + Wave A.1 transplant context)

Companion locks: feedback-pilot-belongs-in-spike-phase-not-execution (sister discipline — empirical validation upstream of wave commit); feedback-verify-framework-extension-maintenance-before-lock (upstream — substrate verification before lock); verifying-before-author (skill — 6-method substrate verification toolkit).

Sister patterns in the dogfood family:

  • Test runners running on their own test files (pytest’s test_pytest.py)
  • Compilers compiling themselves (self-hosting bootstrap)
  • Documentation tools rendering their own docs (mkdocs building the mkdocs-material site)
  • Linters lint-clean on their own source (ruff repo passing ruff)

The Wave C Conftest+Rego dogfood is the same pattern applied at the convention-and-validator pair level. Rule of thumb: if you author X + author validator-for-X, run validator-for-X against X itself before propagating either.

Graph View

Backlinks