Schema Validation Reliability for AI-Drafted Content and Automation Recovery
Test how reliably the blueprint's automated checks catch mistakes in AI-drafted content, and how reliably its scheduled safety net recovers automation steps that a missed event leaves undone.
Hypothesis
Deterministic checks (required fields, diagram rules, and every number traced back to the author's own notes) will catch nearly all seeded factual and structural errors before a human sees the draft, and a scheduled sweep will recover every missed automation within one cycle.
Problem
The blueprint lets AI draft content, but only a person publishes. That rule only holds up if the checks in front of the reviewer actually catch errors, and if automations that depend on events don't silently skip work when an event is missed.
Setup
Validation test: take a set of correct drafts and seed known errors (missing required fields, invalid diagram placements, invented numbers not in the source notes, contradictory figures). Run the checks and record which errors they catch. Recovery test: deliberately drop a share of publish events so their follow-up steps (refresh, asset generation) don't run, then let the scheduled sweep run and record how many are recovered and how long it takes. Metrics: error catch rate by error type, false alarms on clean drafts, recovery rate and time to recovery.
The seeded-error test
- 1. Input
Clean drafts
- 2. Action
Seed known errors
Missing fields, invented numbers
- 3. Decision
Run content checks
- 4. Output
Caught vs missed, by type
Input
Seeded drafts with a known error list; a controlled run of publish events with a known set of dropped ones.
Output
Catch rate by error type and a recovery timeline.
Decision
Status: designed, not run. The checks and the sweep are in use; their reliability has not been measured.
Next Step
Prepare the seeded drafts and the event-drop harness, run both tests, and publish the results here.
Use Cases
- ▸Human-in-the-loop AI content pipelines
- ▸Webhook-driven automations that need a safety net