Agent evals and release criteria

pydantic-evals style gate — borrowed from maverick, shipped in evals/.

playstepchallengeinstructor

Run eval gateIDLE

Pass rate: 0% · cases: 3

Run locally

pnpm evals:run
pnpm evals:write-metrics   # needs compose + psql
pnpm dev:radar             # Eval pass rate card updates

Golden cases cover sentiment scoring, high-risk detection, and secret-leak heuristics.

Release criteria

• 100% pass on golden set before merge
• No silent scorer changes without new cases
• Radar KPI reflects latest run — managers see drift

Lab

Break a keyword in enrich.py, rerun evals, watch gate fail and KPI drop.

Teach-back prompt

Why exit code 1 in CI beats a Slack thread saying “looks fine”?