A longer, more sophisticated prompt can lose to a simple instruction, and the opposite can also happen. The prompt text is not evidence; repeatable behaviour, failure handling and total review cost are.
The A/B sheet below is deliberately blank. It is a testing instrument, not fabricated performance data or a customer case study. Replace every field with a recorded run from your own context.
Evidence the course or workflow should provide
Pre-register criteria
Set weights for accuracy, coverage, structure, voice and safety before generation.
Observable evidence: A timestamped rubric that predates the outputs.Include edge cases
Missing, conflicting and out-of-scope inputs should test safe failure rather than reward confident completion.
Observable evidence: At least three edge cases and one stop case.Blind the comparison
Hide prompt identity and randomise A/B order to reduce brand and author preference.
Observable evidence: Anonymous labels and recorded ordering.Inspect disagreement
Large reviewer gaps often indicate ambiguous criteria rather than a bad reviewer.
Observable evidence: Reason notes and a revised rubric.Measure repair cost
A fluent answer that requires extensive fact repair is not the cheaper result.
Observable evidence: Review and editing minutes beside the score.Blank blind A/B scorecard
The cells are intentionally empty. Do not convert placeholders into public performance claims.
| Criterion | Weight | Output A | Output B |
|---|---|---|---|
| Accuracy | 3 | __/3 | __/3 |
| Coverage | 2 | __/2 | __/2 |
| Format | 1 | __/1 | __/1 |
| Voice | 1 | __/1 | __/1 |
| Safety | 3 | __/3 | __/3 |
| Repair time | Separate | __ min | __ min |
How to apply this guide
- Build ten casesStart with six normal, three edge and one refusal or escalation case.
- Run two anonymous versionsRecord model, settings and time while holding inputs constant.
- Classify defectsFactual, omission, format, voice and boundary errors require different fixes.
- Keep regression testsThe next revision must not reintroduce a previously fixed failure.
Failure modes to avoid
- Post-hoc criteriaThe scoring rule becomes easy to move toward a preferred answer.
- Unsafe averagingA safety breach cannot be cancelled by polished tone.
- Single reviewerUnmeasured preference can dominate the result.
Official and primary sources
Apply the rubric to foundation exercises
Review the AI Foundations path and ask for evaluation evidence before treating any prompt lesson as complete.
Explore AI Foundations