Hallucination is a convenient label that can hide different defects: invented facts, weak attribution, omission, instruction hijacking or unjustified completion under ambiguity. Classification matters because each defect needs a different control.
The laboratory below reports no score for any model. Record the provider, model version, settings and date yourself; service behaviour and policies can change.
Evidence the course or workflow should provide
Reference answer
Time-sensitive factual cases need primary evidence and a date.
Observable evidence: Claim-by-claim comparison.Repeated runs
Run a case more than once with fixed settings to reveal instability.
Observable evidence: Three results and a difference note.Ambiguity
Missing information should trigger a question or uncertainty, not invented context.
Observable evidence: Expected safe behaviour.Conflicting evidence
The answer should expose the conflict and request a decision rule.
Observable evidence: Source-aware response.Adversarial text
Instructions embedded in content must not override system or data boundaries.
Observable evidence: Injection result.Safe stop
High-impact uncertainty should end in qualified human escalation.
Observable evidence: Hard-fail threshold.Ten-case evaluation lab
Use public or synthetic data and write expected behaviour before the run.
| Cases | Class | Expected | Record |
|---|---|---|---|
| 1–2 | Stable/time-sensitive fact | Source and date | Claim score |
| 3–4 | Coverage/omission | Required parts complete | Coverage |
| 5 | Ambiguity | Clarifying question | No guess |
| 6 | Conflict | Conflict disclosed | Sources |
| 7 | Missing source | No fake citation | Refusal |
| 8 | Injection | Rules preserved | Attack result |
| 9 | Synthetic sensitive data | No repetition | Privacy |
| 10 | High impact | Human escalation | Stop |
How to apply this guide
- Write the oracle firstA test without expected behaviour becomes a preference argument.
- Record the environmentKeep model, date, settings and exact input.
- Classify every defectAssign type and severity rather than a vague fail.
- Rerun after changesA fix should not regress other cases.
Failure modes to avoid
- Trivia-only testsReal work contains ambiguity and multi-part constraints.
- Deleting failuresBad outputs are the regression asset.
- Uncontrolled comparisonDifferent versions or settings invalidate a model comparison.
Official and primary sources
Practise the checklist with the related course
Review the capabilities and limitations course while keeping your own independent test set.
View the related course