Day 6. The two hallucinated doses from last Wednesday are still the load-bearing scar on this engagement. You wrote the eval (Ch1), fixed context (Ch2), dialed thinking (Ch3), picked examples (Ch4), and tightened the output schema (Ch5). Legal has signed off on resuming the pilot — but only if today's prompt-plus-defenses keeps hallucination rate under 1% on the eval set, with zero invented doses. Today you stop treating 'don't make stuff up' as a wish and start naming the defenses: forced citations, an 'I don't know' budget that the model is told to spend, a verifier loop the runtime actually enforces, and a refusal hierarchy that survives a red-team. The agent will lie if you let it. You will not let it.
You'll walk out able to
- Name the three failure modes that produced last Wednesday's chart hallucination — hallucination, sycophancy, premature confidence — and the defense that targets each
- Drive a model to refuse a fact that wasn't in the source, using a verifier you can grep for
- Pick the cite-or-refuse defense over softer alternatives and justify why empirically
- Read two prompt drafts and pick the one whose hallucination rate is provably lower
- Survive a 5-attempt red-team against a hardened medical-scribe system prompt without leaking a dose
- Write an adversarial eval set that distinguishes cite-or-refuse from cosmetic safety language
- Run a spec linter on a 'be confident' agent folder and name the load-bearing weakness
Your first task
The dose that wasn't there.
9:02am. Legal's counsel, Renata, is at her desk with the printout of last Wednesday's chart. You're running the candidate v4 prompt against a held-out transcript she pulled from the eval bench. Mateo is watching over your shoulder. The model emits a summary in 1.1 seconds. It looks great. Read it like Renata is going to read it.
What you'll answer
Read the session below. What's the one diagnosis you give the model's behavior?
Free · no card · we save your progress
Save your progress & earn the certificate
You've seen the opening — keep going free. 7 tasks · auto-graded · finish in 80 min and the certificate is yours. Free forever · no card.
No password. To sign in on your phone, request an emailed code on the sign-in page.
Already started? Sign in with an emailed code instead.