Course Content
Prompt Engineering Mastery
6 sections · 32 lessons
How do you measure whether a prompt is effective?
What you need to know
"It looked good on three examples" is not evidence. A prompt change that fixes one case often breaks two others. You only see that with a fixed set of cases scored the same way each time.
- Collect inputs — 50 to 300 real examples, sampled from production, plus known hard cases and past failures.
- Define "good" — an expected label, expected fields, or a written rubric for free text.
- Pick metrics — the table below.
- Automate checks — format, schema, length, forbidden content: cheap and exact.
- Judge the rest — an LLM judge with a rubric, validated against humans on a sample.
- Compare and gate — run old and new prompts on the same set; ship only if the score rises and nothing important regresses.
Metrics by task
| Task | Metric |
|---|---|
| Classification | Accuracy, per-class precision and recall, F1, confusion matrix |
| Extraction | Exact match per field, precision and recall per field |
| JSON output | Percent that parse and match the schema |
| RAG answers | Groundedness (claims supported by the source), answer relevance |
| Free text | Rubric score from an LLM judge, plus human spot checks |
| Code | Percent that pass unit tests |
Always add cost (tokens per call) and latency (p50 and p95). A prompt that adds 2,000 tokens for one point of accuracy may not be worth it.
LLM-as-judge, done properly
A second model scores outputs against a written rubric ("1 if every claim appears in the source, else 0"). Binary or 1–3 scales are more stable than 1–10. Before trusting it, compare its scores with human labels on 50 to 100 items; if agreement is low, fix the rubric.
1def evaluate(prompt_version, cases, classify):2 correct = 03 for case in cases:4 predicted = classify(prompt_version, case["text"])5 correct += predicted == case["expected"]6 return correct / len(cases)78# v2 must beat v1 on the same cases before it ships9print(evaluate("v1", cases, classify), evaluate("v2", cases, classify))The loop is trivial on purpose: the value is in the fixed cases list, not in the code. Run it in CI whenever the prompt file changes.
A real-life example
An invoice-extraction team changes its prompt to handle GST invoices from new suppliers. On the 12 new invoices it looked perfect. The eval set of 400 invoices tells a different story:
| Field | v1 | v2 |
|---|---|---|
| Invoice number | 98% | 99% |
| GSTIN | 91% | 97% |
| Total amount | 96% | 89% |
| JSON valid | 100% | 100% |
The new wording made the model pick the pre-tax subtotal as the total on multi-page invoices. The team adds "the total is the final payable amount including tax" and re-runs: total amount returns to 97% with the GSTIN gain kept. Without the eval set, a 7-point drop on the most important field would have shipped.
Follow-up questions to expect
- "How big should the eval set be?" — Big enough that one case is not a large share of the score; 100 to 300 cases is common to start. Add every production failure you fix.
- "How do you avoid overfitting the prompt to the eval set?" — Keep a held-out set you do not look at while tuning, and refresh it with new production samples.
- "How do you evaluate after launch?" — Sample live traffic, run the same automated checks and judge, and watch user signals such as edits, escalations or thumbs-down.