LLM Evaluation
For engineers preparing for interviews on testing and measuring LLM applications, from RAG assistants to agents. You will be able to explain and compute the core metrics, design and calibrate LLM judges, prove a change is real with confidence intervals, and describe how evals gate releases and monitor production.
Course Content
Prepares you to explain how LLM output quality is measured, why the eval comes before the prompt, and how golden sets, regression gates, agent, adversarial, bias and multi-turn evaluations fit together.
Prepares you to compute and criticise the standard metrics — overlap scores, perplexity, pass@k, exact match and F1 — and to read public benchmarks and leaderboards without being fooled by contamination or saturation.
Prepares you to design, calibrate and defend an LLM-as-judge setup — pointwise or pairwise, its biases, its prompt, its cost, and how to prove with Cohen's kappa that it agrees with humans.
Prepares you to explain how hallucinations are detected and measured — claim-level faithfulness, entailment, SelfCheckGPT, search-based verification and grounding — and how to build a factuality test set that rewards honest abstention.
Prepares you to answer how you would prove one version beats another with confidence intervals, run A/B tests, debug and red-team an LLM system, and keep it healthy in production through reproducible evals, drift detection and the right dashboards.
Prepares you for the system-design questions: how to evaluate a RAG pipeline stage by stage with context precision, context recall and faithfulness, and how to find the weakest component in a multi-stage AI system.