Applied AI Engineering: From Prompt to Production

Course Overview

Advanced

Free Course

For engineers who want the whole applied-AI stack in one course, from prompts and retrieval to agents, fine-tuning, evaluation, deployment, multimodal input and safety. You will build PolicyPal, an HR and IT policy assistant for a 2,000-person company, and take it from a single prompt to a traced, evaluated, guarded production service.

Instructor: MantraMindAI

Sections: 9

Course Content

Section 1: Foundations and the model landscape

See the finished PolicyPal, learn how a language model really behaves, choose one with evidence, and carry the cost and latency numbers you will use for the rest of the course.

1. What you will build: PolicyPal, from prompt to production

2. How LLMs work, in the detail an engineer needs

3. Choosing a model: the landscape and the trade-offs

4. Context, cost and latency: the numbers to carry in your head

Section 2: Prompt engineering for production

Turn a one-line prompt into a versioned contract: a system prompt that holds up against real users, JSON the application can trust, and examples and reasoning used only where they earn their cost.

1. System prompts and templates that survive real users

2. Structured outputs you can parse

3. Examples and reasoning: few-shot and step-by-step, used carefully

Section 3: Retrieval-augmented generation

Give PolicyPal Harbourline's 400 policy PDFs: build the retrieval pipeline, chunk documents so answers stay whole, combine keyword and semantic search with a reranker, and make every sentence traceable to a source.

1. The RAG pipeline end to end

2. Chunking documents so answers survive

3. Hybrid search and reranking

4. Grounding and citations

Section 4: Agents and tool use

Let PolicyPal act as well as answer: define tools the model uses correctly, run them safely, wrap them in a bounded loop, and decide where autonomy is worth its cost.

1. Tools and function calling

2. The agent loop and planning

3. Workflows or agents: choosing the right amount of autonomy

Section 5: Fine-tuning and customisation

Learn what fine-tuning can and cannot do, train a small LoRA router for PolicyPal with transformers and peft, and prove with a held-out test that it beats the prompted baseline where it matters.

1. When to fine-tune, and when not to

2. LoRA and parameter-efficient fine-tuning in practice

3. Preparing data and comparing against the prompted baseline

Section 6: Evaluation and testing

Replace "it seems better" with evidence: build a 250-case eval suite, measure each stage of the pipeline separately, use a model as a judge only after calibrating it, and test changes on real traffic without fooling yourself.

1. Building evaluation sets

2. Metrics for RAG and agents

3. Model-as-judge done carefully

4. Online evaluation and A/B tests

Section 7: Deployment and LLMOps

Run PolicyPal as a real service: a FastAPI app with timeouts, degraded modes and versioned releases, fast through streaming and careful caching, and observable through traces, cost dashboards and quality signals.

1. Serving the app as a service

2. Latency and throughput: caching, batching, streaming

3. Observability: traces, costs and quality signals

Section 8: Multimodal AI

Extend PolicyPal beyond typed text: make scanned policies searchable, read screenshots and scanned forms into validated fields, and answer spoken questions over the phone with a latency budget that feels like a conversation.

1. Vision: documents, screenshots and scanned forms

2. Speech in and out

Section 9: Responsible and safe AI engineering

Attack PolicyPal before others do, control where personal data goes, and assemble every defence in the course into layers that hold when any single one fails.

1. Adversarial attacks and red-teaming

2. Privacy and personal data

3. A layered guardrail architecture, and the road ahead