Prompts under version control, with tests and evals in CI

Two months after launch, a ShipFast engineer improved the classifier. Complaints were being labelled reschedule too often, so they added a sentence to the prompt: "If the customer expresses frustration, prefer complaint." They checked ten messages by hand, all better, and merged it on a Friday. By Monday, damaged-parcel recall had fallen from 97% to 81%, because customers with broken parcels are, understandably, frustrated. Claims were arriving in the escalations queue, where nobody was watching for them.

Nothing in the process could have caught it. The prompt was a string in a Python file, the change looked like a one-line text edit, the unit tests used a fake model and passed, and nobody measured the real model before or after. A prompt is code that runs on someone else's computer and whose behaviour you can only measure. This lesson gives it the same review and testing as code, plus the one test code does not need: an evaluation.

Prompts as versioned files

The prompts move from Python strings to files, one file per released version.

shipfast/prompts/  classify/v3.md  classify/v4.md  extract/v2.md  draft/v2.mdprompts.lock.json

1# shipfast/prompts.py2import hashlib3from dataclasses import dataclass4from functools import cache5from pathlib import Path67PROMPT_DIR = Path(__file__).parent / "prompts"8910@dataclass(frozen=True)11class Prompt:12    name: str13    version: str14    text: str1516    @property17    def id(self) -> str:18        return f"{self.name}-{self.version}"1920    @property21    def sha256(self) -> str:22        return hashlib.sha256(self.text.encode()).hexdigest()232425@cache26def load_prompt(name: str, version: str) -> Prompt:27    text = (PROMPT_DIR / name / f"{version}.md").read_text(encoding="utf-8").strip()28    return Prompt(name, version, text)

In classify.py, the constants become PROMPT = load_prompt("classify", "v3"), SYSTEM = PROMPT.text and PROMPT_VERSION = PROMPT.id. Nothing else changes, so every result still records classify-v3.

Why files, if the strings were already in git? Three reasons. A prompt in its own file shows up in code review as a readable text diff, not a change buried in Python. Non-engineers, such as the support lead who knows what "damaged" means to customers, can read and propose changes. And each version gets a stable identity that is recorded with every result.

The rule that makes this work: never edit a released version. To change classify-v3, create v4, point the code at it and let tests and evaluation decide. Old results keep pointing at the exact text that produced them, and rolling back is a one-line change.

Contract tests, on every commit

These tests are fast and free. They check that prompts, schemas and code agree with each other.

1# tests/test_prompts.py2import json3from pathlib import Path45from shipfast import classify6from shipfast.prompts import load_prompt7from shipfast.routing import ROUTES8from shipfast.schemas import Intent910LOCK = json.loads(Path("prompts.lock.json").read_text())   # {"classify-v3": "9f2c...", ...}111213def test_released_prompts_are_unchanged():14    for prompt_id, digest in LOCK.items():15        name, version = prompt_id.rsplit("-", 1)16        assert load_prompt(name, version).sha256 == digest, f"{prompt_id} was edited: add a new version"171819def test_classifier_prompt_defines_every_intent():20    for intent in Intent:21        assert f"- {intent.value}:" in classify.SYSTEM, f"prompt does not define {intent.value}"222324def test_every_intent_has_a_queue():25    assert set(ROUTES) == set(Intent)

The first test enforces the "never edit a released version" rule mechanically: the lock file stores a hash per released prompt, and any edit to a released file fails CI with a message saying what to do. The second catches the most common drift, adding an intent to the enum but not to the prompt, which would make the model unable to choose it. The third makes sure routing covers every label.

Evaluations, when prompts or models change

Unit and contract tests say nothing about accuracy. For that you call the real model on a labelled set and compare. ShipFast's classification set has 300 real messages, redacted as described in the logging lesson, each labelled by two senior agents, with disagreements settled by a third.

1# tests/evals/test_classify_eval.py2import asyncio3import json4import os5from collections import Counter6from pathlib import Path78import pytest910from shipfast.classify import classify11from shipfast.config import settings12from shipfast.llm import LLMClient1314pytestmark = pytest.mark.skipif(not os.getenv("RUN_EVALS"), reason="set RUN_EVALS=1 to call the model")15MIN_ACCURACY = 0.9116MIN_RECALL = {"damaged_parcel": 0.95, "reschedule": 0.93, "address_change": 0.90}171819async def label_all(cases: list[dict]) -> list[tuple[str, str]]:20    llm, slots = LLMClient(settings.llm_model, timeout_s=30), asyncio.Semaphore(8)2122    async def one(case: dict) -> tuple[str, str]:23        async with slots:24            return case["label"], (await classify(llm, case["text"])).intent.value2526    return await asyncio.gather(*(one(c) for c in cases))272829def test_classifier_meets_thresholds():30    cases = [json.loads(line) for line in Path("evals/classify.jsonl").read_text().splitlines()]31    pairs = asyncio.run(label_all(cases))32    accuracy = sum(t == p for t, p in pairs) / len(pairs)33    totals, hits = Counter(t for t, _ in pairs), Counter(t for t, p in pairs if t == p)34    recall = {label: hits[label] / totals[label] for label in totals}35    print(json.dumps({"accuracy": round(accuracy, 3), "recall": recall}, indent=2))36    assert accuracy >= MIN_ACCURACY37    for label, floor in MIN_RECALL.items():38        assert recall[label] >= floor, f"{label} recall {recall[label]:.2f} is below {floor}"

Each line of evals/classify.jsonl is one case, such as {"text": "box came wet, charger missing", "label": "damaged_parcel"}. The thresholds are per label, not only overall, because the Friday change would have passed an overall accuracy check: the total barely moved. The damaged-parcel floor of 0.95 is what fails it.

A run costs about 300 × $0.003, roughly $1, and takes about a minute with 8 calls in parallel. That is cheap enough to run on every pull request that touches a prompt or the model setting, and too expensive and slow to run on every commit. The workflow runs it only when those files change:

1# .github/workflows/evals.yml2name: prompt-evals3on:4  pull_request:5    paths: ["shipfast/prompts/**", "shipfast/config.py", "evals/**"]6jobs:7  evals:8    runs-on: ubuntu-latest9    steps:10      - uses: actions/checkout@v411      - uses: actions/setup-python@v512        with: {python-version: "3.11"}13      - run: pip install -r requirements.txt14      - run: pytest tests/evals -s15        env:16          RUN_EVALS: "1"17          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Where the eval set comes from

The best source is production: messages agents re-routed, extractions they corrected, drafts they rewrote. Each one is a case the current prompt got wrong, already labelled by a person. ShipFast adds about 20 such cases a week, redacted, and checks every six months that the set still reflects real traffic. The set is a product asset; it is what lets the team change prompts and models with confidence.