MLOps for AI

Course Content

MLOps for AI

4 sections · 9 lessons

Capstone Project: End-to-End MLOps Pipeline for Churn Prediction

Here is the brief you are working to. A telecoms company has 10,000 customers on file. Roughly 26.5% of them cancel within a year. Marketing wants a list, refreshed weekly, of who is about to leave, so they can send a retention offer that costs USD 20 per customer and succeeds about 30% of the time. A retained customer is worth USD 400 in remaining margin.

Most people who attempt this produce a notebook. It loads the CSV, engineers some features, trains a gradient-boosted tree, prints AUC: 0.84, and stops. That notebook is worth almost nothing to the company, because marketing cannot run it, nobody can tell which version produced last week's list, and when it starts performing badly no one will notice.

The gap between that notebook and a system is what you are going to close. Four parts, each producing an artefact the next one consumes:

PartYou buildIt producesConsumed by
1. Track and registerMulti-run training with MLflow, a promotion gateA registered model version with a @champion aliasThe service, which resolves the alias
2. PackageA Flask inference service and a hardened DockerfileAn image identified by digestThe deployment manifest
3. DeployDeployment, Service, HPA manifestsA running, scaling, health-checked endpointReal traffic
4. ObservePrometheus metrics, alerts, documentationEvidence the thing still worksThe next retraining decision

The test of this project is not the AUC. It is whether a colleague who has never seen the repository can find out what is deployed, why it was chosen, and how to undo it — using only what you built.

Choosing the threshold with money, not with F11,80552426,780 USD1,31048031,400 USD70033526,200 USD26316714,780 USDCustomers flaggedActual churnersNet valueThreshold 0.05Threshold 0.15Threshold 0.30Threshold 0.50A retained customer is worth 400 USD; the offer costs 20 USD and works 30 percent of the time.
The threshold that maximises F1 is almost never the one that maximises money — pick it off the profit curve.

Part 1 — Generate data you can reason about

Use synthetic data with a known structure. Real datasets hide their generating process; synthetic data lets you verify that your pipeline recovers signal you deliberately planted, which is how you catch a broken feature pipeline early.

Python
import numpy as np, pandas as pddef make_customers(n: int = 10_000, seed: int = 42) -> pd.DataFrame:    rng = np.random.default_rng(seed)    tenure = rng.integers(1, 73, n)    monthly = np.round(rng.normal(70, 25, n).clip(18.5, 120), 2)    contract = rng.choice(["month-to-month", "one-year", "two-year"],                          n, p=[0.55, 0.24, 0.21])    support_calls = rng.poisson(1.2, n)    has_fibre = rng.random(n) < 0.44    # Planted signal: a log-odds model we can check the trained model recovers.    logit = (-1.97             - 0.030 * tenure             + 0.011 * monthly             + 0.34 * support_calls             + 0.42 * has_fibre             + np.where(contract == "month-to-month", 1.15,               np.where(contract == "one-year", 0.10, -0.85)))    churn = rng.random(n) < 1 / (1 + np.exp(-logit))    return pd.DataFrame({        "customer_id": [f"C{i:06d}" for i in range(n)],        "tenure_months": tenure, "monthly_charges": monthly,        "contract": contract, "support_calls": support_calls,        "has_fibre": has_fibre, "churned": churn.astype(int),    })df = make_customers()print(len(df), df["churned"].mean().round(4))     # 10000 0.266

Two coefficients are worth remembering for later: contract type dominates (a swing of 2.0 in log-odds between two-year and month-to-month), and tenure protects (−0.030 per month, so 24 months of tenure is −0.72). If your trained model's feature importances do not put contract and tenure_months at the top, something in your pipeline is broken — a shuffled join, a leaked column, a mis-encoded category. Synthetic data gives you that check for free.

Train several candidates, log all of them

Python
import mlflow, mlflow.sklearn, subprocess, hashlibfrom mlflow.models import infer_signaturefrom sklearn.compose import ColumnTransformerfrom sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifierfrom sklearn.linear_model import LogisticRegressionfrom sklearn.metrics import roc_auc_score, average_precision_scorefrom sklearn.model_selection import train_test_splitfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import OneHotEncoder, StandardScalerNUM = ["tenure_months", "monthly_charges", "support_calls"]CAT = ["contract", "has_fibre"]def build(estimator) -> Pipeline:    pre = ColumnTransformer([        ("num", StandardScaler(), NUM),        ("cat", OneHotEncoder(handle_unknown="infrequent_if_exist",                              min_frequency=20), CAT),    ])    return Pipeline([("pre", pre), ("clf", estimator)])X = df.drop(columns=["churned", "customer_id"])y = df["churned"]X_tr, X_te, y_tr, y_te = train_test_split(    X, y, test_size=0.2, stratify=y, random_state=42)data_hash = hashlib.sha256(    pd.util.hash_pandas_object(df, index=True).values.tobytes()).hexdigest()[:16]git_sha = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()CANDIDATES = {    "logreg":  LogisticRegression(max_iter=1000, C=1.0, random_state=42),    "rf":      RandomForestClassifier(n_estimators=400, max_depth=8,                                      min_samples_leaf=20, random_state=42),    "gbm":     GradientBoostingClassifier(n_estimators=300, max_depth=3,                                          learning_rate=0.05, random_state=42),}mlflow.set_experiment("churn-prediction")for name, est in CANDIDATES.items():    with mlflow.start_run(run_name=name):        pipe = build(est).fit(X_tr, y_tr)        prob = pipe.predict_proba(X_te)[:, 1]        mlflow.log_params({"model_type": name, **{            f"clf.{k}": v for k, v in est.get_params().items()            if isinstance(v, (int, float, str, bool)) or v is None}})        mlflow.set_tags({"git_sha": git_sha, "data_hash": data_hash,                         "rows": len(df)})        mlflow.log_metrics({            "test_auc": roc_auc_score(y_te, prob),            "test_ap":  average_precision_score(y_te, prob),            "test_pos_rate": float(y_te.mean()),        })        # Slice metrics: the average hides the segments marketing cares about.        for label, mask in {            "month_to_month": X_te["contract"] == "month-to-month",            "new_customers":  X_te["tenure_months"] <= 6,            "high_value":     X_te["monthly_charges"] > 90,        }.items():            m = mask.to_numpy()            if m.sum() >= 100 and y_te[m].nunique() == 2:                mlflow.log_metric(f"slice.{label}.auc",                                  roc_auc_score(y_te[m], prob[m]))                mlflow.log_metric(f"slice.{label}.n", int(m.sum()))        mlflow.sklearn.log_model(            sk_model=pipe, name="model",            signature=infer_signature(X_tr, pipe.predict_proba(X_tr)),            input_example=X_tr.iloc[:3],            pyfunc_predict_fn="predict_proba")   # the service needs probabilities

Logging the whole Pipeline rather than the bare estimator is the decision that prevents the most common bug in this project. If you log only the classifier, your serving code has to reimplement the scaling and one-hot encoding — and the day someone adds a category or changes the scaler, training and serving drift apart silently. One object in, one object out.

The pyfunc_predict_fn="predict_proba" argument matters in Part 2. The service loads the model through MLflow's generic pyfunc interface, and without that argument its predict returns hard 0/1 labels rather than the probabilities the threshold is applied to.

Choose a threshold with money, not with F1

AUC ranks candidates. It does not tell marketing whom to call. That needs a threshold, and the right threshold comes from the economics in the brief.

The test set has 2,000 customers, of whom 532 churn. Score them with the GBM (test AUC about 0.77; the planted signal is deliberately noisy) and count what happens at four thresholds. These are the numbers the code above produces with scikit-learn 1.9; other versions may differ slightly.

ThresholdFlaggedTrue churners caughtOffer costExpected retained (30%)Margin savedNet
0.50263167USD 5,26050.1USD 20,040USD 14,780
0.30700335USD 14,000100.5USD 40,200USD 26,200
0.151,310480USD 26,200144USD 57,600USD 31,400
0.051,805524USD 36,100157.2USD 62,880USD 26,780

Check the arithmetic on the 0.15 row: 1,310 offers × USD 20 = USD 26,200 spent; 480 true churners × 0.30 retention success = 144 customers kept; 144 × USD 400 = USD 57,600; net USD 31,400.

The default threshold of 0.5 leaves 31,400 − 14,780 = USD 16,620 per cycle on the table — because it optimises for being right, and the business is paying for churners caught. Note also that 0.05 is worse than 0.15 despite catching more churners: the extra 495 offers cost USD 9,900 and reach only 44 more churners, worth 44 × 0.30 × 400 = USD 5,280. There is a maximum, and you can predict roughly where it sits before running anything. An offer pays for itself when P(churn) × 0.30 × 400 is more than 20, that is when P(churn) is above 20 / 120 ≈ 0.17. For a reasonably calibrated model the best threshold therefore sits near 0.17, far from any default; the sweep below finds a flat optimum between 0.14 and 0.18, and this project uses 0.15, a round number inside it.

The model is chosen by its ranking quality; the threshold is chosen by the cost of being wrong. Only the first of those two ever appears on a leaderboard.

Python
import numpy as npdef best_threshold(y_true, prob, offer_cost=20, margin=400, success=0.30):    rows = []    for t in np.arange(0.05, 0.95, 0.01):        flagged = prob >= t        tp = int((flagged & (y_true == 1)).sum())        net = tp * success * margin - int(flagged.sum()) * offer_cost        rows.append((round(float(t), 2), int(flagged.sum()), tp, round(net, 2)))    return max(rows, key=lambda r: r[3])

Log the chosen threshold as a parameter of the run. A probability without its decision threshold is not a decision, and a metric computed at an unrecorded threshold is not evidence.

Register the champion

Python
from mlflow import MlflowClientclient = MlflowClient()runs = mlflow.search_runs(experiment_names=["churn-prediction"],                          filter_string=f"tags.data_hash = '{data_hash}'",                          order_by=["metrics.test_auc DESC"])best = runs.iloc[0]mv = mlflow.register_model(f"runs:/{best.run_id}/model", "churn-predictor")for k, v in {"git_sha": git_sha, "data_hash": data_hash,             "test_auc": f"{best['metrics.test_auc']:.4f}",             "threshold": "0.15",             "worst_slice_auc": f"{min(best[c] for c in runs.columns if c.startswith('metrics.slice.') and c.endswith('.auc')):.4f}"}.items():    client.set_model_version_tag("churn-predictor", mv.version, k, v)# Remember what we are replacing BEFORE we replace it.try:    prev = client.get_model_version_by_alias("churn-predictor", "champion")    client.set_registered_model_alias("churn-predictor", "previous-champion",                                      prev.version)except mlflow.exceptions.MlflowException:    passclient.set_registered_model_alias("churn-predictor", "champion", mv.version)

The data_hash filter on the search is not optional. Ranking runs across different data snapshots produces a leaderboard whose winner is whoever got the easiest split, and it looks exactly like a result.

The previous-champion alias set one line before promotion is what turns a future rollback from an investigation into a lookup.

Part 2 — A service, and an image worth deploying

Python
# src/churn/api.pyimport os, threading, time, loggingimport mlflow, pandas as pdfrom flask import Flask, request, jsonifyfrom prometheus_client import Counter, Histogram, Gauge, generate_latestMODEL_NAME = os.environ.get("MODEL_NAME", "churn-predictor")ALIAS = os.environ.get("MODEL_ALIAS", "champion")THRESHOLD = float(os.environ.get("THRESHOLD", "0.15"))REQUESTS = Counter("churn_requests_total", "Requests", ["version", "status"])LATENCY = Histogram("churn_latency_seconds", "Latency", ["version"],                    buckets=(.005, .01, .025, .05, .1, .25, .5, 1, 2.5))SCORE = Histogram("churn_score", "Predicted probability",                  buckets=[i / 20 for i in range(21)])READY = Gauge("churn_ready", "1 when a model is loaded")app = Flask(__name__)log = logging.getLogger("churn")REQUIRED = ["tenure_months", "monthly_charges", "contract",            "support_calls", "has_fibre"]class ModelHolder:    def __init__(self):        self._lock = threading.RLock()        self.model = None        self.version = "none"        self.refresh()        threading.Thread(target=self._poll, daemon=True).start()    def refresh(self):        c = mlflow.MlflowClient()        mv = c.get_model_version_by_alias(MODEL_NAME, ALIAS)        if mv.version == self.version:            return        loaded = mlflow.pyfunc.load_model(f"models:/{MODEL_NAME}@{ALIAS}")        with self._lock:                      # swap only after a clean load            self.model, self.version = loaded, mv.version        READY.set(1)        log.info("loaded %s version %s", MODEL_NAME, mv.version)    def _poll(self):        while True:            time.sleep(60)            try:                self.refresh()            except Exception as exc:                log.warning("refresh failed, keeping v%s: %s", self.version, exc)    def predict(self, frame):        with self._lock:            return self.model.predict(frame), self.versionholder = ModelHolder()@app.get("/health")def health():    return jsonify(status="ok"), 200          # process is alive@app.get("/ready")def ready():    ok = holder.model is not None    return jsonify(ready=ok, version=holder.version), (200 if ok else 503)@app.post("/predict")def predict():    payload = request.get_json(silent=True)    if payload is None:        REQUESTS.labels(holder.version, "bad_request").inc()        return jsonify(error="body must be JSON"), 400    rows = payload if isinstance(payload, list) else [payload]    if len(rows) > 1000:        REQUESTS.labels(holder.version, "too_large").inc()        return jsonify(error="max 1000 rows per request"), 413    missing = [f for f in REQUIRED if f not in rows[0]]    if missing:        REQUESTS.labels(holder.version, "bad_request").inc()        return jsonify(error=f"missing fields: {missing}"), 400    try:        with LATENCY.labels(holder.version).time():            proba, version = holder.predict(pd.DataFrame(rows)[REQUIRED])        scores = [float(p[1]) for p in proba]    except Exception:        REQUESTS.labels(holder.version, "error").inc()        log.exception("prediction failed")        return jsonify(error="internal error"), 500    for s in scores:        SCORE.observe(s)    REQUESTS.labels(version, "ok").inc()    return jsonify(model_version=version, threshold=THRESHOLD,                   predictions=[{"score": round(s, 6),                                 "flag": s >= THRESHOLD} for s in scores])@app.get("/metrics")def metrics():    return generate_latest(), 200, {"Content-Type": "text/plain"}

Four things in that file exist because of specific production failures.

  • /health and /ready are different endpoints. Health means the process is alive; readiness means it can serve. A pod reloading its model should leave the load-balancer rotation without being restarted, and conflating the two makes that impossible.
  • The 1,000-row cap. This is the sale-day OOM in miniature. An unbounded batch endpoint lets one caller allocate arbitrary memory and take the pod down for everyone.
  • Every response carries model_version. Without it you cannot attribute a change in downstream behaviour to a deployment.
  • The generic 500 body. Returning a stack trace to a caller leaks file paths, library versions and sometimes data. Log the detail; return nothing.
Text
# docker/serve.DockerfileFROM python:3.11-slim@sha256:2f4c9d3b71e08a5c6d4bf2190ac83e7d5b6f01c2a9e4 AS builderWORKDIR /buildRUN apt-get update && apt-get install -y --no-install-recommends build-essential \    && rm -rf /var/lib/apt/lists/*COPY requirements-serve.txt .RUN pip install --no-cache-dir --prefix=/install -r requirements-serve.txtFROM python:3.11-slim@sha256:2f4c9d3b71e08a5c6d4bf2190ac83e7d5b6f01c2a9e4WORKDIR /appCOPY --from=builder /install /usr/localCOPY src/churn/ ./churn/RUN useradd --create-home --uid 1001 appuserUSER appuserENV PYTHONUNBUFFERED=1 OMP_NUM_THREADS=2 THRESHOLD=0.15EXPOSE 8000CMD ["gunicorn", "-w", "2", "-b", "0.0.0.0:8000", "--timeout", "60", \     "--access-logfile", "-", "churn.api:app"]

Copy requirements-serve.txt before the source so the dependency layer stays cached when only code changes — that is the difference between an 8-second rebuild and a 6-minute one. OMP_NUM_THREADS=2 matches the CPU limit set in the deployment; leave it unset and the BLAS library spawns one thread per host core, burns the container's CPU quota in a fraction of each scheduling period, and produces latency plateaus that no profiler explains.

Never run flask run in production. It is a single-threaded development server with no timeout handling. Gunicorn with two workers gives you process isolation, so one wedged request does not stall the other worker.

Part 3 — Deploy it so it survives Monday

Text
apiVersion: apps/v1kind: Deploymentmetadata: {name: churn-api, namespace: ml-production}spec:  replicas: 3  strategy:    rollingUpdate: {maxSurge: 1, maxUnavailable: 0}  selector: {matchLabels: {app: churn-api}}  template:    metadata:      labels: {app: churn-api}      annotations: {prometheus.io/scrape: "true", prometheus.io/port: "8000"}    spec:      containers:      - name: api        image: ghcr.io/acme/churn-api@sha256:8b2e1f...        ports: [{containerPort: 8000}]        env:        - {name: MODEL_ALIAS, value: champion}        - {name: THRESHOLD, value: "0.15"}        - name: MLFLOW_TRACKING_URI          valueFrom: {configMapKeyRef: {name: ml-config, key: tracking_uri}}        resources:          requests: {cpu: "500m", memory: "1Gi"}          limits:   {cpu: "1000m", memory: "2Gi"}        startupProbe:          httpGet: {path: /ready, port: 8000}          failureThreshold: 24          periodSeconds: 5        readinessProbe:          httpGet: {path: /ready, port: 8000}          periodSeconds: 10        livenessProbe:          httpGet: {path: /health, port: 8000}          periodSeconds: 20        lifecycle: {preStop: {exec: {command: ["sleep", "10"]}}}      terminationGracePeriodSeconds: 45---apiVersion: v1kind: Servicemetadata: {name: churn-api, namespace: ml-production}spec:  selector: {app: churn-api}  ports: [{port: 80, targetPort: 8000}]---apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata: {name: churn-api, namespace: ml-production}spec:  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: churn-api}  minReplicas: 3  maxReplicas: 15  metrics:  - type: Resource    resource: {name: cpu, target: {type: Utilization, averageUtilization: 65}}  behavior:    scaleUp:   {stabilizationWindowSeconds: 0,                policies: [{type: Percent, value: 100, periodSeconds: 30}]}    scaleDown: {stabilizationWindowSeconds: 300,                policies: [{type: Percent, value: 25, periodSeconds: 60}]}

The startupProbe grants 24 × 5 = 120 seconds to pull the model from the registry before liveness checking begins. Omit it and a slow registry means the liveness probe kills the pod within about a minute, every time, forever — a crash loop with no error in the logs beyond "connection refused".

maxUnavailable: 0 keeps all three replicas serving throughout a rollout. preStop: sleep 10 gives endpoint removal time to propagate before the process exits, so a deploy does not produce a scatter of connection-refused errors.

Verify with the checks that would actually catch a broken deploy, not just kubectl get pods:

Bash
kubectl -n ml-production rollout status deployment/churn-api --timeout=300s# The version the pods actually loaded should equal the registry's champion.kubectl -n ml-production port-forward svc/churn-api 8080:80 &curl -s localhost:8080/ready | python -m json.tool# A real prediction, and its arithmetic sanity: a two-year contract customer# with long tenure should score well below a new month-to-month one.curl -s -X POST localhost:8080/predict -H 'Content-Type: application/json' \  -d '[{"tenure_months":60,"monthly_charges":45.0,"contract":"two-year",        "support_calls":0,"has_fibre":false},       {"tenure_months":2,"monthly_charges":99.0,"contract":"month-to-month",        "support_calls":4,"has_fibre":true}]' | python -m json.tool

A rollback you have never performed is a rollback you do not have. Rehearse it in staging before you need it at three in the morning.

Part 4 — Evidence that it still works

The metrics are already exported by the service. What remains is deciding what wakes someone up.

Text
groups:- name: churn-api  rules:  - alert: ChurnApiErrorRate    expr: |      sum(rate(churn_requests_total{status="error"}[5m]))      / sum(rate(churn_requests_total[5m])) > 0.02    for: 5m    labels: {severity: page}  - alert: ChurnApiLatencyP99    expr: |      histogram_quantile(0.99,        sum by (le) (rate(churn_latency_seconds_bucket[5m]))) > 0.5    for: 10m    labels: {severity: page}  - alert: ChurnScoreDrift    expr: |      abs(rate(churn_score_sum[1h]) / rate(churn_score_count[1h]) - 0.265) > 0.06    for: 30m    labels: {severity: ticket}  - alert: ChurnFlagRateAnomaly    expr: |      sum(rate(churn_score_bucket{le="0.15"}[1h]))      / sum(rate(churn_score_count[1h])) < 0.20    for: 1h    labels: {severity: ticket}

ChurnScoreDrift is the alert that distinguishes this from an ordinary web service. The training base rate is 0.265, so a well-behaved model scoring representative traffic should average close to that. If the mean predicted probability moves to 0.35, something upstream has changed — a feature's units, a new contract type encoded as unknown, a data pipeline that started sending nulls. Nothing errors. Latency is fine. Only the score distribution shows it.

Note the severities: operational failures page, distributional shifts open a ticket. Paging on drift at 03:00 for something that cannot be fixed until morning is how teams learn to ignore pages.

Documentation that is actually used

Write four short documents, not one long one, because they have four different readers.

DocumentReaderMust answer
README.mdA new engineerHow do I run this locally in under ten minutes?
MODEL_CARD.mdMarketing, complianceWhat does it predict, on what data, how accurate per segment, what are its limits?
API.mdThe team calling the endpointRequest shape, response shape, every error code, rate limits
RUNBOOK.mdWhoever is on call at 03:00For each alert: what it means, first three things to check, exact rollback command

The runbook is the one most often skipped and the only one read under stress. Its rollback entry should be a command that can be copied, not a description of a procedure:

Bash
# Roll back the model (no rebuild, no redeploy - pods reload within 60s)python -c "from mlflow import MlflowClientc = MlflowClient()prev = c.get_model_version_by_alias('churn-predictor', 'previous-champion')bad  = c.get_model_version_by_alias('churn-predictor', 'champion')c.set_registered_model_alias('churn-predictor', 'champion', prev.version)c.set_model_version_tag('churn-predictor', bad.version, 'blocked', 'true')print(f'champion: {bad.version} -> {prev.version}')"# Roll back the SERVICE (code or config problem, not a model problem)kubectl -n ml-production rollout undo deployment/churn-api

Two different rollbacks for two different failures. Confusing them wastes the first ten minutes of every incident.

Judging your own work

Grade yourself against what the system can do, not what you wrote. Each row below is either demonstrably true or it is not.

ClaimHow to prove it
Every deployed model traces to a runFrom the live model_version, recover git SHA, data hash and metrics without opening a notebook
The threshold is justifiedPoint to the expected-value table and the parameter recording it
Rollback worksActually do it in a staging namespace and time it — under two minutes
Serving matches trainingA test asserting the API's score equals the training pipeline's for the same ten rows, to 1e-6
A bad input fails cleanlyPOST a missing field, a null, a 5,000-row batch, an unknown contract type — 400 or 413, never 500
Degradation is visibleSend traffic with all contracts set to month-to-month; the drift alert should fire
A colleague can run itSomeone else follows the README on a clean machine without asking you anything

The mistakes that show up in almost every attempt

MistakeWhy it happensWhat it costs
Logging the estimator, not the whole pipelineThe classifier feels like "the model"Serving reimplements preprocessing; the two drift apart silently
Hardcoding a model version in the manifestIt is the obvious thing to typeRollback needs a code change, review and deploy instead of one metadata write
Using threshold 0.5It is the defaultUSD 16,620 per cycle in this project's economics
One probe endpoint for liveness and readinessThey look the samePods restarted mid-reload; crash loops during model refresh
No startupProbeNothing fails until the model gets bigPermanent crash loop with no useful error
Unbounded batch endpointNobody sends 5,000 rows in testingOne caller OOMs the pod for everyone
Comparing runs across data versionsThe leaderboard sorts by AUC and looks authoritativePromoting whichever run got the easiest split
No previous-champion pointerYou only need it onceRollback becomes an archaeology exercise during an outage

What to build first if you only have a day

If you cannot complete all four parts, the order that produces the most working system per hour is not the order they are listed in. Build the service and its rollback path first, against a model you register in ten minutes with default hyperparameters. A mediocre model you can deploy, observe and revert is a system. An excellent model in a notebook is a screenshot.

Then add the score histogram and the drift alert, because they are what tell you whether anything you do next matters. Then improve the model — and now every improvement is measurable, promotable and reversible, which means you can move quickly precisely because you can undo things.

The instinct to perfect the model first is strong and it is backwards. In the brief you were given, the difference between AUC 0.77 and 0.79 is worth a few hundred dollars a cycle. The difference between a two-minute rollback and a four-hour one, on the day something goes wrong, is worth an order of magnitude more.