Continuous Training and Retraining Pipelines

A food delivery company ships an ETA model in January. On the held-out January data its mean absolute error is 3.1 minutes, which is good enough that the support team stops getting "where is my food" calls. Nobody touches it again.

In June, an analyst pulls a report for an unrelated reason and notices the ETA model's MAE is now 7.4 minutes. It did not fail on any particular day. It slid. Three things happened between January and June, none of them announced: the company expanded into two new cities with different traffic patterns, a partner courier fleet came online with slower average speeds, and the restaurant mix shifted from 60% quick-service to 45% as sit-down restaurants joined the platform.

The model was never wrong. The world it was fitted to stopped existing.

The naive fix is "retrain every month". That is better than nothing and worse than it sounds, because it answers only one of the four questions a retraining system has to answer: when. It says nothing about whether the new data is fit to train on, whether the new model is actually better, or what happens if it is worse. Continuous training is the discipline of automating all four.

What continuous training actually automates

People hear "continuous training" and picture a model that learns from every request in real time. That is online learning, it is rare, and it is dangerous for most business problems. Continuous training means something more mundane and far more useful: the sequence from raw data to a validated, deployed model runs without a human doing any of the steps by hand — while humans still control the gates.

The pipeline has six stages, and the interesting part is that four of them are gates that can stop the process.

A retraining pipeline without gates is not automation, it is an unattended way to deploy a worse model. The gates are the product; the training is the easy part.

The gate people skip: validating the data before training

Model training is remarkably tolerant of garbage. Feed it a column where a unit changed from metres to kilometres and it will happily fit, report a plausible-looking metric, and produce a model that is quietly wrong. Nothing throws.

So the validation gate has to be explicit and its expectations have to be versioned alongside the code.

1from dataclasses import dataclass2import pandas as pd34@dataclass5class ValidationResult:6    passed: bool7    failures: list89EXPECTED = {10    "distance_km":     {"dtype": "float64", "min": 0.0,  "max": 60.0,  "max_null": 0.01},11    "prep_time_min":   {"dtype": "float64", "min": 1.0,  "max": 180.0, "max_null": 0.02},12    "courier_type":    {"categories": {"bike", "scooter", "car", "walk"}},13    "actual_eta_min":  {"dtype": "float64", "min": 1.0,  "max": 240.0, "max_null": 0.0},14}1516def validate(df: pd.DataFrame, baseline_rows: int) -> ValidationResult:17    fails = []1819    # Volume: a silent upstream breakage usually shows up here first.20    ratio = len(df) / max(baseline_rows, 1)21    if not 0.7 <= ratio <= 1.3:22        fails.append(f"row count {len(df)} is {ratio:.2f}x baseline {baseline_rows}")2324    for col, rule in EXPECTED.items():25        if col not in df.columns:26            fails.append(f"missing column {col}")27            continue28        s = df[col]2930        null_rate = s.isna().mean()31        if null_rate > rule.get("max_null", 1.0):32            fails.append(f"{col}: null rate {null_rate:.3f} > {rule['max_null']}")3334        if "min" in rule:35            below = (s < rule["min"]).sum()36            above = (s > rule["max"]).sum()37            if below or above:38                fails.append(f"{col}: {below} below min, {above} above max")3940        if "categories" in rule:41            unseen = set(s.dropna().unique()) - rule["categories"]42            if unseen:43                fails.append(f"{col}: unseen categories {sorted(unseen)}")4445    return ValidationResult(passed=not fails, failures=fails)

The unseen categories check is the one that pays for the whole function. When the delivery company added a "drone" courier type, a one-hot encoder trained without it would either throw at serving time or — worse, with handle_unknown="ignore" — silently encode drone deliveries as all-zeros, which the model reads as "some average courier". The gate turns a subtle wrongness into a loud, actionable failure.

The label maturity trap

Here is a failure that survives every schema check and quietly poisons every automated retrain.

Suppose churn is defined as "no order within 30 days". You retrain nightly on a 180-day window containing 100,000 rows, evenly distributed at about 555 rows per day. The most recent 30 days contribute 30 × 555.6 ≈ 16,667 rows, and every one of those customers is labelled "not churned" — not because they stayed, but because their 30-day window has not elapsed yet.

If the true churn rate is 5%, those 16,667 rows should have contained about 833 positives. Instead the training set shows 5,000 − 833 = 4,167 positives out of 100,000, a base rate of 4.17% rather than 5%. That is a 16.7% relative understatement, and it is systematically concentrated in the newest data — precisely the rows the model weights most heavily if you use recency weighting.

The model learns that recent customers do not churn. It becomes progressively more optimistic every night. The fix is one line of arithmetic in the query: the training window must end at today − label_horizon, never at today.

1-- WRONG: the last 30 days have censored labels2WHERE observation_date >= CURRENT_DATE - INTERVAL '180 days'34-- RIGHT: only rows whose label has had time to mature5WHERE observation_date >= CURRENT_DATE - INTERVAL '210 days'6  AND observation_date <  CURRENT_DATE - INTERVAL '30 days'

What triggers a retrain

Four trigger types exist and they suit different problems.

Most production systems combine a schedule with a drift trigger: retrain weekly regardless, and immediately if drift crosses a threshold.

Measuring drift concretely: the Population Stability Index

Drift needs a number, not a feeling. The most widely used one is PSI, which compares the distribution of a feature at training time (the expected distribution) with its distribution now (the actual). Bin the feature, take the proportion in each bin, and sum:

Work it through for distance_km, binned into five quantile buckets defined at training time. At training each bucket held exactly 20% of rows by construction. Last week's live traffic distributes differently:

Summing the contributions: 0.0693 + 0.0144 + 0 + 0.0112 + 0.0405 = 0.135.

The conventional reading is: below 0.10, no meaningful shift; 0.10 to 0.25, moderate shift worth investigating; above 0.25, significant shift, act. So 0.135 says the delivery distances have genuinely moved longer — consistent with expansion into suburban areas — but it is not yet an emergency. Notice that PSI is symmetric in an important way: mass moving out of the short-distance bins contributes just as much as mass moving into the long ones.

1import numpy as np23def psi(expected: np.ndarray, actual: np.ndarray, bins: int = 10) -> float:4    # Bin edges come from the TRAINING data and must be frozen with the model.5    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))6    edges[0], edges[-1] = -np.inf, np.inf78    e_counts, _ = np.histogram(expected, bins=edges)9    a_counts, _ = np.histogram(actual, bins=edges)1011    # Laplace smoothing: an empty bin would make the log term infinite.12    e = (e_counts + 1) / (e_counts.sum() + bins)13    a = (a_counts + 1) / (a_counts.sum() + bins)1415    return float(np.sum((a - e) * np.log(a / e)))

Two implementation details are where people get this wrong. First, the bin edges must be computed once, at training time, and stored with the model. Recomputing quantile edges on the new data forces both distributions to look uniform and PSI collapses to roughly zero — you get a drift detector that never detects drift. Second, without smoothing, a bin that is empty in production divides by zero; with a tiny sample it produces enormous spurious PSI values. Do not run PSI on fewer than a few thousand rows.

Drift is a leading indicator, not a verdict. Inputs can shift without hurting accuracy, and accuracy can collapse with no input drift at all when the relationship between inputs and outcome changes.

The promotion gate: proving the challenger is better

A freshly trained model is a challenger. The model currently serving is the champion. Promotion needs a rule, and the rule must handle three things people usually forget.

1def should_promote(champ: dict, chall: dict,2                   min_gain: float, slice_tolerance: float = 0.02) -> tuple:3    """Return (decision, reason). Metrics are error-like: lower is better."""45    # 1. Same evaluation set, or the comparison is meaningless.6    if champ["eval_set_hash"] != chall["eval_set_hash"]:7        return False, "champion and challenger scored on different data"89    gain = champ["mae"] - chall["mae"]10    if gain < min_gain:11        return False, (f"gain {gain:.3f} below required {min_gain:.3f} "12                       f"(champion {champ['mae']:.3f}, challenger {chall['mae']:.3f})")1314    # 2. No slice may get materially worse, even if the average improves.15    for name, champ_mae in champ["by_slice"].items():16        chall_mae = chall["by_slice"].get(name)17        if chall_mae is None:18            return False, f"challenger has no metric for slice {name}"19        if chall_mae > champ_mae * (1 + slice_tolerance):20            return False, (f"slice {name} regressed: "21                           f"{champ_mae:.3f} -> {chall_mae:.3f}")2223    # 3. Sanity: a suspiciously huge jump is usually leakage, not genius.24    if gain > champ["mae"] * 0.5:25        return False, f"gain {gain:.3f} implausibly large - check for leakage"2627    return True, f"promote: MAE {champ['mae']:.3f} -> {chall['mae']:.3f}"

Setting min_gain is where judgement enters. It should be at least one standard deviation of the metric's seed-to-seed noise, measured by training the same configuration with several different seeds. If retraining the identical config five times gives MAE values of 3.10, 3.18, 3.05, 3.14 and 3.08, the mean is 15.55 / 5 = 3.11 and the deviations are −0.01, +0.07, −0.06, +0.03, −0.03; their squares sum to 0.0104, so the sample standard deviation is $\sqrt{0.0104/4} = 0.051$. A challenger at 3.07 is inside the noise. Requiring a gain of at least 0.05 stops the pipeline from promoting a coin flip every week.

The slice check catches the failure mode that damages products most. A challenger can improve overall MAE from 3.11 to 2.94 while making the "new city" slice worse from 4.2 to 5.6, because the new-city rows are 4% of the data and the average does not notice. Users in that city notice.

Orchestration: where the pipeline actually runs

You need something that runs the steps in order, retries the flaky ones, and tells you when a step fails. The options differ mainly in how much they cost you when things go wrong.

A pipeline expressed in Airflow's TaskFlow API reads close to ordinary Python. The imports below are the Airflow 3 ones; on Airflow 2 the decorators come from airflow.decorators and the exception from airflow.exceptions.

1from datetime import datetime, timedelta2from airflow.sdk import dag, task                        # Airflow 3 import path3from airflow.sdk.exceptions import AirflowSkipException45@dag(schedule="0 3 * * 1",              # 03:00 every Monday6     start_date=datetime(2024, 1, 1),7     catchup=False,8     default_args={"retries": 2, "retry_delay": timedelta(minutes=10)},9     tags=["eta", "continuous-training"])10def eta_retrain():1112    @task13    def ingest() -> dict:14        # Explicit bounds; label horizon respected.15        return pull_snapshot(end_offset_days=30, window_days=180)1617    @task18    def gate_data(snap: dict) -> dict:19        result = validate(load(snap["path"]), snap["baseline_rows"])20        if not result.passed:21            raise ValueError("data validation failed: " + "; ".join(result.failures))22        return snap2324    @task25    def train(snap: dict) -> dict:26        return train_model(snap, seed=42)2728    @task29    def evaluate(run: dict) -> dict:30        return score_with_slices(run)3132    @task33    def gate_promote(chall: dict) -> str:34        champ = load_champion_metrics()35        ok, reason = should_promote(champ, chall, min_gain=0.05)36        if not ok:37            raise AirflowSkipException(reason)   # skip, do not fail: this is normal38        return promote_to_canary(chall["model_uri"])3940    gate_promote(evaluate(train(gate_data(ingest()))))4142eta_retrain()

The detail worth copying is raising a skip rather than a failure when the challenger loses. A challenger that fails to beat the champion is the expected outcome most weeks. If that colours the DAG red, the team stops reading alerts within a fortnight, and then a real failure goes unnoticed. Alert fatigue is a design bug, not a people problem.

Rolling out safely, and undoing it fast

Passing the offline gate is not proof. Offline evaluation uses historical data; production has traffic patterns, latency budgets and inputs that never appeared in any dataset. Two mechanisms bridge the gap.

Shadow mode sends real traffic to the challenger, records its predictions, and throws them away — the champion still answers. You get real-input evaluation with zero user risk. It costs double inference compute and it cannot measure anything that depends on the model's own actions, so it will not tell you whether a recommender's new ranking changes what users click.

Canary gives the challenger a small slice of real traffic — 5%, then 25%, then 50% — with automatic rollback on any of: error rate above baseline, p99 latency above budget, or prediction distribution far from expectation.

1STAGES = [(0.05, 30), (0.25, 60), (0.50, 60), (1.00, None)]  # (share, minutes)23def canary(model_uri: str) -> str:4    baseline = current_health()5    for share, hold_minutes in STAGES:6        set_traffic_split(challenger=model_uri, share=share)7        if hold_minutes is None:8            return "promoted"9        h = observe(minutes=hold_minutes)10        if (h.error_rate > baseline.error_rate * 1.511                or h.p99_latency_ms > baseline.p99_latency_ms * 1.312                or h.psi_predictions > 0.25):13            set_traffic_split(challenger=model_uri, share=0.0)14            alert(f"canary rolled back at {share:.0%}: {h}")15            return "rolled_back"16    return "promoted"

The hold times are not arbitrary. Thirty minutes at 5% of traffic on a service doing 200 requests per second is 30 × 60 × 200 × 0.05 = 18,000 requests — enough to see an error rate of 0.5% with confidence, but nowhere near enough to detect a 2% accuracy regression. Canaries catch operational failures quickly and statistical ones slowly. Size the hold to the thing you are trying to detect, and be honest that accuracy regressions usually need days.

Failure modes specific to automation

The feedback loop deserves elaboration because it is the one that looks like success. Suppose the ETA model predicts a long wait, so the app shows a discouraging estimate, so fewer users order from that restaurant, so you collect fewer examples from it — and the next model has even less evidence about it and predicts from an ever-thinner slice. The system converges on a self-confirming picture of the world. Holding out a small percentage of traffic where decisions are made randomly (or with a deliberately different policy) is the standard antidote, and it must be designed in from the start; you cannot recover the counterfactual afterwards.

Deciding what to build first

Continuous training is worth building when three conditions hold: labels arrive on a timescale shorter than the useful life of a model, the data genuinely changes, and the cost of a stale model is measurable. If labels take nine months and the domain is stable, a calendar reminder to check the model every quarter is the correct engineering decision, and building an Airflow cluster for it is not.

When it is worth building, the order matters. Build the gates before the automation. A team that automates training first and adds validation later spends the intervening months deploying models trained on broken data faster than they could before. The sequence that works is:

1. Measure the champion continuously, per slice, and store the numbers. You cannot gate on a metric you do not have.
2. Add the data validation gate and run it manually against every snapshot for a few weeks. Tune the thresholds until it stops crying wolf.
3. Write the promotion rule and apply it by hand to the next three retrains. Check that it would have made the decision you actually made.
4. Only then let a scheduler run the whole chain, still requiring a human to approve the final promotion.
5. Remove the human once the rule has agreed with you ten times running.

That last step is optional forever. Plenty of good teams keep a person on the promotion gate permanently, because the automation's value was never in removing the human — it was in making every retrain identical, logged, and reversible. The delivery company's June problem was not that a person had to click a button. It was that nobody was looking.