Course Content

Tracking Experiments, Metrics, and Artifacts

A compliance officer sends one email: "For the loan-approval model that went live on 12 March, please provide the evidence that it was evaluated separately on applicants under 25 and over 65, and the exact training data used."

You open the tracking record for that model. It contains three parameters — n_estimators, max_depth, learning_rate — and one metric, auc: 0.887. That is the entire record of a decision that now affects thousands of loan applications.

You cannot answer the question. You cannot even answer the easier one behind it: was the model evaluated on those slices at all? Somebody probably did look, in a notebook, in February. The notebook has since been edited.

The gap here is not tooling. The team had a tracking server. The gap is that nobody decided what a run must contain before it counts as a run. Tracking is only as good as the completeness contract you enforce, and "log whatever seems interesting" produces exactly the record above.

An average of averages weights a 20-sample batch as heavily as a 100-sample one, and reports a model you do not have.

What a complete run record contains

Work backwards from the questions you will be asked six months later. There are five of them, and each maps to a category of thing to log.

Question you will be asked	Category	What to log	Cost of omitting it
"What code produced this?"	Provenance	Git SHA, dirty-tree flag, branch, entry-point command, Python and library versions	You cannot rebuild the model at all
"What data was it trained on?"	Data identity	Source table, date bounds, row and column counts, content hash, class balance, null rates	Two runs are incomparable and nobody notices
"How was it configured?"	Params	Every hyperparameter, including defaults you did not set; preprocessing choices; the split seed	"Best so far" cannot be reproduced
"How good was it, for whom?"	Metrics	Headline metrics, per-slice metrics, the threshold used, curves over training	The compliance email above
"What can I look at?"	Artefacts	Model file, signature, confusion matrix, calibration plot, feature importances, config snapshot, environment lock	Every investigation requires rerunning training

Log for the person who will be handed this model after you have left the team. They cannot ask you what you meant.

The category people mislabel

Trackers distinguish params (immutable, set once, one value per run), metrics (a numeric series with a step index), tags (mutable strings for search) and artefacts (files). Putting something in the wrong bucket has real consequences.

A metric logged as a param becomes a string. You can no longer sort runs by it, filter metrics.auc > 0.88, or plot it.
A param logged as a metric loses its immutability guarantee, and its history becomes a series of one point that clutters every chart.
Anything mutable logged as a param throws. A model's stage ("candidate" then "production") is a tag, not a param, precisely because it changes.
A 4 GB dataset logged as an artefact of every run stores one copy per run. Log its content hash instead and store the bytes once.

Logging metrics without lying to yourself

Metric logging looks trivial and contains three traps that quietly corrupt comparisons.

Trap 1: averaging averages

Suppose you accumulate per-batch accuracy and log the mean at the end of an epoch. Two batches:

Batch	Samples	Correct	Batch accuracy
A	100	90	0.90
B (last, partial)	20	10	0.50

The mean of the batch accuracies is (0.90 + 0.50) / 2 = 0.70. The actual accuracy is (90 + 10) / (100 + 20) = 100 / 120 = 0.8333. The gap is 13 percentage points, caused entirely by giving a 20-sample batch the same weight as a 100-sample one. Real training loops hit this constantly: with 10,000 rows and a batch size of 256 you get 39 full batches plus a final one of 16 rows, so that last batch carries 1/40 = 2.5% of the reported average while representing 16/10,000 = 0.16% of the data — a fifteen-fold over-weighting of your noisiest measurement.

Accumulate counts, not averages:

Python

1correct = total = 02for xb, yb in loader:3    preds = model(xb).argmax(dim=1)4    correct += (preds == yb).sum().item()5    total   += yb.numel()6epoch_accuracy = correct / total          # correct

The same applies to loss. If your loss function returns a per-batch mean, weight each batch's loss by its size before summing.

Trap 2: reporting one number for a threshold-dependent metric

Take a fraud model scored on 10,000 held-out transactions, of which 400 (4%) are fraudulent. At the default threshold of 0.5:

Threshold 0.5	Predicted fraud	Predicted legitimate
Actually fraud	140 (TP)	260 (FN)
Actually legitimate	60 (FP)	9,540 (TN)

Precision = 140 / (140 + 60) = 140/200 = 0.700. Recall = 140 / 400 = 0.350. F1 = 2 × 0.700 × 0.350 / (0.700 + 0.350) = 0.490 / 1.050 = 0.4667. Accuracy = (140 + 9,540)/10,000 = 0.968.

Now the identical model at threshold 0.25:

Threshold 0.25	Predicted fraud	Predicted legitimate
Actually fraud	280 (TP)	120 (FN)
Actually legitimate	520 (FP)	9,080 (TN)

Precision = 280/800 = 0.350. Recall = 280/400 = 0.700. F1 = 2 × 0.350 × 0.700 / 1.050 = 0.490/1.050 = 0.4667 — identical. Accuracy = (280 + 9,080)/10,000 = 0.936.

Two wildly different operating points. One catches 35% of fraud with 200 alerts; the other catches 70% with 800 alerts. Their F1 scores are the same to four decimal places. And note the accuracy: a model that predicted "legitimate" for every single transaction would score 9,600/10,000 = 0.960, better than the recall-oriented threshold.

The lesson is not "F1 is bad". It is that any threshold-dependent metric is a property of model plus threshold, and logging it without the threshold makes it uninterpretable. Log the threshold as a param, log threshold-free metrics (ROC AUC, average precision) as the primary comparison, and log the full confusion matrix as an artefact so anyone can recompute whatever they need.

If a metric changes when you change a number nobody logged, the metric is not a record of anything.

Trap 3: logging only the final value

Two runs both end at validation loss 0.31. One descended smoothly and flattened. The other hit 0.28 at epoch 14 and climbed back to 0.31 by epoch 40 — it overfitted and you saved the wrong checkpoint. The final numbers are identical; the curves are not. Metrics carry a step argument for exactly this reason, and it costs nothing to use.

Python

1for epoch in range(n_epochs):2    tr_loss = train_one_epoch(model, train_loader)3    va_loss, va_auc = evaluate(model, val_loader)4    mlflow.log_metrics({"train_loss": tr_loss,5                        "val_loss":   va_loss,6                        "val_auc":    va_auc}, step=epoch)

A tracker that enforces the contract

Discipline that depends on remembering does not survive a deadline. Put the contract in code, so an incomplete run cannot be created.

Python

1import contextlib, hashlib, json, platform, subprocess, tempfile2from pathlib import Path3import numpy as np4import pandas as pd5import mlflow6from mlflow.models import infer_signature789class ExperimentTracker:10    """Enforces a minimum run record: provenance, data identity, slices."""1112    REQUIRED_TAGS = ("git_sha", "data_hash", "owner")1314    def __init__(self, experiment: str, tracking_uri: str | None = None):15        if tracking_uri:16            mlflow.set_tracking_uri(tracking_uri)17        mlflow.set_experiment(experiment)1819    # ---------- provenance ----------20    @staticmethod21    def _git(*args: str) -> str:22        return subprocess.check_output(["git", *args], text=True).strip()2324    def _provenance(self) -> dict:25        dirty = bool(self._git("status", "--porcelain"))26        if dirty:27            raise RuntimeError(28                "Working tree is dirty. The git_sha would not describe the code that ran."29            )30        import sklearn31        return {32            "git_sha": self._git("rev-parse", "HEAD"),33            "git_branch": self._git("rev-parse", "--abbrev-ref", "HEAD"),34            "python": platform.python_version(),35            "sklearn": sklearn.__version__,36            "platform": platform.platform(),37        }3839    # ---------- data identity ----------40    @staticmethod41    def describe_data(df: pd.DataFrame, label: str) -> dict:42        h = hashlib.sha256(43            pd.util.hash_pandas_object(df, index=True).values.tobytes()44        ).hexdigest()[:16]45        return {46            "data_hash": h,47            "rows": len(df),48            "cols": df.shape[1],49            "pos_rate": round(float(df[label].mean()), 6),50            "null_rate": round(float(df.isna().mean().mean()), 6),51        }5253    @contextlib.contextmanager54    def run(self, name: str, params: dict, data_desc: dict, owner: str):55        prov = self._provenance()56        with mlflow.start_run(run_name=name) as r:57            mlflow.log_params(_flatten(params))58            mlflow.log_params({f"data.{k}": v for k, v in data_desc.items()59                               if k != "data_hash"})60            mlflow.set_tags({**prov, "owner": owner,61                             "data_hash": data_desc["data_hash"]})62            missing = [t for t in self.REQUIRED_TAGS63                       if not mlflow.get_run(r.info.run_id).data.tags.get(t)]64            if missing:65                raise RuntimeError(f"run missing required tags: {missing}")66            yield self6768    # ---------- metrics ----------69    @staticmethod70    def log_slices(y_true: np.ndarray, y_prob: np.ndarray,71                   slices: dict[str, np.ndarray], threshold: float) -> None:72        """slices maps a name to a boolean mask over the evaluation rows."""73        from sklearn.metrics import roc_auc_score, precision_score, recall_score74        y_pred = (y_prob >= threshold).astype(int)75        for name, mask in slices.items():76            n = int(mask.sum())77            if n < 100 or len(np.unique(y_true[mask])) < 2:78                mlflow.set_tag(f"slice.{name}.status",79                               f"skipped: n={n}, classes={len(np.unique(y_true[mask]))}")80                continue81            mlflow.log_metrics({82                f"slice.{name}.n": n,83                f"slice.{name}.auc": roc_auc_score(y_true[mask], y_prob[mask]),84                f"slice.{name}.precision": precision_score(85                    y_true[mask], y_pred[mask], zero_division=0),86                f"slice.{name}.recall": recall_score(87                    y_true[mask], y_pred[mask], zero_division=0),88                f"slice.{name}.pos_rate": float(y_true[mask].mean()),89            })9091    # ---------- artefacts ----------92    @staticmethod93    def log_confusion(y_true: np.ndarray, y_pred: np.ndarray) -> None:94        from sklearn.metrics import confusion_matrix95        cm = confusion_matrix(y_true, y_pred)96        with tempfile.TemporaryDirectory() as d:97            p = Path(d) / "confusion_matrix.json"98            p.write_text(json.dumps({"matrix": cm.tolist(),99                                     "labels": ["negative", "positive"]}, indent=2))100            mlflow.log_artifact(str(p), artifact_path="evaluation")101102103def _flatten(d: dict, prefix: str = "") -> dict:104    out = {}105    for k, v in d.items():106        key = f"{prefix}{k}"107        out.update(_flatten(v, f"{key}.") if isinstance(v, dict) else {key: v})108    return out

Three design choices in there are worth defending.

The dirty-tree exception. It feels obstructive and it prevents the most common provenance lie: a run tagged with a commit that does not contain the code that ran. Anyone reproducing from that tag gets a different model and spends a day confused.

The slice minimum of 100 rows. AUC computed on 30 rows is noise wearing a metric's clothing; two extra positives can move it by 0.15. Rather than log a meaningless number, the tracker records why the slice was skipped. The compliance officer's real question — "did you evaluate on over-65s?" — is then answerable either way: here is the metric, or here is the reason it could not be computed.

Data description as params, hash as a tag. Row count and positive rate become params so you can sort and filter by them; the hash becomes a tag because it is an identity, not a quantity.

Comparing runs programmatically

The web UI is for browsing. Anything you want to be repeatable — a weekly comparison, a promotion decision — should be code.

Python

1import mlflow2import pandas as pd34def compare(experiment: str, metric: str = "val_auc",5            data_hash: str | None = None, top: int = 10) -> pd.DataFrame:6    flt = f"metrics.{metric} > 0"7    if data_hash:8        # Comparing across data versions is the classic invalid comparison.9        flt += f" and tags.data_hash = '{data_hash}'"1011    runs = mlflow.search_runs(12        experiment_names=[experiment],13        filter_string=flt,14        order_by=[f"metrics.{metric} DESC"],15        max_results=top,16    )17    keep = ["run_id", "tags.mlflow.runName", "tags.git_sha",18            f"metrics.{metric}"]19    keep += [c for c in runs.columns if c.startswith("metrics.slice.")]20    keep += [c for c in runs.columns if c.startswith("params.model.")]21    out = runs[[c for c in keep if c in runs.columns]].copy()22    out["tags.git_sha"] = out["tags.git_sha"].str[:8]23    return out242526def worst_slice_gap(runs: pd.DataFrame, metric: str = "val_auc") -> pd.Series:27    """How far below the headline does the weakest evaluated slice fall?"""28    slice_cols = [c for c in runs.columns29                  if c.startswith("metrics.slice.") and c.endswith(".auc")]30    return runs[f"metrics.{metric}"] - runs[slice_cols].min(axis=1)

The data_hash filter is the guard rail that matters most. Ranking runs by AUC across different data versions produces a leaderboard where the winner is whoever happened to evaluate on the easiest split. It looks like a result and it is an artefact of the query.

worst_slice_gap encodes a habit worth building: never look at a headline metric without looking at the worst slice beside it. A run with AUC 0.891 and a worst-slice AUC of 0.62 is a different proposition from one with AUC 0.884 and a worst slice at 0.85, and the leaderboard ordered by headline puts the wrong one on top.

A fully tracked training run

Python

1import numpy as np2from sklearn.ensemble import GradientBoostingClassifier3from sklearn.metrics import roc_auc_score, average_precision_score4from mlflow.models import infer_signature5import mlflow67tracker = ExperimentTracker("loan-approval", "http://mlflow.internal:5000")89params = {10    "model": {"type": "gbm", "n_estimators": 300,11              "max_depth": 3, "learning_rate": 0.05, "random_state": 42},12    "preprocessing": {"impute": "median", "scale": False},13    "decision": {"threshold": 0.30},14}1516train_df, val_df = load_split(as_of="2024-03-01", seed=42)17data_desc = tracker.describe_data(train_df, label="defaulted")1819with tracker.run("gbm-d3-lr05", params, data_desc, owner="priya"):20    X_tr, y_tr = split_xy(train_df, "defaulted")21    X_va, y_va = split_xy(val_df, "defaulted")2223    model = GradientBoostingClassifier(**params["model"]).fit(X_tr, y_tr)24    prob = model.predict_proba(X_va)[:, 1]25    thr = params["decision"]["threshold"]26    pred = (prob >= thr).astype(int)2728    # Threshold-free headline metrics first.29    mlflow.log_metrics({30        "val_auc": roc_auc_score(y_va, prob),31        "val_ap":  average_precision_score(y_va, prob),32        "val_pos_rate": float(y_va.mean()),33        "val_alert_rate": float(pred.mean()),34    })3536    tracker.log_slices(37        y_va.to_numpy(), prob,38        slices={39            "age_under_25": (X_va["age"] < 25).to_numpy(),40            "age_over_65":  (X_va["age"] > 65).to_numpy(),41            "first_time_applicant": (X_va["prior_loans"] == 0).to_numpy(),42            "region_north": (X_va["region"] == "north").to_numpy(),43        },44        threshold=thr,45    )46    tracker.log_confusion(y_va.to_numpy(), pred)4748    # Feature importances as a sortable artefact, not a printout.49    imp = sorted(zip(X_tr.columns, model.feature_importances_),50                 key=lambda t: -t[1])51    mlflow.log_text(52        "\n".join(f"{n},{v:.6f}" for n, v in imp),53        artifact_file="evaluation/feature_importance.csv",54    )5556    mlflow.sklearn.log_model(57        sk_model=model,58        name="model",59        signature=infer_signature(X_tr, model.predict_proba(X_tr)),60        input_example=X_tr.iloc[:5],61    )62    mlflow.log_artifact("requirements.txt", artifact_path="environment")

That run answers the compliance email. Slice metrics for under-25 and over-65 exist as first-class metrics. The data hash plus the row count identify the training set. The git SHA identifies the code. The environment lock identifies the library versions. The confusion matrix lets anyone recompute precision at a different threshold without touching the model.

Failure modes that survive having a tracking server

Failure	How it shows up	Fix
Comparing across data versions	A leaderboard whose top run is untouchable and unexplainable	Filter on `data_hash`; make it a required tag
Metric name drift	`auc`, `val_auc`, `AUC`, `roc_auc` all present; no chart can span the experiment	Centralise logging in a wrapper; never call `log_metric` ad hoc
Untracked "quick tests" that get deployed	A production model with no run at all	Deploy only from a registered model, and register only from a run
Enormous artefacts	Storage bill grows linearly with experiments; UI becomes slow	Log data hashes, not data; cap artefact size in the wrapper
Slice metrics on tiny slices	A slice AUC of 1.00 on 14 rows, treated as evidence	Enforce a minimum count; record a skip reason instead
Runs left open	Ghost runs stuck in RUNNING for weeks; nested runs attach to the wrong parent	Always use the `with` context manager, never bare `start_run()`

Nested runs, used correctly

A hyperparameter sweep should be one parent run with a child per trial, not 200 unrelated top-level runs cluttering the experiment.

Python

1with mlflow.start_run(run_name="gbm-sweep-2024-03") as parent:2    mlflow.log_param("search_space", json.dumps(grid))3    best = (None, -1.0)4    for i, cfg in enumerate(grid):5        with mlflow.start_run(run_name=f"trial-{i:03d}", nested=True):6            mlflow.log_params(cfg)7            auc = fit_and_score(cfg)8            mlflow.log_metric("val_auc", auc)9        if auc > best[1]:10            best = (cfg, auc)11    mlflow.log_params({f"best.{k}": v for k, v in best[0].items()})12    mlflow.log_metric("best_val_auc", best[1])

The parent then carries the sweep-level answer, so a colleague scanning the experiment sees one row saying "sweep, best AUC 0.893" rather than 200 rows they have to aggregate mentally.

What to do on Monday

The practical move is to write down your team's minimum run record as a list, put it in a wrapper, and make the wrapper the only sanctioned way to train. A reasonable starting list is: git SHA with a dirty-tree check; data hash, row count and positive rate; every hyperparameter flattened; a threshold-free headline metric; the decision threshold as a param; metrics for the three or four slices your business or regulator cares about; a confusion matrix and feature importances as artefacts; and the dependency lock file.

Then apply one test to decide whether the tracking is real: pick a model currently serving traffic and try to answer, using only its run record, three questions you have not prepared for. Which commit built it. What its recall was on your smallest important customer segment. What it would have predicted if the threshold had been 0.2 instead of 0.5. If the answers require rerunning anything, the record is a diary, not a system of record — and diaries are exactly what fails when a compliance officer emails.