Harness Engineering: Making Coding Agents Dependable

Course Content

Harness Engineering: Making Coding Agents Dependable

5 sections · 23 lessons

Milestone 5: the progress file and resuming

After milestone 4, one Kite session is reliable. Real work needs many sessions: a goal split into features, sessions that fail and must be resumed, and nights where Kite works through a list unattended. This milestone builds kite/progress.py, the memory that carries work from one session to the next, and changes the command line to work from it.

Everything here comes from the section on memory across sessions: a fresh session per feature, a briefing built from a file the harness owns, statuses that only verified outcomes can change, and a git commit as a save point after every session.

One session, from briefing to save pointLoad progress,pick next featureBrief fromverified factsSession withgate plus verifyRecord outcome,queue proposalsCommit, orstash thencommitExit code 0, 1 or 2 lets a shell loop run a whole batch.
Whatever the outcome, each session ends with a clean tree and a commit, so the next one starts from known ground.

Loading, saving and choosing

Python
"""The progress file: the only memory that survives from one session to the next."""import jsonimport subprocessfrom datetime import datefrom pathlib import Pathdef load(path: Path) -> dict:    return json.loads(path.read_text())def save(path: Path, data: dict) -> None:    tmp = path.with_suffix(".tmp")    tmp.write_text(json.dumps(data, indent=2) + "\n")    tmp.replace(path)                      # atomic: a crash never leaves half a filedef next_feature(data: dict) -> dict | None:    return next((f for f in data["features"] if f["status"] in ("todo", "in_progress")), None)

save writes a temporary file and renames it over the real one. On the same file system a rename is atomic: at every moment, the path holds either the complete old file or the complete new one. next_feature takes the first feature that is todo or in_progress, so an unfinished feature is always resumed before a new one starts, and blocked and proposed items are skipped until a human acts.

Briefing in, record out

Python
def briefing(data: dict, feature: dict) -> str:    done = [f"- {f['id']} {f['title']}" for f in data["features"] if f["status"] == "done"]    return "\n".join([        f"Project: {data['goal']}",        "Already done and verified by the harness (do not redo):", *(done or ["- nothing yet"]),        "",        f"Your task this session: {feature['id']} - {feature['title']}",        f"It is accepted when this passes: {feature['verify']}, and so do the standard checks.",        f"Notes from earlier attempts: {feature.get('notes') or 'none - this is the first attempt'}",    ])def record(data: dict, feature: dict, outcome) -> None:    feature["attempts"] = feature.get("attempts", 0) + 1    if outcome.status == "done":        feature["status"] = "done"    else:        feature["status"] = "blocked" if feature["attempts"] >= 3 else "in_progress"    feature["notes"] = outcome.summary[-1500:]    for line in outcome.summary.splitlines():          # scope control: queue it, never do it        if line.startswith("QUEUE:"):            data["features"].append({"id": f"Q{len(data['features']) + 1}", "title": line[6:].strip(),                                     "status": "proposed", "verify": "", "notes": f"found during {feature['id']}"})    data["log"].append(f"{date.today()} {feature['id']} attempt {feature['attempts']}: "                       f"{outcome.status} in {outcome.turns} turns, {outcome.tokens} tokens")

briefing is the handoff from the lesson on why sessions start from zero: goal, verified work, this session's task and its acceptance command, and notes from earlier attempts. It becomes the session's first user message, so the task is in view from turn 1.

record is the gate from the progress-file lesson. Only an outcome with status done — which the loop produces only when the completion gate passed — can mark a feature done. The agent's summary becomes the notes; QUEUE: lines become proposals a human must promote.

Save points

Python
def finish(root: Path, path: Path, data: dict, feature: dict, outcome) -> str:    """Save the session's result and leave a clean tree; return the new commit hash."""    if outcome.status != "done":                       # keep failed work, but out of the way        subprocess.run(["git", "stash", "push", "-u", "-q", "-m", f"kite {feature['id']} failed"], cwd=root)    record(data, feature, outcome)    save(path, data)    subprocess.run(["git", "add", "-A"], cwd=root, check=True)    subprocess.run(["git", "commit", "-q", "-m", f"kite: {feature['id']} {outcome.status}"], cwd=root, check=True)    return subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=root,                          capture_output=True, text=True).stdout.strip()

This is the function from the lesson on git save points. A done feature becomes one commit with its code, tests and progress update. A failed attempt goes into a labelled stash, and only the progress update is committed. Either way the next session starts from a clean tree, which the gate's "what changed" rule depends on.

The command line, driven by the progress file

Replace main() in kite/__main__.py, and add from . import progress to the imports (Gate is already imported since milestone 4). The --task option goes away: the task now comes from the progress file, so update the module docstring to say python -m kite REPO [--unattended]: one session on the next open feature.

Python
def main(argv=None) -> int:    ap = argparse.ArgumentParser(prog="kite")    ap.add_argument("repo", type=Path)    ap.add_argument("--unattended", action="store_true", help="refuse anything that needs a human")    args = ap.parse_args(argv)    cfg, root = Config(), args.repo.resolve()    path = root / cfg.progress_file    data = progress.load(path)    feature = progress.next_feature(data)    if feature is None:        print(f"No open features in {path}.")        return 2    feature["status"] = "in_progress"    policy = Policy(refuse if args.unattended else ask_human)    toolbox = Toolbox(root, cfg, check=policy.check)    gate = Gate(root, cfg.checks + [feature["verify"]], cfg.command_timeout)    model = AnthropicModel(cfg.model, cfg.max_tokens)    outcome = run_session(progress.briefing(data, feature), model, toolbox, cfg,                          system=SYSTEM, gate=gate)    commit = progress.finish(root, path, data, feature, outcome)    print(f"{feature['id']}: {outcome.status} in {outcome.turns} turns, "          f"{outcome.tokens:,} tokens. Commit {commit}.")    return 0 if outcome.status == "done" else 1

The gate now includes the feature's own verify command after the standard checks. The exit code tells a script what happened: 0 for a finished feature, 1 for a failed session, 2 when nothing is left to do. That makes a batch a one-line shell loop:

Bash
while python -m kite ../ledgerly-agent --unattended; do :; done

It keeps going while features finish and stops at the first failure or when the list is empty. Stopping at the first failure is a choice: a human looks at the problem before more sessions build on top of it. If you would rather let later, independent features continue, loop until the exit code is 2 and let blocked statuses collect the failures for the morning.

Features that fit in one session

The progress file only works if each feature fits comfortably in one session. On Ledgerly, a good feature takes 15 to 30 turns, changes one behaviour, and has one verify command that proves it. Compare:

Too bigRight size
"Late fees everywhere""2% late fee on invoices unpaid more than 30 days after the due date"
"Improve reminders""Reminder emails show the late fee and the new total"
"Export and import invoices as CSV""CSV export of invoices for a date range" (import is a separate feature)

A feature that is too big shows up in the log as out_of_turns with a summary that says "partially done". Split it and move on; a third attempt at a feature that is too big rarely succeeds. A feature that is too small — "add a docstring to apply_late_fee" — wastes the fixed cost of a session: the briefing, the reading, and a 45-second gate. Group small, related changes into one feature with one test.

Test it

Create tests/test_progress.py:

Python
import jsonimport subprocessfrom kite import progressfrom kite.loop import OutcomeDATA = {"goal": "Ledgerly late fees", "log": [], "features": [    {"id": "F1", "title": "2% late fee", "status": "done", "verify": "true", "notes": ""},    {"id": "F2", "title": "Fee on reminders", "status": "todo", "verify": "true", "notes": ""}]}def test_briefing_and_failed_attempt(repo):    path = repo / "kite-progress.json"    progress.save(path, json.loads(json.dumps(DATA)))    subprocess.run("git add -A && git commit -qm progress", shell=True, cwd=repo, check=True)    data = progress.load(path)    feature = progress.next_feature(data)    assert feature["id"] == "F2" and "- F1 2% late fee" in progress.briefing(data, feature)    (repo / "ledgerly" / "half_done.py").write_text("x = 1\n")    outcome = Outcome("gate_failed", 12, 90_000, "Tests fail.\nQUEUE: utils.parse_date ignores time zones")    progress.finish(repo, path, data, feature, outcome)    saved = progress.load(path)    assert saved["features"][1]["status"] == "in_progress"    assert saved["features"][2]["status"] == "proposed"    assert not (repo / "ledgerly" / "half_done.py").exists()          # stashed, not lost    assert "kite F2 failed" in subprocess.run(["git", "stash", "list"], cwd=repo,                                              capture_output=True, text=True).stdout

json.loads(json.dumps(DATA)) makes a deep copy, so the test never changes the module-level data. The test walks one full failed session: choose F2, brief it, fail, stash the half-done file, keep F2 in progress, turn the QUEUE: line into a proposal. For a real run, write Ledgerly's kite-progress.json by hand or with an initialisation session, commit it, and start the loop.

Check your understanding

0 of 3 answered

1.Why does next_feature pick in_progress features as well as todo ones?

2.A session ends out_of_tokens after editing three files. What does finish leave behind?

3.Why does Kite's batch loop while python -m kite ... --unattended; do :; done stop at the first failed session?