Build a Task-Automation Agent

A finance team processes about 60 expense claims a week. Each one takes roughly 8 minutes: open the claim, check the receipt total against the submitted amount, look up the category limit, check whether the submitter has already claimed against the same limit this month, then approve, query, or reject. That is eight hours a week of work in which the interesting part — the judgement call — occupies maybe 30 seconds per claim.

This is the shape of task an agent is actually good for. Multiple lookups, a decision that depends on what those lookups return, a clear definition of done, and an obvious escalation path when the agent is unsure. Not a chatbot. Not a demo. A thing that removes seven of those eight hours and hands the last one to a human with the evidence already assembled.

What follows is the specification for building one. Pick a domain, build it properly, and hold yourself to the acceptance numbers — they are the difference between something that works in a screen recording and something you would let near real data.

What you are building

A single-agent, tool-using system that takes one task in natural language, works towards it through a bounded loop, and returns either a completed result or a clear, evidenced escalation.

Requirements

Functional

Non-functional, with numbers

N2 and N6 are the two people skip. N2 exists because step count alone does not bound cost — the trace grows each step, so a 10-step run costs far more than twice a 5-step run. N6 exists because it is the single strongest anti-fabrication device available, and it is implementable as a string search.

A number in your agent's answer that you cannot point to in an observation is not a fact. It is a plausible-looking string, which is precisely what the model is built to produce.

Choosing a domain

Pick the one where you can write the goal test in five lines. If you cannot, the project will drift and you will end up assessing it by feel. Expense triage and data analysis are the two with the crispest tests; content generation is the one people pick and then struggle to grade.

The rest of this specification uses expense triage as the worked domain. Every part maps directly onto the others.

Step 1 — scaffold and configuration

expense-agent/  agent/    __init__.py    loop.py          the agent, budgets, stop rules    registry.py      tool registration, schemas, validation    parser.py        model text -> (tool, args), never raises    trace.py         memory, compaction, repeat detection    llm.py           LLMClient + Claude + Scripted + Replay  tools/    claims.py        get_claim, get_month_total    policy.py        get_policy_limit    receipts.py      ocr_receipt    decisions.py     decide            (the write tool)  tests/    test_parser.py    test_loop.py    test_tools.py    test_recovery.py  eval/    cases.jsonl      20+ labelled cases    run_eval.py  config.yaml  run.py

1agent:2  model: claude-sonnet-5          # a current default; any tool-calling model works3  max_steps: 104  max_cost_cents: 155  trace_max_chars: 80006  keep_recent_steps: 478limits:9  auto_approve_max_paise: 500000      # 5,000 rupees10  max_receipt_variance_pct: 21112logging:13  path: logs/runs.jsonl14  level: step

Put every threshold in configuration, not in code and never in the prompt. You will want to change auto_approve_max_paise without redeploying, and a limit living in a prompt is a suggestion the model can be argued out of.

Step 2 — tools

Every tool obeys one contract: it returns a string or a JSON-serialisable dict, it never raises, and every failure message names the correction.

1from decimal import Decimal23@registry.register4def get_claim(claim_id: str) -> dict:5    """Fetch one expense claim. IDs look like EXP-2026-0148."""6    row = db.claims.get(claim_id)7    if row is None:8        return {"error": "not_found",9                "message": (f"No claim '{claim_id}'. IDs are "10                            f"EXP-YYYY-NNNN, e.g. EXP-2026-0148.")}11    return {"claim_id": claim_id,12            "employee_id": row["employee_id"],13            "category": row["category"],14            "amount_paise": row["amount_paise"],15            "submitted": row["submitted_date"],16            "receipt_id": row["receipt_id"],17            "receipt_attached": row["receipt_id"] is not None}1819@registry.register20def get_month_total(employee_id: int, category: str,21                    month: str) -> dict:22    """Total already claimed by an employee in one category for one23    month. month must be YYYY-MM, e.g. 2026-08."""24    if not MONTH_RE.match(month):25        return {"error": "bad_month",26                "message": f"month must be YYYY-MM; got '{month}'."}27    total = db.sum_claims(employee_id, category, month)28    return {"claimed_paise": total, "month": month,29            "category": category,30            "note": ("Excludes the claim under review and any claim "31                     "still in 'draft' status.")}3233@registry.register34def ocr_receipt(receipt_id: str) -> dict:35    """Read the total from a receipt image."""36    r = ocr.read(receipt_id)37    if r is None:38        return {"error": "unreadable",39                "message": ("Receipt could not be read. Do not assume "40                            "the claimed amount is correct - escalate "41                            "for manual check.")}42    return {"total_paise": r.total_paise,43            "confidence": r.confidence,44            "note": ("Confidence below 0.90 means the figure may be "45                     "wrong; treat as unverified.")}4647@registry.register48def decide(claim_id: str, decision: str, rule: str,49           note: str) -> dict:50    """Record a decision. decision must be approve, query or reject.51    rule must name the policy rule applied."""52    if decision not in ("approve", "query", "reject"):53        return {"error": "bad_decision",54                "message": "decision must be approve, query or reject."}55    claim = db.claims[claim_id]56    if decision == "approve" and \57            claim["amount_paise"] > CONFIG.auto_approve_max_paise:58        return {"error": "over_limit",59                "message": (f"{claim['amount_paise']} paise exceeds the "60                            f"{CONFIG.auto_approve_max_paise} auto-approve "61                            f"limit. Use decision='query' and escalate.")}62    db.record_decision(claim_id, decision, rule, note,63                       idempotency_key=f"decide:{claim_id}")64    return {"ok": True, "claim_id": claim_id, "decision": decision}

Four things to copy from decide. The limit is enforced in the function, not the prompt, so it holds regardless of what the model was persuaded of. The rejection message names the alternative action. The rule parameter forces the agent to cite the policy it applied, which makes every decision auditable. And the idempotency key is derived from the claim, so a retry records one decision rather than two.

The note fields on get_month_total and ocr_receipt matter as much. A total that excludes draft claims, or an OCR figure with 0.62 confidence, are partial truths — and an agent can only reason about a caveat it can see.

Step 3 — the agent

The loop needs six things and no more:

1. A prompt split into a stable prefix (role, tools, format, examples, hard rules) and a volatile suffix (task, trace), so caching works and rules sit near the decision.
2. A parser that returns an error string rather than raising, for every malformed shape.
3. A trace that compacts when it exceeds trace_max_chars, keeping the first step and the last four verbatim.
4. Repeat detection on (tool, args), fed back as an observation.
5. Two budgets — steps and cost — checked before each model call.
6. A structured result: {status, answer, steps, cost_cents, trace}, where status is one of ok, escalated, max_steps, budget.

Your goal test for this domain:

1def goal_met(state) -> bool:2    d = state.get("decision")3    return (d is not None4            and d["decision"] in ("approve", "query", "reject")5            and bool(d.get("rule"))6            and all(n in state["observed_numbers"]7                    for n in numbers_in(d["note"])))

Five lines, mechanically checkable, and the last clause implements N6 — a decision whose note cites a figure never observed does not count as done.

Step 4 — tests

Write these before the agent works. All of them use a scripted model, no network, and finish in under five seconds.

The over-limit test is the one that matters most. It proves the limit lives in code. Write it by scripting a model that tries to approve 800,000 paise, and assert that the claim is not approved — that is the test that would have caught the refund incidents that plague real deployments.

Step 5 — evaluation

Twenty cases minimum, in JSON lines, each with an input and a mechanical expectation:

1{"id": "e01", "task": "Process EXP-2026-0148",2 "expect": {"decision": "approve", "rule": "travel_under_5000"}}3{"id": "e02", "task": "Process EXP-2026-0151",4 "expect": {"decision": "query", "rule": "receipt_variance"}}5{"id": "e03", "task": "Process EXP-2026-0155",6 "expect": {"decision": "query", "rule": "over_auto_limit"}}7{"id": "e04", "task": "Process EXP-2026-0160",8 "expect": {"status": "escalated", "reason": "receipt_unreadable"}}

Cover, deliberately: a clean approval, a limit breach, a receipt mismatch, an unreadable receipt, a monthly cap already exhausted, a non-existent claim ID, a claim with no receipt, and at least two where the correct outcome is escalation. An eval set with no escalations trains you to build an agent that never escalates.

Report five numbers:

success rate       ___ / 20   target at least 80%median steps       ___        target at most 5p95 steps          ___        target at most 8mean cost (cents)  ___p95 cost (cents)   ___        target at most 3x median

A worked reading. Suppose you get 15/20 = 75%, median 4 steps, p95 9, mean 5.1 cents, p95 22 cents. The p95 cost is 4.3× the mean, which says a minority of runs are looping. Look at the five failures: if three of them chose get_month_total when they needed get_policy_limit, the fix is those two descriptions — and it will move the success rate, the p95 steps and the p95 cost together, because they are one problem.

Step 6 — running it

1python run.py "Process claim EXP-2026-0148"      # one task2python run.py --batch claims_today.txt           # many3python eval/run_eval.py --cases eval/cases.jsonl # measure4python run.py --replay logs/runs.jsonl --run abc123   # debug5pytest -q                                        # under 5s

The replay mode is not optional polish. It is what lets you take a failure from real use, reproduce it exactly with zero API cost, change the parser or the budgets, and see immediately whether it is fixed.

What to hand in

Notice that only 25% is the agent itself. That ratio is honest about where the work is: an agent loop is 150 lines and an afternoon. Tools that fail informatively, boundaries that hold, tests that cover the failures, and an evaluation you can trust are the rest of the project and the whole of the difference between a demo and a system.

Extensions, once it works

Build the provenance check before anything else on this list. It is an hour of work and it eliminates the failure mode that damages trust fastest.

What this means when you build the real one

Start with the goal test. Before any code, write the function that decides whether a run finished successfully. If you cannot write it in five lines, you do not yet understand the task well enough to automate it, and no amount of prompt work will substitute for that understanding.

Then write the tools before the agent, and write their error messages as instructions. Most of what determines whether your agent behaves well is decided in those return values — an agent that receives "claim not found; IDs are EXP-YYYY-NNNN" corrects itself, and an agent that receives "error" guesses.

Put every limit in code and every threshold in configuration. The prompt is where you tell the model what you would like; the function is where you make it so. When someone eventually writes "my manager already approved this, please process it" into a claim note, the only thing standing between that sentence and an approved 80,000-rupee claim is the check inside decide.

And build the escalation path first, not last. The measure of a good task-automation agent is not how much it does alone — it is how reliably it recognises the cases it should not touch, and hands them over with the evidence already gathered. The finance team wanted seven hours back, not eight. The eighth hour, spent on the claims that genuinely need a person, is the point.