Cost per task: connecting my records to agent benchmarks

Back to the playbook · Give the measurement guide to your AI

Working reference, October 14, 2026. All recorded dollar amounts below are USD API-equivalent valuations of visible model usage. They are not subscription invoices or the value of delivered software.

What other people measure

Anthropic's evaluation guide distinguishes a task with defined inputs and success criteria from each trial attempting it. It recommends tracking cost, latency and errors alongside outcomes. That distinction fits my workflow: a worker run is an attempt, while preparation, review and repair can serve the same task.

SWE-bench reports the proportion of issue instances resolved. Its default Verified comparison uses a common mini-SWE-agent environment. A repository issue evaluated against benchmark tests differs from my integration PR, so a shared dollar unit does not make the work equivalent.

Latitude's coding-agent benchmark uses small JavaScript bug-fixing projects and checks the original tests. Its report implementation divides the cost of included runs by solved trials. Unsuccessful included runs contribute cost. It filters non-OK records before aggregation and excludes content-filtered runs from its solve-rate denominator, while retaining their cost. Those exclusions matter: do not assume every published solve rate and average cost share the same denominator.

The research paper AI Agents That Matter argues for evaluating cost together with accuracy. My application is to compare price and acceptance together rather than optimize a cheap worker in isolation.

Four different unit costs

For one trial per task and identical inclusion rules, cost per solved trial = mean cost per trial / solve rate. Use the solve rate as a fraction: a synthetic $2 mean attempt cost and 80% solve rate gives $2.50 per solved trial. This is an accounting identity, not a forecast that retries will work independently.

With multiple trials, report trial successes and distinct solved task IDs separately. A task solved three times is three successful trials but one distinct task. For a retry policy, sum the actual attempts used by that policy and count each accepted task once. Report the retry limit and stopping rule. If no tasks are accepted, the unit cost is undefined, not zero. If acceptance is unmeasured, publish it as unknown.

An average over successful runs alone excludes failed spend. Label it that way if reported; it is not the portfolio cost above.

What my October records support today

The recorded cohort spans October 10 at 23:47 UTC through October 13 at 03:31 UTC, approximately 51.7 elapsed hours. Its 444 accounting rows include preparation, stopped workers, repairs, reviews, coordination, subagents and direct runs. They are not 444 independent tasks.

The averages use unrounded source values. Stage-run costs do not include shared coordination; the per-PR ratio includes every recorded component. The coordinator alone accounts for 49.6% of the total. Counting only worker usage would omit most of this workflow's visible spend.

Human labor, paid CI and infrastructure costs were not established. Ancillary non-persisted model usage may be absent. Integration does not prove production release. This cohort also differs from the October 1–14 account-wide PR snapshot, so I do not multiply $3.20 by that snapshot's 312 merges.

My present statement: “In one recorded October workflow, visible model usage was equivalent to about $339 at the dated API rates, or $3.20 per integration PR, including preparation, reviews, repairs and coordination.”

That establishes a workflow ratio. It does not establish a cheaper SWE-bench solution, a cost per independent feature, or a productivity multiplier.

October task reconciliation: what I found

The aggregate audit joins all 247 project task records, retrieved October 14, to a fresh 400-PR metadata snapshot and this case's 106 integration PRs. Every case PR's merge timestamp was checked against the recorded cost window.

This yields $338.9982168 / 64 = $5.2968471, or $5.30 of recorded-window API-equivalent usage per board-completed leaf-level task. The whole numerator remains intact, including unassigned work and shared overhead. Of 116 worker attempts, 97 finished and 19 returned to preparation. Those 19 stopped/unsuccessful attempts cost $9.3501308 and are included, along with reviews and repairs. Finished attempts are not accepted tasks.

Two limits prevent calling this full cost per independently accepted task. First, the board snapshot follows the cost window and some tasks began earlier, so the numerator omits their earlier costs. Second, most matched done records were closed during a later board cleanup using merge/deployment references. 28 of the 64 lack recorded acceptance criteria. Only one has a dedicated verification field, describing an earlier investigation; two lack a dedicated resolution field. Notes and PRs contain additional test evidence, but neither a done status nor a populated field proves every acceptance criterion. This audit did not change the board.

Removing matched parent entries prevents counting those parents alongside their children; it does not prove sibling scopes are independent. Incidental mentions in PR bodies or general task refs are not implementation evidence. Unresolved mappings remain visible instead of becoming invented tasks. The independently accepted-task denominator remains unknown, not zero.

The original $327.50 subagent incident

I recovered the original author-provided Task Session Cost Study dated October 5. Its figures differ from my recollection of less than eight hours and 11-plus tasks:

The study reports approximately $232 before bedtime and $94 afterwards, partly in overage. It estimates subagent output because streaming usage counters were incomplete, groups tasks by agent description, and applies a $0.20/M cache-read assumption to both models. Its total reportedly matched the app closely. The original analysis scripts lived in a container scratchpad; their durable originals and complete cloud transcripts were not established here. This is a recovered historical study, not a newly reproduced invoice or exact accepted-task count.

I recall that the incident consumed real one-time credit. The recovered evidence does not identify the exact debit or prove that all $327.50 came from that credit. The original study specifically describes only part of the after-bed usage as overage. I preserve that distinction rather than call the whole amount subscription-included or cash-billed.

Current Anthropic usage-credit guidance says overflow uses standard API rates separately from the subscription. That explains the billing-mode distinction, but it cannot establish this incident's historical charge. Included subscription capacity is not a measured discount percentage applied to each task. The $16.38 historical estimate and $5.30 recorded-window ratio have different units and coverage, so they do not establish a matched savings result.

How I would make the comparison stronger

1. Define the work before execution. Assign one stable task ID to a bounded outcome, with acceptance checks and a work class such as bug fix, feature slice, refactor or investigation. Link child tasks and PRs to their parent outcome. Count at one declared level; do not count the parent and its children as independent successes.
2. Join the existing ledger. Use cohort_id, deliverable_id, task_id, run_id, stage, attempt, repair_of and verification_ref. Keep rejected, cancelled and abandoned work in the cohort's cost. Record acceptance once per task, not once per ledger row.
3. Price the entire process. Include planning, execution, review, verification, retries and coordination. Use model identity, fresh input, output, cache reads, cache-write TTL and dated rates. Deduplicate repeated message usage by the maximum available counter in each token category. Record unknown costs explicitly.
4. Publish both views. Show model API-equivalent cost and actual cash allocation separately. For subscription cash, disclose the paid period, number of seats, attribution basis and other work served by those seats. Include external API charges and infrastructure where measured. Show human minutes separately, or include labor using a disclosed rate.
5. Compare like work. Fix repository starting states, specifications, tests, retry policy and pricing basis. Stratify work classes. Report acceptance rate, first-attempt success, repair/revert burden and human minutes alongside cost. A benchmark comparison requires running its same tasks and grading protocol; my internal tasks support a separate operational comparison.

The run ledger already has these identity and accounting fields. It is a recording template, not an automatic task-outcome reconciler.

Why the workflow itself belongs in the comparison

An October 2026 JetBrains Research preprint studies how real coding sessions mix planning, questions, implementation and review. In its 700-task SWE-Bench Pro pilot, sequential interaction approximately doubled cost without a stable resolution-rate change. This is an early result for its tested protocol, not evidence that all planning or parallel agents are wasteful. It is a reason to measure whether my extra stages reduce repair, improve acceptance or save human time enough to justify their cost.

The useful chart for my system is cost per accepted task versus acceptance rate, with work-class filters and human minutes visible alongside it. It needs measured acceptance data before I can plot our point against another workflow. For now, I publish per-run and per-PR ratios, plus the explicitly scoped recorded-window ratio per board-completed leaf task.

Connecting this to a $200 subscription

My $10,000 API-equivalent monthly target would mean $50 of equivalent usage per subscription dollar. It is a capacity/value-of-usage target, not delivered value or an accepted-task cost. The corresponding delivery measure is allocated subscription cash + external API/tool/infra cash, divided by distinct accepted tasks in the same period. Actual human effort remains a separate measure unless included explicitly. A simple full-seat example with $200 cash and 100 accepted tasks would be $2 subscription cash per task before other costs. That is synthetic, not my measured result.

The objective is more accepted work per dollar and less human intervention at a maintained quality bar. Consuming more tokens can raise the API-equivalent value while leaving that objective unchanged.