Measuring the Total Cost of Multi-Agent Workflows

Abstract

A less expensive model call does not necessarily produce a less expensive delivered result. Multi-agent workflows add routing, repeated context, tool use, coordination, review, retries and acceptance work; they may also improve quality enough to justify those costs. Yet economic comparisons often stop at tokens, list prices or the runtime of one component. This paper proposes an accounting boundary from request admission to accepted outcome. It separates one-time development from recurring operation, records every attempt rather than only successful runs, and reports cost against a declared quality threshold. Existing work on cascades and model routing demonstrates real cost–quality trade-offs under benchmark-specific conditions, while ensemble research shows why additional calls may improve performance. Those results support optimisation inside the workflow, not an inference that the whole workflow is cheaper. The accompanying worksheet provides a reproducible event ledger, acceptance denominator and comparison protocol. This is a measurement design, not evidence that multi-agent systems generally save money.

1. Research question and scope

How should an organisation measure whether a multi-agent workflow lowers the total cost of producing an accepted result, rather than merely lowering the price of one model call?

The scope is an operational workflow in which one or more models, tools, coordinators, validators or human reviewers contribute to a deliverable. “Cost” includes directly invoiced services and explicitly valued internal resources. “Accepted result” means an output that satisfies a predeclared task contract and quality threshold. The method can compare a multi-agent candidate with a single-model baseline, two orchestration designs, or successive versions of the same system.

The proposal does not reduce every consequence to money. Security, privacy, fairness, safety and unacceptable failure modes remain constraints. Latency, energy, operator attention and opportunity cost should be reported in their native units unless an organisation has a defensible conversion policy. The aim is disciplined comparison, not a universal scalar for value.

2. Why model price is an incomplete denominator

Research on economical LLM use shows that substantial savings are possible within a defined inference task. FrugalGPT models API cost from prompt length, generated length and sometimes a fixed per-query fee, then uses prompt adaptation, approximation and cascades to optimise performance under a budget [1]. RouteLLM learns to choose between stronger and weaker models and evaluates a quality–cost frontier on public benchmarks [2]. These are important results: model selection and escalation policy can change the cost of inference while preserving a specified benchmark quality under study conditions.

The accounting boundary in an operational agent workflow is wider. Before a worker model is called, somebody or something must interpret the request, prepare context, select a route and establish acceptance criteria. After generation, the system may run tools, reconcile outputs, review evidence, repair defects and verify the final state. A cheap output that needs extensive correction can transfer cost rather than remove it. A more expensive route can be economical if it avoids failures or raises the accepted yield.

Additional agents can also be a deliberate quality investment. More Agents Is All You Need reports performance gains from repeated sampling and voting across several benchmark tasks, with results dependent on task, model and ensemble size [3]. Its token-budget analysis makes the trade-off visible: more samples are not free. The operational question is therefore not “are more agents good?” but “at what total cost does this workflow cross the required acceptance boundary?”

A first-party prototype record illustrates the measurement trap without establishing a general result. In a small matched pilot, a deterministic composition reduced several local counters and mean runtime, while the count of primary-model calls was unchanged; the acceptance record expressly withheld an end-to-end efficiency or financial-payback claim [5]. Component improvement was observed. Whole-delivery savings were not.

3. Define the unit before counting it

Economic comparisons fail when their units differ. One arm may report a completed model response while another reports a reviewed, deployable artifact. One may exclude failed attempts; another may include them. One may use a warm cache while the other starts cold. The comparison is then precise in arithmetic and invalid in meaning.

A measurement contract should define five items before trials begin:

1. Task population: the distribution or fixed set of requests being compared, stratified by difficulty and consequence.
2. Outcome unit: the delivered object or state—for example, an accepted code patch, reconciled record or reviewed research answer.
3. Acceptance contract: objective checks, reviewer rubric, prohibited failures and any human sign-off.
4. Accounting boundary: the first admitted event and the final accepted, rejected or abandoned state.
5. Observation horizon: the period in which delayed failures, rework or reversals are attributed.

The denominator should normally be accepted outcomes at the declared threshold, not generated outputs or successful API responses. If 100 admitted tasks produce 130 generations but only 82 accepted outcomes, 130 is an activity count; 82 is the delivery denominator.

4. A lifecycle cost model

For a study window, let total workflow cost be:

C_total = C_build_amortised
        + C_model
        + C_tools
        + C_coordination
        + C_review
        + C_retry_and_repair
        + C_failure
        + C_operations

Where:

• Build amortised covers one-time design, integration, test-fixture and evaluation work allocated across a stated expected volume. Report the unamortised total and allocation rule as well.
• Model covers all model invocations: coordinators, workers, judges, summaries and unsuccessful attempts. Separate input, cached input and output according to the provider’s actual billing semantics.
• Tools covers search, storage, execution, external APIs and other metered services.
• Coordination covers packet preparation, routing, context assembly, merging, state transfer and human orchestration.
• Review covers deterministic validation, model-based evaluation and human inspection. Model judges belong in both the event ledger and model cost.
• Retry and repair covers repeated calls, rollback, defect diagnosis and correction after a failed candidate.
• Failure records abandoned work, incident response and consequential rework within the observation horizon.
• Operations covers recurring monitoring, maintenance and infrastructure reasonably attributable to the workflow.

The primary economic metric is:

cost per accepted outcome = C_total / N_accepted

It must be paired with accepted yield (N_accepted / N_admitted), quality results and failure severity. A system can improve cost per acceptance by silently lowering the bar, which is why the acceptance contract must be fixed or changes must be disclosed.

Queue time and end-to-end latency should be reported separately: median, tail percentile and time to accepted outcome. Converting time to money can be useful, but only with a documented labour or delay-cost rule. Idle machine waiting is not automatically labour cost; human interruption is not automatically captured by elapsed time.

5. The event ledger: count paths, not anecdotes

Aggregate token totals cannot explain where cost moved. Each admitted task should have an event ledger with a stable task identifier and ordered records for:

admission → preparation → route → model/tool events
→ validation → review → repair/retry → acceptance or terminal failure

For every event, record the actor or component class, start and end time, model/tool version, input/output units, cache treatment, direct charge where available, human active minutes, attempt number and parent event. Record zero-cost events too when they affect latency or explain control flow.

Preserve failed and superseded attempts. Survivorship bias appears immediately when only the accepted trajectory is retained. Likewise, do not label token volume “invoice cost” unless the pricing rule, cache discount, batch discount and provider record support that conversion. Historical list prices may be useful for normalised comparison, but should be labelled estimates and held constant across arms.

The ledger also needs terminal states: accepted, rejected, abandoned, timed out, blocked by authority, and invalid trial. Invalid trials should be rare and governed by predeclared exclusion rules. An inconvenient failure is not an invalid trial.

6. Quality-adjusted comparison without a decorative score

A single weighted “cost-quality score” can hide important trade-offs. Prefer a frontier or threshold approach:

1. define minimum quality and non-negotiable safety conditions;
2. exclude configurations that fail those conditions;
3. compare total cost and latency among the qualifying configurations;
4. report the full outcome distribution, not only the mean.

RouteLLM explicitly treats routing as a trade-off between response quality and strong-model use [2]. FrugalGPT formulates performance maximisation under an average cost constraint [1]. The same logic can be extended to workflow acceptance, but the metric must match the operational deliverable rather than assume that a benchmark answer score captures reviewability, side effects or downstream correctness.

For heterogeneous task classes, report strata before aggregation. A router may save cost on routine tasks and fail disproportionately on rare, consequential ones. Consequence-weighted analysis may be added, but the weights should be declared by the deploying organisation. NIST’s AI RMF emphasises context, socio-technical effects and lifecycle measurement rather than a context-free trustworthiness number [4]. Economic evaluation should not erase that context.

7. Experimental protocol

A defensible comparison uses matched, repeated trials.

Freeze the contract. Predeclare task fixtures, acceptance checks, reviewer instructions, cost rules, cache conditions, time horizon and exclusion criteria.

Balance the arms. Use the same task sample or randomise tasks across arms. Record difficulty, model versions, orchestration configuration, concurrency and infrastructure. If one arm receives richer context or more human assistance, count it.

Separate cold and steady state. Startup, cache warming and first-run configuration may be real costs. Report them separately from steady-state operation rather than deleting them or spreading them invisibly.

Use independent acceptance where feasible. The evaluator should not merely repeat the candidate workflow’s self-assessment. Blind human review, executable checks or authoritative state inspection can reduce shared failure modes, though none is infallible.

Retain every attempt. Include route errors, rejected outputs, retries and reviewer repairs. Report accepted yield, cost per accepted outcome, time to acceptance, severity-weighted failures and confidence intervals or raw distributions appropriate to the sample size.

Re-run after change. Provider prices, models, cache semantics and task mix change. A result is a dated property of a configuration and workload, not a permanent property of “multi-agent systems.”

8. Alternatives and related methods

Token and API-cost dashboards are simple and useful for budget control, but they omit labour and downstream work unless integrated with workflow events. Application-performance traces expose latency and parent–child calls, but require an acceptance layer to identify useful outcomes. Activity-based costing can allocate human and infrastructure cost, though excessive precision can cost more than it reveals. Benchmark studies offer reproducibility and comparison, but may not represent the production task distribution or consequence profile.

Simulation can estimate scaling before deployment, but simulated reviewer behaviour and failure rates should not be presented as observations. Randomised online experiments provide stronger operational evidence but may be inappropriate for high-consequence tasks. A practical programme often begins with replay or synthetic fixtures, proceeds to a bounded pilot and only then measures production under explicit safeguards.

9. Practical application

The accompanying Whole-Workflow Cost and Quality Worksheet contains the accounting boundary, event schema, comparison table and synthetic worked example. Teams can begin with a small matched set and a coarse ledger. The minimum useful discipline is to capture all model/tool calls, active human minutes, retries, terminal state and acceptance evidence for each admitted task.

The worksheet also includes a stop rule. If a candidate cannot meet the baseline acceptance threshold, economic comparison should not rehabilitate it through a low token bill. Conversely, if it improves quality materially, report the incremental cost and decision context rather than calling it inefficient by definition.

10. Limitations

This paper proposes an accounting method and does not validate it through an independent comparative study. Cost allocation remains partly conventional: shared infrastructure, salaried attention and delayed opportunity cost can be assigned in more than one defensible way. Human-time logs are intrusive and noisy. Acceptance rubrics can drift, reviewers can disagree, and automated evaluators can share failure modes with the system under test. Rare severe failures need longer observation than most pilots provide.

The method also does not establish causality from every ledger event. Coordination work may prevent failures that never become visible. Amortising development cost requires an expected volume that may not materialise. Provider invoices may aggregate usage in ways that complicate task-level attribution. Environmental impacts are not equivalent to financial charges and require separate measurement.

Finally, economic efficiency is not sufficient for responsible deployment. A cheaper workflow that violates authority, privacy or safety constraints is not an acceptable optimisation.

11. Conclusion

Multi-agent economics should be measured at the boundary where work becomes acceptable, not where generation ends. Model routing, cascades and repeated sampling can improve a local cost–quality trade-off, but each introduces conditions, overheads and evaluation choices. Tokens are evidence about one resource. They are not a receipt for the whole job.

A credible comparison defines equivalent tasks, fixes the acceptance threshold, traces every attempt from admission to terminal state, separates one-time from recurring cost and divides by accepted outcomes. It then reports quality, failures and latency alongside money. This approach may show that delegation is economical, that a more expensive route earns its place, or that apparent savings were moved into coordination and review. All three are useful findings. The spreadsheet should be allowed to disappoint the architecture; otherwise it is merely architecture-themed stationery.

References

[1] L. Chen, M. Zaharia and J. Zou, “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance,” arXiv:2305.05176, 9 May 2023. https://arxiv.org/abs/2305.05176

[2] I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous and I. Stoica, “RouteLLM: Learning to Route LLMs with Preference Data,” arXiv:2406.18665v4; ICLR 2025, revised 23 February 2025. https://arxiv.org/abs/2406.18665

[3] J. Li, Q. Zhang, Y. Yu, Q. Fu and D. Ye, “More Agents Is All You Need,” Transactions on Machine Learning Research, 2024. https://arxiv.org/abs/2402.05120

[4] National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023. https://doi.org/10.6028/NIST.AI.100-1

[5] Nordlith, “Bounded deterministic-composition prototype acceptance record,” first-party engineering record, 18 October 2026. Not independently validated; retained privately to avoid disclosing operational details.