Whole-Workflow Cost and Quality Worksheet
Practical companion protocol. Examples and proposed controls retain their stated evidence limits.
Contents
Status: reusable proposed protocol; the worked numbers are synthetic and are not Nordlith operating results.
1. Measurement contract
Complete this before running either arm.
| Field | Required entry |
|---|---|
| Decision | What architecture or routing decision will this study inform? |
| Baseline | Exact version, models, tools and orchestration configuration |
| Candidate | Exact version, models, tools and orchestration configuration |
| Task population | Fixed fixture set or sampling frame; difficulty/consequence strata |
| Admission event | First event included in the accounting boundary |
| Outcome unit | Delivered object or authoritative state being accepted |
| Acceptance contract | Checks, rubric, prohibited failures and sign-off |
| Observation horizon | Period for delayed repair, reversal and incident attribution |
| Cache condition | Cold/warm/mixed; provider billing treatment |
| Price basis | Invoice, current list price or fixed normalised price |
| Human-time rule | Roles, active-minute method and loaded hourly rates |
| One-time allocation | Unamortised build cost, expected volume and amortisation period |
| Exclusion rule | Predeclared reasons a trial may be invalidated |
| Stop rule | Safety, quality or budget condition that halts the candidate |
2. Per-task event ledger
Use one row per event. Preserve rejected and superseded attempts.
| task_id | event_id | parent_event | arm | attempt | event_class | component/version | start | end | input units | cached input | output units | direct charge | active human min | terminal state | evidence ref |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| T-001 | E-001 | — | baseline | 1 | admission | fixture-v1 | … | … | 0 | 0 | 0 | 0 | 2 | — | contract/T-001 |
Allowed event_class values: preparation, routing, model, tool, coordination, validation, review, repair, rollback, operations, terminal.
Allowed terminal states: accepted, rejected, abandoned, timed_out, authority_blocked, invalid_trial.
Rules:
- A provider acknowledgement is an event, not an accepted outcome.
- Cached input is recorded separately and monetised only under the provider’s verified rule.
- Model judges are recorded as model events and validation events.
- Human waiting time is not active labour unless the declared policy says otherwise.
- Every admitted task reaches exactly one terminal state.
3. Cost roll-up
For each arm, calculate:
C_total = build_amortised
+ model
+ tools
+ coordination
+ review
+ retry_and_repair
+ failure
+ operations
accepted_yield = N_accepted / N_admitted
cost_per_accepted_outcome = C_total / N_accepted
Do not calculate cost_per_accepted_outcome when N_accepted = 0; report it as undefined.
Keep these measures beside the cost result:
- acceptance score or pass rate under the frozen rubric;
- prohibited-failure count and severity;
- median and tail time to accepted outcome;
- retries per admitted task;
- reviewer minutes per accepted outcome;
- model/tool event counts per accepted outcome;
- invalid-trial count with reasons;
- task-stratum results.
4. Comparison table
| Measure | Baseline | Candidate | Difference | Interpretation |
|---|---|---|---|---|
| Admitted tasks | Same population? | |||
| Accepted outcomes | Same contract? | |||
| Accepted yield | ||||
| Direct model cost | Includes all agents/judges? | |||
| Tool cost | ||||
| Coordination cost | Human + metered automation | |||
| Review cost | ||||
| Retry/repair cost | Includes failed attempts | |||
| Failure cost | Within horizon | |||
| Operations cost | ||||
| Amortised build cost | Show unamortised total separately | |||
| Total cost | ||||
| Cost per accepted outcome | Primary economic comparison | |||
| Median / p95 time to acceptance | Keep time in native units | |||
| Prohibited failures | Gate, not a discountable cost |
5. Synthetic worked example
Assume both arms process 100 matched tasks under the same acceptance contract. Dollar values below are illustrative and already include the declared human-time conversion. The figures were arithmetically checked; they are not observed results.
| Cost category | Baseline | Candidate |
|---|---|---|
| Model | $42 | $18 |
| Tools | $8 | $12 |
| Coordination | $120 | $200 |
| Review | $360 | $540 |
| Retry/repair and failure | $90 | $120 |
| Amortised build | $100 | $150 |
| Total | $720 | $1,040 |
| Accepted outcomes | 90 | 92 |
| Cost per accepted outcome | $8.00 | $11.30 |
Calculation:
baseline = 720 / 90 = $8.00 per accepted outcome
candidate = 1040 / 92 = $11.304347... ≈ $11.30
relative difference = (11.304347 / 8.00) - 1 ≈ 41.3% higher
Interpretation: the candidate reduces direct model cost by $24 and accepts two additional tasks, but its extra coordination and review make its cost per accepted outcome about 41.3% higher in this synthetic case. This does not prove the architecture is inferior. The two extra accepted outcomes may be important, the review process may improve with learning, or severe failures may differ. It does prove that the phrase “cheaper model” cannot settle the workflow comparison.
6. Release decision protocol
- Check comparability. Reject the comparison if task populations or acceptance contracts materially differ without stratification.
- Apply hard gates. Do not advance a candidate with prohibited authority, privacy, safety or integrity failures.
- Check quality. If the candidate misses the declared acceptance threshold, report the economic result but do not call it a qualifying saving.
- Compare qualifying arms. Use cost per accepted outcome, accepted yield, latency and failure distribution.
- Inspect transfers. Identify whether model savings moved into preparation, coordination, review or repair.
- Stress-test assumptions. Recalculate with plausible human-rate, volume and amortisation scenarios.
- Record the decision. State the chosen configuration, workload, date, evidence and conditions for remeasurement.
7. Audit checklist
- [ ] Every admitted task has one terminal state.
- [ ] Failed and superseded attempts remain in the ledger.
- [ ] Coordinator, worker and evaluator calls are all counted.
- [ ] Cache-inclusive tokens are not mistaken for billed tokens.
- [ ] Direct charges are distinguished from estimated prices.
- [ ] Human active time is distinguished from elapsed time.
- [ ] One-time costs and their amortisation rule are visible.
- [ ] Acceptance criteria were fixed before results were inspected.
- [ ] Quality and severe failures are reported beside cost.
- [ ] Model, tool, prompt, routing and infrastructure versions are recorded.
- [ ] Results are stratified where task difficulty or consequence differs.
- [ ] Claims are limited to the measured configuration and observation horizon.