# Don't pay AI to guess at math

Maya's last message in the group chat says she added everything up from the receipts: "$1,289.85 total, so $322.46 each", and she's sending the requests tonight. The chat is a sample scenario, a weekend at a lake cabin for four friends, and her total leaves out the $18.60 Sam paid in cash for firewood and ice, with no receipt.

```chat
label: The group chat
note: Sample scenario
from: Maya
time: 6/13 4:12 PM
text: cabin's booked!! paid the whole thing on my card, $712.36 w/ the cleaning fee

gap: 2 more messages

from: Olivia
time: Sat 11:02 AM
text: groceries done, ~$188 at Hannaford. breakfast stuff, burger stuff, snacks

from: Olivia
time: Sat 11:03 AM
text: also grabbed sunscreen + bug spray, 23.84, that's for everyone right?

from: Jake
time: Sat 11:20 AM
text: I didn't use the sunscreen once lol

from: Sam
time: Sat 5:31 PM
text: firewood + 2 bags of ice at the camp store, ==$18.60 cash==, no receipt sorry

gap: 3 more messages

from: Maya
time: Sun 8:02 PM
text: ok I added everything up from the receipts: ==$1,289.85 total==, so $322.46 each. sending requests tonight
caption: Part of the sample chat. Highlighted: the cash with no receipt, and the total that leaves it out. The workflow reads all of it, plus the receipts people shared.
```

That chat holds three kinds of work. There's reading: who paid for what, that Jake sat out the kayaks, that the "~$188" of groceries is the $187.94 on the receipt. There's math: the total, the shares, who pays whom. And there's a call nobody in the chat settles. Olivia bought $23.84 of sunscreen and bug spray and asks if that's for everyone, and Jake says he didn't use the sunscreen once.

Each kind of work belongs with someone different. Give the AI the reading. Give a script every step that has one right answer, including the step that checks what the AI read. Ask a person about the calls neither of them can settle. Every step you hand the AI instead is another paid guess that can come back different next time.

## Same input, same answer?

```plainwords
term: Deterministic
means: The same input gives the same answer each time, like a calculator. A script works that way, since it is a fixed set of steps written as plain code.

term: Non-deterministic
means: An LLM, the kind of AI behind ChatGPT and Claude, writes by choosing each next word by probability. Because every word is a weighted guess, the same input can come back different, and no answer is certain to be right.
```

That [sampling from likely options](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/) holds even with the AI's randomness setting turned all the way down: [Anthropic's own docs](https://platform.claude.com/docs/en/about-claude/glossary) say "identical inputs may produce different outputs". The same guessing is what lets an AI make sense of a messy chat. It is also a good reason to keep it away from the sums.

```figure
src: /blog/decide-the-math-first/deterministic-vs-not.png
dark: /blog/decide-the-math-first/deterministic-vs-not-dark.png
alt: Two panels. On the left, a calculator given the same sum three times gives the same answer each time. On the right, an AI asked to describe the same photo three times gives three different descriptions of a dog on a beach.
caption: A script repeats itself. An AI makes a fresh guess each time.
```

So here is a test for any step. If a step has one right answer, don't pay for a guess. Give it to a script, which works that answer out the same way every time. If it takes reading or writing, give it to the AI. A call about what's fair, which the AI could only guess at, goes to a person.

None of this is new. A 2022 research paper called [PAL](https://arxiv.org/abs/2211.10435) found that models "often make logical and arithmetic mistakes" even after breaking a problem down correctly, so it had the model read while a program solved. [Anthropic's guide to building AI agents](https://www.anthropic.com/engineering/building-effective-agents) says fixed steps offer "predictability and consistency for well-defined tasks", and has agents "pause for human feedback at checkpoints".

A chatbot might well get a trip this small right. The trouble is that you can't tell which time it will.

## Why the guesses add up

Every step you give an AI is a fresh guess. [Research on long tasks](https://arxiv.org/abs/2509.09677) found that a model's accuracy on each step falls as the steps pile up, and that models slip more often once their own earlier mistakes are in front of them. [Anthropic warns](https://www.anthropic.com/engineering/building-effective-agents) that letting an AI run a job on its own brings "higher costs, and the potential for compounding errors".

```figure
src: /blog/decide-the-math-first/uncertainty-adds-up.png
dark: /blog/decide-the-math-first/uncertainty-adds-up-dark.png
alt: Two chains of five boxes. In the top chain every step is an AI step, four paid guesses, and a widening band shows the spread growing toward a result that may be right. In the bottom chain only the first step is AI and scripts do the rest, so the band narrows to a single line and the result is the same every time.
caption: A picture of the idea, not a measurement.
```

Roscoe measured this on a ledger with 80 entries to work through, 30 runs each, with [Haiku](#term 'The cheap Claude model this benchmark ran on.') on both sides. Stronger models on their own reach about 100% too, and [the evidence page](/evidence) has the details.

```keynumber
value: 1 in 3
prefix: About
bar: 37
label: Right final answers from a single Claude agent on Haiku that had code tools and used them every run

value: 100%
prefix: About
bar: 100
emphasis: yes
label: The same model doing only the reading, with a fixed script doing the math

source: Roscoe’s benchmark on the evidence page
source-url: /evidence
```

## Four steps, one of them AI

A workflow is a set of steps that run in order, some done by AI and some by scripts.

```workflow
id: demo-10-split-the-trip
caption: One AI step reads, a script checks the reading, you make the call, and a second script does the math.
```

```steps
step: Read
kind: ai
text: Reads the chat and receipts on Haiku and writes down each expense, with no arithmetic at all. Where it made a call, it says what it chose, why, and the other way it could have gone.

step: Check
kind: script
text: Makes sure the reading accounts for every dollar amount in the text, uses no amount that isn't there, and names only people on the trip. If anything is off, the AI reads again with a note, up to 3 tries in all.

step: Ask
kind: you
text: The run stops and shows you each judgement call the AI made, to keep or to overrule.

step: Settle
kind: script
text: Adds up every expense, compares its total with the chat's, splits each cost only among the people it was for, and ends on who pays whom.
```

## What the run did

```video
key: media/blog/decide-the-math-first-run-v1
poster: decide-the-math-first-run
duration: 27.43
```

On the sample scenario, the AI's first reading accounted for all 19 dollar amounts in the chat and receipts, so the check never sent it back.

That reading made one judgement call, splitting the $23.84 of sunscreen and bug spray among all four friends. The reason it gave: "Olivia explicitly asked if the purchase was for everyone when she bought it; Jake's response saying he didn't use the sunscreen appears joking rather than a firm objection."

That's a fair reading, and also a guess about whether Jake was joking, which a script can't decide. So the run stopped and asked. The person overruled it, and Jake was left out of the $23.84 in the sample.

Then the script did every sum on that answer, starting with: "Recomputed from 8 expenses: the chat's running total is off by $18.60 (Maya said $1,289.85; the expenses add up to $1,308.45)." That gap was a person's mistake, typed into the sample scenario's chat. The AI did no math at all.

In the sample, Maya's $322.46 each also assumed an even split. With Jake out of the kayaks and now the sunscreen, the fair shares come to $337.76 to $337.80 for the other three and $295.11 for Jake. Maya paid $816.61 up front, so she gets back $478.81: Sam owes $255.89, Olivia owes $126.00 and Jake owes $96.92. The run cost $0.0852, a subscription-equivalent estimate at list price, not money charged.

```pullquote
text: If a step has one right answer, don't pay for a guess.
```

## What the check can't see

The check counts; it can't judge. And the pause shows only the calls the AI flagged. If the AI misreads who a cost was for and doesn't treat it as a call, the mistake goes straight to the math without anyone being asked. The script then does exact sums on the wrong split.

```limits
title: What the check can't catch
item: It counts; it can't judge whether the reading is right.
item: The pause shows only the calls the AI flagged. A misreading it didn't flag goes straight to the math.
item: It spots only amounts with a $ sign or cents, so “forty bucks” would slip past.
item: Two identical charges look like one.
item: At the pause you can keep or overrule, not edit.
item: The sample is a scenario with its wrong total put there on purpose.
```

## Run it yourself

Run it from Claude Code or Cowork. Press Run on the sample scenario, answer the pause about the sunscreen, and you get the script's total set against the chat's and a table of who pays whom. Then paste in your own trip's chat and receipts. It carries a $1 spending cap.

```download
workflow: demo-10-split-the-trip
```