Don't pay AI to guess at math

Maya's last message in the group chat says she added everything up from the receipts: "$1,289.85 total, so $322.46 each", and she's sending the requests tonight. The chat is a sample scenario, a weekend at a lake cabin for four friends, and her total leaves out the $18.60 Sam paid in cash for firewood and ice, with no receipt.

That chat holds three kinds of work. There's reading: who paid for what, that Jake sat out the kayaks, that the "~$188" of groceries is the $187.94 on the receipt. There's math: the total, the shares, who pays whom. And there's a call nobody in the chat settles. Olivia bought $23.84 of sunscreen and bug spray and asks if that's for everyone, and Jake says he didn't use the sunscreen once.

Each kind of work belongs with someone different. Give the AI the reading. Give a script every step that has one right answer, including the step that checks what the AI read. Ask a person about the calls neither of them can settle. Every step you hand the AI instead is another paid guess that can come back different next time.

Same input, same answer?#

That sampling from likely options holds even with the AI's randomness setting turned all the way down: Anthropic's own docs say "identical inputs may produce different outputs". The same guessing is what lets an AI make sense of a messy chat. It is also a good reason to keep it away from the sums.

So here is a test for any step. If a step has one right answer, don't pay for a guess. Give it to a script, which works that answer out the same way every time. If it takes reading or writing, give it to the AI. A call about what's fair, which the AI could only guess at, goes to a person.

None of this is new. A 2022 research paper called PAL found that models "often make logical and arithmetic mistakes" even after breaking a problem down correctly, so it had the model read while a program solved. Anthropic's guide to building AI agents says fixed steps offer "predictability and consistency for well-defined tasks", and has agents "pause for human feedback at checkpoints".

A chatbot might well get a trip this small right. The trouble is that you can't tell which time it will.

Why the guesses add up#

Every step you give an AI is a fresh guess. Research on long tasks found that a model's accuracy on each step falls as the steps pile up, and that models slip more often once their own earlier mistakes are in front of them. Anthropic warns that letting an AI run a job on its own brings "higher costs, and the potential for compounding errors".

Roscoe measured this on a ledger with 80 entries to work through, 30 runs each, with Haiku on both sides. Stronger models on their own reach about 100% too, and the evidence page has the details.

Four steps, one of them AI#

A workflow is a set of steps that run in order, some done by AI and some by scripts.

1. 01

   ReadAI

   Reads the chat and receipts on Haiku and writes down each expense, with no arithmetic at all. Where it made a call, it says what it chose, why, and the other way it could have gone.

2. 02

   CheckScript

   Makes sure the reading accounts for every dollar amount in the text, uses no amount that isn't there, and names only people on the trip. If anything is off, the AI reads again with a note, up to 3 tries in all.

3. 03

   AskYou

   The run stops and shows you each judgement call the AI made, to keep or to overrule.

4. 04

   SettleScript

   Adds up every expense, compares its total with the chat's, splits each cost only among the people it was for, and ends on who pays whom.

What the run did#

Recorded rundecide-the-math-first-run-v1

Watch the runVideo · 0:27

On the sample scenario, the AI's first reading accounted for all 19 dollar amounts in the chat and receipts, so the check never sent it back.

That reading made one judgement call, splitting the $23.84 of sunscreen and bug spray among all four friends. The reason it gave: "Olivia explicitly asked if the purchase was for everyone when she bought it; Jake's response saying he didn't use the sunscreen appears joking rather than a firm objection."

That's a fair reading, and also a guess about whether Jake was joking, which a script can't decide. So the run stopped and asked. The person overruled it, and Jake was left out of the $23.84 in the sample.

Then the script did every sum on that answer, starting with: "Recomputed from 8 expenses: the chat's running total is off by $18.60 (Maya said $1,289.85; the expenses add up to $1,308.45)." That gap was a person's mistake, typed into the sample scenario's chat. The AI did no math at all.

In the sample, Maya's $322.46 each also assumed an even split. With Jake out of the kayaks and now the sunscreen, the fair shares come to $337.76 to $337.80 for the other three and $295.11 for Jake. Maya paid $816.61 up front, so she gets back $478.81: Sam owes $255.89, Olivia owes $126.00 and Jake owes $96.92. The run cost $0.0852, a subscription-equivalent estimate at list price, not money charged.

“If a step has one right answer, don't pay for a guess.”

What the check can't see#

The check counts; it can't judge. And the pause shows only the calls the AI flagged. If the AI misreads who a cost was for and doesn't treat it as a call, the mistake goes straight to the math without anyone being asked. The script then does exact sums on the wrong split.

Run it yourself#

Run it from Claude Code or Cowork. Press Run on the sample scenario, answer the pause about the sunscreen, and you get the script's total set against the chat's and a table of who pays whom. Then paste in your own trip's chat and receipts. It carries a $1 spending cap.