Let a script check every quote word for word

In the interview transcript, the director of a neighbourhood food pantry says the pantry "might have to cut back to three days a week" this winter. In the news draft written from it, she says it "will". The pantry, its director and the draft are a scenario, and that one word turns a worry into an announcement.

Quotation marks tell the reader these are the speaker's exact words, and the US Supreme Court said in Masson v. New Yorker Magazine that this is what gives a quotation its authority. The Associated Press does not alter quotations "even to correct grammatical errors or word usage", and says an ellipsis "must not alter the speaker's meaning".

So when you check a write-up against an interview, split the work. Give the quotes to a script, meaning a few lines of plain code that follow the same fixed steps and give the same answer every time. A script checks the quotes because they have one right answer, and the AI checks the paraphrases because they don't. AP asks that a paraphrase be "completely true to the original quote", and deciding that takes judgement.

Why not ask a chatbot?#

A chatbot given the draft and the transcript will often spot the changed word, but you can't know which time. In SummEdits, a 2023 test of spotting small factual edits to a summary, most language models performed close to random chance. Models also do worse when what they need sits in the middle of a long input, which is where most of a transcript is. And Anthropic's own docs say asking twice can get two different answers.

In BBC research published in February 2025, four AI assistants (ChatGPT, Copilot, Gemini and Perplexity) answered news questions from BBC articles. Thirteen percent of the quotes they took from those articles were altered or weren't there at all. They were writing answers rather than checking quotes, which is why a person checks.

A script names the changed word, the same way every run. Quotation marks are a promise, and a script can check it every time.

Four steps, two of them AI#

A workflow runs steps one after another.

1. 01

   Check every quote word for wordScript

   Looks for each quoted passage in the transcript, ignoring what doesn't change the words: curly or straight quote marks, capital letters, punctuation, timestamps and speaker names. An ellipsis counts as a gap, and the table says how many words it left out.

2. 02

   Find the paraphrasesAI

   The AI, on Haiku, lists every sentence that reports what the speaker said without quotation marks.

3. 03

   Does the transcript back it?AI

   Each paraphrase goes to its own judge, which sees only that sentence and the transcript. It answers Supported, Partly or Not in the transcript, copying the line it relies on.

4. 04

   What matches?Script

   Checks each judge's copied line really is in the transcript, then builds the table.

The quote check is its own small workflow that others can call, so one fix reaches every caller. Roscoe's blog pipeline checks the quotes in each post's sources with a script like it, ellipses counted as gaps.

What the run showed#

Recorded runcheck-quotes-word-for-word-run-v1

Watch the runVideo · 0:28

On the sample scenario, the run ended on its headline: "1 of 5 quotes doesn't match the transcript word for word; 1 of 7 paraphrases is only partly backed by it".

The failing quote is the director's sentence from the top. The script matched every word but one, across two timestamped transcript lines. That change was planted in the sample, so it shows the check working, not how often drafts go wrong.

The quote "Donations are up … maybe ten percent, and demand is up closer to forty" passed, and the table named the 4 words the gap left out ("honestly, but they're up"). The meaning survives here, and the table shows the gap so you can judge that.

The judge that answered Partly had the draft's line "Lindqvist said the pantry's squeeze comes down to the end of the county's grant." It showed "The county grant ending hurts, and it's a big part of it, along with more families coming in who've never needed us before." That's because she names the grant "along with" demand from new families, not as the sole cause. You might let "comes down to" stand. Each verdict is a model's first read, so it sits beside its line.

“Quotation marks are a promise, and a script can check it every time.”

What it can't check#

The script checks words, not meaning. A quote can match every word and still be taken out of context. A gap can join two halves into something the speaker never meant, which is why the table reports every gap. The script doesn't check who said the words, and the AI can miss a paraphrase.

It also trusts the transcript. A study of OpenAI's Whisper found whole phrases or sentences that were never spoken in roughly one transcription in a hundred. The recording is still the final word.

Check your own draft#

You need Claude Code or Cowork to run it, because each paraphrase goes to its own agent. Press Run on the scenario to get the headline and the table, then swap in your own draft and transcript, or show notes and the episode's transcript. A $1 spending cap is set.