Model tests
Reading card statements (Test 1)
What 34 AI models got right and wrong reading UK card statements with the same plain prompt, and what each would cost once checking and mistakes are counted.
- Two models got almost every fee right. Grok 4.7 and Claude Opus 5.5 got almost no fees wrong across 16 statements, with the same plain prompt as every other model. Grok 4.7 cost 4.6 cents a statement and Claude Opus 5.5 cost 15.5 cents.
- Cheaper models came close. Gemini 3.8 Flash got 0.37% of fees wrong at 2.6 cents a statement, and GLM 5.3 got 0.48% at 1.4 cents. That's about $40 to $50 of fees wrong on a typical statement with $10,700 of fees.
- The mistakes were classification, not reading. Every usable model found the fee lines and read the amounts. The mistakes came from putting fees under the wrong type, such as higher charges on non-UK cards filed as penalties.
- Checking would cost more than the model. On the assumptions in this article, only Grok 4.7 and Claude Opus 5.5 are accurate enough to use without checks, at an estimated $0.05 and $0.16 a statement in full. Every other model would need a person checking some or all of its work, which takes the estimated full cost to between $5 and $16 a statement.
The Test 1 leaderboard ranks 34 AI models on reading UK card acquiring statements, all with the same plain prompt. This article covers what the results mean, a second run in which seven finalists read 22 statements including six hard ones, and what each model would cost once a person's checking time and the mistakes that get through are counted.
The test
ScalePoint Research gave 34 models from the US, Europe and China one task: read UK card acquiring statements, find every fee and spot any errors. Every model got the same plain prompt, and the results are the Test 1 leaderboard. A second run then gave seven finalists harder statements, with the same prompt, to see which of them could spot a faulty statement.
How the second run worked
- Seven finalists, chosen from the 25 usable models across the range of prices, read 22 synthetic statements: the 16 behind the leaderboard plus six hard ones with scanned pages, three outlets on one statement and two faulty statements.
- Each finalist got the same plain prompt as the leaderboard.
- Every fee in each answer was checked against the correct answer, and the errors were added up as a share of the statement's total fees. Every finalist ran on a route with zero data retention.
- Each finalist was also given an estimated full cost per statement: its price, plus the time a person would spend checking its work, plus the mistakes that would get through. No one checked the models' work in this test. The checking and mistake costs come from the assumptions in How full cost is worked out, and readers can change them in the full cost calculator.
- All money figures are shown in US dollars at $1.34 to the pound. Figures quoted from a statement stay in pounds.
The findings
Grok 4.7 and Claude Opus 5.5 got almost every fee right.
Each dot is one model's result with the plain prompt, across the 16 statements behind the leaderboard.
Claude Opus 5.5 and Grok 4.7 got almost no fees wrong across the 16 statements behind the leaderboard. The next group, Gemini 3.8 Flash, GLM 5.3, Nemotron 3 Ultra, Kimi K3 and Mistral Large 3, got 0.37% to 0.56% wrong, about $40 to $60 on a typical statement with $10,700 of fees. In all, 25 of the 34 models gave usable answers. Every usable model found the fee lines and read the amounts correctly. The mistakes came from putting fees under the wrong type, mostly higher charges on non-UK and commercial cards filed as penalties. The leaderboard has every model's result.
Checking changes the order. On the assumptions in How full cost is worked out, a model with 0.1% of fees wrong or less can be used as is, one with up to 0.5% wrong needs spot checks, and one with up to 2% wrong needs a person checking every statement. Only Grok 4.7 and Claude Opus 5.5 clear the first bar, so their estimated full cost is close to their price: $0.05 and $0.16 a statement. Gemini 3.8 Flash and GLM 5.3 would need spot checks, which takes them to about $5 and $6 a statement. From Mistral Large 3 down, every model would need a full review, at an estimated $14 to $16 a statement. At 10,000 statements a month, that would be about $5,500 a year for Grok 4.7 against about $1.7 million for Mistral Large 3, which costs 1.2 cents a statement to run. Reading a statement by hand would cost about $41.
GPT-5.5 was the surprise. OpenAI's flagship got 0.94% of fees wrong, about $100 on a typical statement, behind cheaper models from China and France, at 13.9 cents a statement. On these assumptions it would need a full review, at an estimated $14.87 a statement.
Judgement separated the finalists. In the second run, seven finalists read 22 statements with the same plain prompt, including two faulty ones: a subtotal that doesn't add up, and a statement that claims to pass interchange through at cost but charges one flat rate on every card. Claude Opus 5.5, Grok 4.7, Gemini 3.8 Flash, GLM 5.3 and DeepSeek V4.1 Flash flagged both. Mistral Large 3 and GPT-5.5 flagged neither.
What would each model cost you, mistakes and checking included?
Each model is costed the cheapest way its accuracy would allow: with no checks, with spot checks, or with a person checking every statement. The estimated full cost adds the model's price, the assumed checking time and the mistakes that would still get through. Change the three settings to use your own numbers.
Showing the default assumptions.
| # | Model | Fees wrong | Model $ | Checking $ | Mistakes $ | Full cost per statement | Cost a year |
|---|
Fixed assumptions: $10,700 of fees on a typical statement, 15 minutes to check a statement in full, 45 minutes to read one by hand, spot checks on 1 statement in 10, and a check catches 90% of errors. Models within 0.05 points of each other on fees wrong are marked as tied and share the same mistake cost, because gaps that small come from one or two fees. Results are from the 16 statements behind the leaderboard, with the plain prompt. The 10% loss share is a judgement, explained in How full cost is worked out. All figures are in US dollars, at $1.34 to the pound for the statement fees.
What this means in practice
With the plain prompt, the two most accurate models were also the cheapest to run once checking is counted, although neither had the lowest price. A model at under a cent a statement that gets 1% of fees wrong would need a person checking every statement, and on the assumed staff cost that checking would cost far more than the model saves.
A weak model choice doesn't show on the model bill. The business would pay for it in checking time or in mistakes. Hence a business needs a test set of its own, scored in money, before it chooses a model. Rules, machine learning or a model covers where a model fits in the wider process.
Questions this test leaves open
- Page images. Would sending the page image to models that can read it beat OCR on scanned statements, and at what cost?
- A harder task. Where does a difference in capability show on judgement calls that need reasoning across a whole document?
- A whole workflow run by an agent. Do accuracy and cost hold up when a model takes several steps on its own, such as fetching statements, checking them and writing up the results?
- Checking in practice. How long does a finance team really take to check a model's fee list, and how many errors does that check catch? The full cost figures rest on assumptions for both.
- A fine-tuned open-weight model. Could a small model trained on statements match Grok 4.7 and Claude Opus 5.5?
ScalePoint Research is happy to talk through the method with anyone planning a test like this on their own tasks or workflows. Contact Martin Koderisch at martin@scalepointpartners.com.
How the test was built
This part holds the statements, the faulty statements, the detailed results for the finalists and how full cost is worked out. Every model's result is on the leaderboard, and the method every test shares is on the Method page.
The statements and the second run
The seven finalists were chosen from the 25 usable models on the leaderboard, across the range of prices: the two most accurate at any price, three accurate models at the low end, one of the very cheapest, and GPT-5.5, added because readers would look for it. Each read 22 synthetic statements: the 16 behind the leaderboard plus six hard ones with scanned pages, three outlets on one statement and two faulty statements.
Every model, in both runs, got the same plain prompt: nine rules, such as record each fee once and keep currencies separate, plus the names of 18 fee types. Every model got the text of the statement, with Tesseract OCR on the scanned pages, through OpenRouter on routes with zero data retention, at temperature zero. Code marked every fee against the answer key. The whole test cost about $62 in model fees.
The faulty statements
Whether each finalist flagged the two faulty statements with the right label in the second run: a subtotal that doesn't add up, and a statement that claims to pass interchange through at cost but charges one flat rate on every card.
| Model | Faulty subtotal | Fake pass-through |
|---|---|---|
| Claude Opus 5.5 | 1/1 | 1/1 |
| Grok 4.7 | 1/1 | 1/1 |
| Gemini 3.8 Flash | 1/1 | 1/1 |
| GLM 5.3 | 1/1 | 1/1 |
| DeepSeek V4.1 Flash | 1/1 | 1/1 |
| Mistral Large 3 | 0/1 | 0/1 |
| GPT-5.5 | 0/1 | 0/1 |
An example: fake pass-through pricing
One of the hard statements is for an invented pub group, Juniper Lane Taverns. The front page says "Pricing: Interchange Plus Plus" and lists interchange "passed through at cost" of £1,492.57. These rows are from the detail page.
| Card type on the statement | Sales (£) | Interchange rate shown | What interchange should look like |
|---|---|---|---|
| Visa consumer debit, UK | 80,748.52 | 0.7100% | Capped at 0.2% |
| Mastercard consumer credit, UK | 29,746.00 | 0.7100% | Capped at 0.3% |
| Visa commercial credit, UK | 6,490.75 | 0.7099% | Not capped, so usually higher |
| Visa consumer credit, international | 3,153.36 | 0.7100% | Usually higher than UK cards |
Every row shows interchange at about 0.71% whatever the card. Real interchange changes with the card type, so the same rate on every row means the acquirer is charging one blended rate and calling it pass-through. On the UK consumer debit row alone, interchange at the 0.2% cap would have been about £161, against the £573 shown.
Mistral Large 3 spotted the faulty subtotal with the exact figures but filed it under "other", and the score only counts the right label, because any automated routing to a person works from the label.
Results in detail
Fees wrong for each finalist with the plain prompt, on the 16 statements behind the leaderboard and on the 22 statements of the second run, and the model's cost per statement on the leaderboard.
| Model | 16 statements (leaderboard) | 22 statements (second run) | Cost per statement |
|---|---|---|---|
| Claude Opus 5.5 | 0.00% | 0.04% | $0.155 |
| Grok 4.7 | 0.00% | 0.04% | $0.046 |
| Gemini 3.8 Flash | 0.37% | 0.31% | $0.026 |
| GLM 5.3 | 0.48% | 0.61% | $0.014 |
| Mistral Large 3 | 0.56% | 0.56% | $0.012 |
| GPT-5.5 | 0.94% | 0.86% | $0.139 |
| DeepSeek V4.1 Flash | 1.12% | 0.99% | $0.006 |
The second run adds six hard statements, so its figures differ a little from the leaderboard. On one scanned statement the OCR misread two figures, so no model could get those two lines right, and that accounts for most of the 0.04% the top two got wrong.
How full cost is worked out
Full cost per statement is the model's price, plus the cost of a person checking its work, plus the cost of the mistakes that still get through. Each model is costed three ways and ranked on the cheapest way its error rate allows.
| Way of running it | Who checks | Allowed when fees wrong is | Checking cost per statement | Mistakes that get through |
|---|---|---|---|---|
| Use as is | Nobody | 0.1% or less | $0 | All of them |
| Spot checks | A person checks 1 statement in 10 | 0.5% or less | $1.38 | 91% of them |
| Full review | A person checks every statement | 2% or less | $13.75 | 10% of them |
| Not acceptable | Over 2% |
Defaults. $10,700 of fees on a typical statement, $55 an hour, 15 minutes to check a statement, a check that catches 90% of the errors it looks at, and 10% of fee errors becoming a real loss. The 10% is a judgement: almost every error in this test was a fee filed under the wrong type with the amount read correctly, and that only costs money when it hides a charge the business would otherwise challenge. Readers can set their own volume, staff cost and loss share in the calculator. Models within 0.05 points of each other are treated as tied and share the same mistake cost.
Limits
- Synthetic statements. Real statements come in more layouts and many arrive as poor scans, so any business should repeat the test on its own documents.
- A small sample. The finalists read 22 statements once each, so gaps of a few hundredths of a percentage point at the top are noise.
- Text only. The models never saw the page image, so results on scans reflect the OCR as much as the model.
- Prices move quickly. All prices are OpenRouter prices in September and October 2026. Gemini 3.8 Flash's price is set to double in January 2027.
This article reflects the author's own views. The test statements are synthetic, and prices are OpenRouter prices in September and October 2026; results on real documents and at other prices will differ. Model names and trademarks belong to their owners.
This is the article behind Test 1. The first-round results for all 34 models are on the Test 1 leaderboard. The article behind Test 2 is Reconciling settlements.