Introduction
Why test models on payments and fintech tasks
There are already hundreds of AI models to choose from, and new ones arrive every few weeks. Choosing one is a question of AI economics. The model needs enough capability for the task, at a price that works at the business's volumes, and it has to clear the business's other rules, such as where the model is hosted, where the data goes and who the supplier is.
Public benchmarks test AI models in general, but few look at the AI economics of fintech tasks. Test 1 takes one payments task, reading UK card acquiring statements, and measures accuracy in money against price. It is a basic task, chosen as a first test because every answer can be checked to the penny. Every model's plain-prompt result is on the leaderboard, and the method behind the second round is further down this page.
The test
ScalePoint Research gave 34 models from the US, Europe and China one task: read UK card acquiring statements, find every fee and spot any errors. The first round used a plain prompt, and its results are the Test 1 leaderboard. The second round asked a different question: would better instructions let a cheaper model match the most expensive one?
How the second round worked
- Seven finalists, chosen from the 25 usable models across the range of prices, read 22 synthetic statements, including six hard ones with scanned pages, three outlets on one statement and two faulty statements.
- Each finalist read every statement with three prompts: the plain prompt from the leaderboard, a middle prompt that adds a one-line definition of each fee type, and an expert prompt that adds a 1,500-word briefing on how UK acquirers lay out statements and where fees hide.
- Every fee in each answer was checked against the correct answer, and the errors were added up as a share of the statement's total fees. Every finalist ran on a route with zero data retention.
- Each finalist was also costed on full cost per statement: its price, plus a person's checking time, plus the mistakes that get through. The assumptions are in section 2, and readers can change them in the full cost calculator.
- All money figures are shown in US dollars at $1.34 to the pound. Figures quoted from a statement stay in pounds.
The findings
With the plain prompt, the finalists were far apart. Claude Opus 5.5 and Grok 4.7 got almost no fees wrong, while the other five got 0.3% to 1% wrong, about $33 to $107 on a typical statement with $10,700 of fees. Every model found the fee lines and read the amounts correctly. The mistakes came from putting fees under the wrong type, mostly higher charges on non-UK and commercial cards filed as penalties. The leaderboard has every model's plain-prompt result.
The expert prompt took every finalist into the use-as-is band. All seven got 0.07% of fees wrong or less, so none of them needed checking. For GLM 5.3, Mistral Large 3, GPT-5.5 and DeepSeek V4.1 Flash, full cost fell by 95% to 97% against the same model on the plain prompt, from around $14.50 a statement to between $0.45 and $0.69. Gemini 3.8 Flash fell by 89%, from $4.42 to $0.47. At 10,000 statements a month, that is $0.5 million to $1.7 million a year for each of the five models that moved band. The briefing that did this is about 1,500 words a business can write once and own.
Once accuracy was level, price decided. With the expert prompt, the seven finalists were within one or two fees of each other across 22 statements, which is within the test's noise. Treated as tied, each carries the same mistake cost of $0.44 a statement, so their full cost runs from $0.45 for DeepSeek V4.1 Flash to $0.69 for GPT-5.5, in the order of their prices. The cheapest finalist, at half a cent a statement, matched the most expensive at 25 cents. Grok 4.7 and Claude Opus 5.5 come out a few cents higher with the expert prompt than with the plain one, because as part of the tied group they carry its average mistake rate rather than their own. At 10,000 statements a month that is $54,000 to $83,000 a year, of which the model is $590 to $29,560 and the estimated mistakes $52,800 for every finalist. The test that showed which models were good enough cost about $62.
GPT-5.5 was the surprise. OpenAI's flagship got 0.94% of fees wrong with the plain prompt, about $100 on a typical statement, behind cheaper models from China and France. It caught up with the expert prompt but still missed the faulty subtotal, and it was the most expensive finalist. Hence the common view held for Claude Opus 5.5 and Grok 4.7 and did not hold for GPT-5.5. With the plain prompt it needed a full review, at $14.81 a statement.
The higher price bought judgement. Claude Opus 5.5 and Grok 4.7 flagged both faulty statements at every prompt level and returned no failed answers. The cheaper finalists needed the expert prompt to flag them.
What this means for your AI strategy
The model market is moving fast. During this test, Google released Gemini 3.8 Flash three weeks after 3.7 Flash, Claude Opus moved from 4.8 to 5.5 in four months, and the hosts for two finalists halved their prices within a day. Hence ScalePoint Research expects more companies to run several models side by side over the next 12 months, each matched to a task and changed as better or cheaper ones arrive.
Open-weight models are already part of that shift. A September 2026 survey by Enterprise Technology Research found they handle 34% of enterprise AI token usage, up from 23% a year earlier[1]. The Financial Times has reported Tinder moving some work to open-weight models to control cost, with PNC, CH Robinson and Siemens also using them[2]. That points to a tiered set of models: open-weight models for high-volume routine tasks, and the leading closed models for the hard judgement calls.
This test adds a point about where the money is. Reading and classifying the fees was within reach of a model at half a cent a statement, but only with a good briefing. Without it, five of the seven finalists needed a person checking their work, and that checking cost far more than any model. A weak setup, whether the wrong model or a thin prompt, doesn't show on the model bill. The business pays for it in checking time or in mistakes. A router that switches models automatically doesn't fix this on its own. It picks on price, speed and general quality scores, and it only knows a task's error cost if the business gives it a test set with known answers, scored in money. Hence the instructions a business writes, and the test set it checks them against, are worth more than the choice between models of similar accuracy.
In 12 months, good practice is likely to include a test set with known answers for each important task, built from the business's own data and scored in money. It also includes prompts the business owns, so it can move between models, and a business case that counts the time people spend checking AI output. A locked model version in production should be retested when a new version or price arrives, and data and supplier rules should be set before testing starts.
Questions for management
- Which model does each of our AI use cases run on, and when did we last test that choice against the alternatives?
- Do we measure accuracy in money, on a test set built from our own data with known answers?
- How much of our own knowledge is written into our prompts, and who owns and maintains them?
- Where does each model run, under what retention terms, and which companies in the chain, including any router, see our data?
- For each task, what does each option cost once our people's checking time and the mistakes that get through are counted?
- Who checks our AI output today, how long does it take them, and is that time in the business case?
Download the executive summary with these six questions (PDF)
Questions this test leaves open
- Page images. Would sending the page image to models that can read it beat OCR on scanned statements, and at what cost?
- A harder task. Where does a difference in capability show on judgement calls that need reasoning across a whole document?
- A whole workflow run by an agent. Do accuracy and cost hold up when a model takes several steps on its own, such as fetching statements, checking them and writing up the results?
- Checking in practice. How long does a finance team really take to check a model's fee list, and how many errors does that check catch? The full cost figures rest on assumptions for both.
- A fine-tuned open-weight model. Could a small model trained on statements match the expert-prompt results without the briefing?
ScalePoint Research is happy to talk through the method with anyone planning a test like this on their own tasks or workflows. Contact Martin Koderisch at martin@scalepointpartners.com.
MMethod and results
The method and results behind the second round. The first-round results for all 34 models, and the method every test shares, are on the leaderboard and the Method page.
1How the second round worked
The seven finalists were chosen from the 25 usable models on the leaderboard, across the range of prices: the two most accurate at any price, three accurate models at the low end, one of the very cheapest, and GPT-5.5, added because readers would look for it. Each read 22 synthetic statements: the 16 behind the leaderboard plus six hard ones with scanned pages, three outlets on one statement and two faulty statements.
- Plain prompt. Nine rules, such as record each fee once and keep currencies separate, plus the names of 18 fee types. The same prompt as the leaderboard.
- Middle prompt. The plain prompt plus a one-line definition of each fee type.
- Expert prompt. The middle prompt plus a briefing of about 1,500 words on how UK acquirers lay out statements, where fees hide and how to classify the awkward ones. It was written with an AI drafting tool that never saw the test statements, so it couldn't be tuned to the answers.
Every model got the text of the statement, with Tesseract OCR on the scanned pages, through OpenRouter on routes with zero data retention, at temperature zero. Code marked every fee against the answer key. The whole test, both rounds, cost about $62 in model fees.
2How full cost is worked out
Full cost per statement is the model's price, plus the cost of a person checking its work, plus the cost of the mistakes that still get through. Each model is costed three ways and ranked on the cheapest way its error rate allows.
| Way of running it | Who checks | Allowed when fees wrong is | Checking cost per statement | Mistakes that get through |
|---|---|---|---|---|
| Use as is | Nobody | 0.1% or less | $0 | All of them |
| Spot checks | A person checks 1 statement in 10 | 0.5% or less | $1.38 | 91% of them |
| Full review | A person checks every statement | 2% or less | $13.75 | 10% of them |
| Not acceptable | Over 2% |
Defaults. $10,700 of fees on a typical statement, $55 an hour, 15 minutes to check a statement, a check that catches 90% of the errors it looks at, and 10% of fee errors becoming a real loss. The 10% is a judgement: almost every error in this test was a fee filed under the wrong type with the amount read correctly, and that only costs money when it hides a charge the business would otherwise challenge. Readers can set their own volume, staff cost and loss share in the calculator. Models within 0.05 points of each other are treated as tied and share the same mistake cost.
3Fees wrong at each prompt level
Fees wrong for each finalist at each prompt level, across 22 statements, and the model's cost per statement with the expert prompt.
| Model | Plain | Middle | Expert | Cost per statement, expert |
|---|---|---|---|---|
| Claude Opus 5.5 | 0.04% | 0.03% | 0.03% | $0.187 |
| Grok 4.7 | 0.04% | 0.11% | 0.03% | $0.049 |
| Gemini 3.8 Flash | 0.31% | 0.12% | 0.06% | $0.029 |
| Mistral Large 3 | 0.56% | 0.19% | 0.04% | $0.013 |
| GLM 5.3 | 0.61% | 0.27% | 0.03% | $0.008 |
| GPT-5.5 | 0.86% | 0.07% | 0.03% | $0.246 |
| DeepSeek V4.1 Flash | 0.99% | 2.31% | 0.07% | $0.005 |
DeepSeek V4.1 Flash did worse with the middle prompt because of one answer: on one hard statement it kept writing its working-out, hit its length limit and returned nothing, which counts as every fee missed. On one scanned statement the OCR misread two figures, so no model could get those two lines right, and that accounts for the 0.03% no model beat.
4The faulty statements, and the limits
How often each finalist flagged the two faulty statements with the right label: a subtotal that doesn't add up, and a statement that claims to pass interchange through at cost but charges one flat rate on every card. The four cheapest finalists ran twice at the middle and expert levels.
| Model | Faulty subtotal | Fake pass-through | ||||
|---|---|---|---|---|---|---|
| Plain | Middle | Expert | Plain | Middle | Expert | |
| Claude Opus 5.5 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 |
| Grok 4.7 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 |
| Gemini 3.8 Flash | 1/1 | 2/2 | 2/2 | 1/1 | 1/2 | 2/2 |
| GLM 5.3 | 1/1 | 2/2 | 2/2 | 1/1 | 0/2 | 2/2 |
| DeepSeek V4.1 Flash | 1/1 | 1/2 | 2/2 | 1/1 | 0/2 | 2/2 |
| Mistral Large 3 | 0/1 | 0/2 | 0/2 | 0/1 | 0/2 | 0/2 |
| GPT-5.5 | 0/1 | 0/1 | 0/1 | 0/1 | 0/1 | 1/1 |
An example: fake pass-through pricing
One of the hard statements is for an invented pub group, Juniper Lane Taverns. The front page says "Pricing: Interchange Plus Plus" and lists interchange "passed through at cost" of £1,492.57. These rows are from the detail page.
| Card type on the statement | Sales (£) | Interchange rate shown | What interchange should look like |
|---|---|---|---|
| Visa consumer debit, UK | 80,748.52 | 0.7100% | Capped at 0.2% |
| Mastercard consumer credit, UK | 29,746.00 | 0.7100% | Capped at 0.3% |
| Visa commercial credit, UK | 6,490.75 | 0.7099% | Not capped, so usually higher |
| Visa consumer credit, international | 3,153.36 | 0.7100% | Usually higher than UK cards |
Every row shows interchange at about 0.71% whatever the card. Real interchange changes with the card type, so the same rate on every row means the acquirer is charging one blended rate and calling it pass-through. On the UK consumer debit row alone, interchange at the 0.2% cap would have been about £161, against the £573 shown.
Mistral Large 3 spotted the faulty subtotal with the exact figures but filed it under "other", and the score only counts the right label, because any automated routing to a person works from the label.
Limits
- Synthetic statements. Real statements come in more layouts and many arrive as poor scans, so any business should repeat the test on its own documents.
- A small sample. The finalists read 22 statements once or twice each, so gaps of a few hundredths of a percentage point at the top are noise.
- Text only. The models never saw the page image, so results on scans reflect the OCR as much as the model.
- Prices move quickly. All prices are OpenRouter prices in September and October 2026. Gemini 3.8 Flash's price is set to double in January 2027.
Sources
- Techstrong.ai, "Open-Weight AI Models Expected to Capture 41% of Enterprise Token Usage", reporting Enterprise Technology Research's September 2026 survey, 22 September 2026 (techstrong.ai)
- Financial Times, report on US companies moving work to open-weight models, 27 September 2026 (ft.com)
This article reflects the author's own views. The test statements are synthetic, and prices are OpenRouter prices in September and October 2026; results on real documents and at other prices will differ. Model names and trademarks belong to their owners.