Introduction
Why test models on reconciliation
Reconciliation is how a finance team checks that the money it expected actually arrived. For a merchant taking cards, that means matching what the tills recorded against what each acquirer settled, what landed in the bank and what was booked in the ledger. Fees, refunds, chargebacks and payouts to third parties all sit in between.
Test 1 tested one step of this: reading a card statement and pulling out the fees. With good instructions, cheap models did as well as expensive ones. Reconciliation asks more of a model. It has to hold several files at once, match records that use different references, apply the contract and decide which differences are errors and which are timing. Hence the question for Test 2: does a cheap model still match an expensive one when the job needs reasoning across several sources?
The test
ScalePoint Research built an invented merchant group of restaurants, shops and online stores, with a UK company and a Spanish company, and a month of its files for August 2026: settlement reports from two acquirers, bank statements, the group ledger, fee invoices, payout reports and the contracts. Every error in the files was planted, so the right answer was known to the penny.
Each model got the files and the same prompt, and had to find every error, value it and say what to do about it, in one reply with no tools. It ran three times on each of two sizes of the same month: Batch, with daily batch totals, and Slice, which adds every card transaction at the Barcelona restaurant. The test ran on Inspect[2], the open-source test framework from the UK AI Security Institute, and code marked every answer against the answer key in pounds missed. The leaderboard has the rest of the method.
The findings
Claude Opus 5.5 was the only model to stay under £35 missed in every run. It missed £11 on Batch and £9 on Slice on average, and found every error in three of its six runs. GPT-5.5 and Grok 4.7 came next, then Kimi K3 and Claude Sonnet 4.6. Every other model missed hundreds of pounds on average. The chart above shows each model's average and worst run.
The cheapest models fell well behind. DeepSeek V4.1 Flash costs 4 cents a run. It missed £379 on Batch and £1,061 on Slice on average, and £2,337 in its worst run, although its best Slice run found all 18 errors. GLM 5.3 found about as many errors as GPT-5.5 on Batch but missed eight times as much money, because all three of its runs missed the £705 settlement that was never paid.
Grok 4.7 was the value pick on the larger test. It missed the same £43 in all three Slice runs at $0.37 a run. On Batch one run missed the £450 fee that isn't in the contract, which pulls its average there to £198.
Some models gave no answer at all. Gemini 3.8 Flash ran out of output before answering in three of its six runs. When it did answer, it missed £64 to £498. Mistral Large 4 gave no answer in any run, which the next section covers.
False alarms were rare. There were 8 in 54 runs, and six came from Gemini 3.1 Pro, mostly about Acquirer B fees in the ledger.
The errors missed most were small and needed two files read together. Size didn't predict difficulty: the €1,704 misposting was found in 25 of 27 Batch runs, while a €50 Bizum payment with no ledger entry was found in 14 and a £16 margin overcharge in 12.
The six errors found least often: runs that found each one, out of 27 runs by the nine models that answered. Batch Slice. The debit-rate error is in Slice only.
- Bizum payment with no ledger entry. The error is an entry that isn't there. To find it, a model has to start from Acquirer B's Bizum lines and look for each one in the ledger.
- Margin above contract at the UK restaurant. The invoice shows only the service charge. The model has to work out the contract charge, 0.25% of sales plus 2p per sale, and compare. The same error at the UK shop was found in 26 of 27 runs. The restaurant's invoice also carries the £450 fee that isn't in the contract, and models may stop once they've flagged that. That idea hasn't been tested.
- Debit cards charged the credit rate. The model has to check the fee on each card line against the card type, across a week of transactions.
- Short weekend settlement. Acquirer B paid three batches, including the short Saturday one, in one bank credit. The model has to add up the three batches before comparing with the bank.
- Settlement never paid. Found in 22 of 27 Batch runs and only 15 of 27 Slice runs. The extra transaction lines in Slice seem to push it out.
Mistral Large 4 and the side test
Mistral Large 4 gave no answer in 13 runs on three settings. With thinking capped at 16,000 tokens, all six runs thought to the 64,000-token output limit. On reasoning effort medium, five of six hit the limit and one stopped with no answer the scorer could read. A single run on effort low also hit the limit, after 18 minutes.
Its reasoning wasn't going round in circles. In the low-effort run it was on a final check of errors it had already found, and the five amounts it quoted all match the answer key. It went back over each one several times and ran out of output before writing the answer. Changing the setting made no difference to how long it thought, so on OpenRouter its thinking can't be capped. Artificial Analysis measured about 52,000 reasoning tokens per task for Large 4[1].
Mistral Large 4 is very new. Mistral released it on 6 October 2026 as a public preview, ten days before these runs, and has said it will publish the weights by the end of October, which would let it run on European hosting. Hence ScalePoint Research is keen to test it properly and is running a side test with the same prompt, files and scorer. The side test changes only how the model is called:
- Reasoning switched off, with the same 64,000-token limit as every other model. This result can be ranked against the other models.
- Reasoning effort low, with the output limit doubled to 128,000 tokens. This shows how far the model gets with more room. It won't be ranked, because the budget is no longer equal.
This article and the leaderboard will be updated when the side test is done, and again when Mistral releases the final version of the model.
Mistral Large 3 (v25.12[3]) answered its single check run on Batch and found 5 of 17 errors. The full run stalled on repeated "too many requests" errors from Mistral's endpoint on OpenRouter, so it's recorded as not completed because of provider rate limits.
What this means for your AI strategy
For reconciliation, the choice of model matters far more than it did for reading statements. Opus 5.5 costs under a dollar a run here, and one missed settlement of £705 pays for about a thousand runs. Hence the price per run should be weighed against the money at stake, and the worst run looked at as well as the average. Before trusting a model with a process, a finance team can run it several times on the same month and compare the answers.
Cheaper options can still make sense. Grok 4.7 is the value pick on the larger test. Kimi K3 has open weights, which matters for card and bank data. The provider's data terms need checking either way.
Code can do much of the matching. The plain-code reconciler written to check the answer key found every error in seconds with no model. It was written knowing what was planted, so it isn't a fair comparison. It does show that the matching rules here are fixed, and a model is probably better used on the exceptions: explaining each one and deciding what to do.
Real months are bigger. The full invented group has 30 sites, and its month comes to about 800,000 tokens. That won't fit in one reply for any of these models, so a real set-up would split the work.
Questions for management
- Which steps in our reconciliation follow fixed rules that code could do today?
- If we use a model, how do we check its worst run as well as its average?
- What does a missed error cost us, compared with the price of a run?
- Where does our card and bank data go, and could the model run on hosting we control?
Questions this test leaves open
- A hybrid set-up. How well does a model do when code does the matching and the model only explains each exception and decides what to do?
- A full month. Do the results hold on the full 30-site month, split across several calls?
- Tools. Would a model that can run code or use a spreadsheet do better on the arithmetic?
- Prompts. Would a more detailed prompt narrow the gap, as it did in Test 1?
- Routes. Test 2 ran on OpenRouter's default routes. Do the results change on the zero-retention routes a regulated firm would have to use? The next version of the test will show.
ScalePoint Research is happy to talk through the method with anyone planning a test like this on their own reconciliation. Contact Martin Koderisch at martin@scalepointpartners.com.
MThe detail
This part holds the files and the full list of planted errors behind the article above. The method, every model's results and the prices are on the leaderboard.
1The merchant group and its files
The month is August 2026, which ends on a UK bank holiday.
- Acquirer A serves the UK restaurants and shops. It prices on interchange plus plus, pays each day's sales gross the next working day and sends a fee invoice at month end.
- Acquirer B serves the online stores and Spain. It charges blended rates, pays net two working days later and combines all batches for one settlement account into one bank credit per value date. It also handles Bizum, pay by bank and the group's payouts.
- Concessions are other brands trading inside the UK shops. The group collects their card sales and pays them out weekly, less a commission set in their agreement.
| File | What it holds |
|---|---|
| Acquirer A batches | One row per site per day: sales, refunds and amount settled |
| Acquirer A refund lines | Every refund, with the original payment it refers to |
| Acquirer A invoice | Monthly fees per merchant ID |
| Acquirer B settlement summary | One row per batch, with the payment reference that ties it to the bank |
| Acquirer B exception lines | Refunds, chargebacks, chargeback reversals, Bizum and pay by bank payments |
| Acquirer B payout report | Concession payouts and bank-transfer refunds |
| Bank statements | UK Ltd in GBP and Spain S.L. in EUR, each in its bank's own format |
| Group ledger | Journal lines for takings, refunds, chargebacks and payouts |
| Chargeback notices | Notices the acquirers sent during the month |
| Contracts | Pricing for both acquirers and the concession agreements |
| FX reference rates | Daily EUR to GBP rate |
Slice also includes every card transaction for the Barcelona restaurant, with the fee Acquirer B charged on each one.
2The planted errors
The month holds 17 planted errors worth £3,606 in Batch, and one more in Slice that needs the transaction lines. They fall into six types:
- Money that never arrived or arrived short, such as a settlement that was never paid.
- Fees charged above the contract, such as a fee that isn't in the contract at all.
- Refunds and chargebacks handled wrongly, such as a refund larger than the one agreed.
- Payouts to third parties paid wrongly, such as a commission at the wrong rate.
- Ledger entries missing, doubled or booked to the wrong company.
- Currency and account errors, such as a euro settlement paid into the sterling account.
They range from a four-figure misposting to a margin overcharge of about £12. The files also hold items that look like errors and aren't, such as settlements due after month end, a settlement paid late but in full, and small rounding differences between batch fees and line fees. Flagging any of them counts as a false alarm. The largest error is about 40% of the money at stake, so errors found is reported next to money missed.
The full list of errors, where they sit and their amounts stays private, so future models can't learn it and the test still measures reasoning when it's rerun.
3Limits
- The files are clean. Every file is a tidy CSV or text file. Real acquirer reports and bank exports are messier, and some arrive as PDFs.
- The errors and the tolerances were chosen by the author. Four errors were added after the first pilot runs because the test was too easy. A different set of errors would give different scores.
- One reply, no tools. A model that can run code or use a spreadsheet may do better on the arithmetic. Test 2 also didn't test whether a more detailed prompt narrows the gap, as Test 1 did.
- Three runs per model. That's enough to show which models are inconsistent, and too few to rank models that sit close together, such as GPT-5.5 and Kimi K3 on Batch.
- List prices. Costs are OpenRouter list prices. Direct contracts and batch pricing can be lower.
- Fixes during the build. After the check runs, four models flagged a concession column whose label was ambiguous. The label was fixed before the full runs. The way the scorer matches answers to errors was also tightened, and every saved run was rescored with the final version.
Sources
Sources are cited by number in the article above.
- eesel, "Mistral Large 4: specs, pricing, benchmarks, and who should use it", October 2026, citing Artificial Analysis (eesel.ai)
- UK AI Security Institute, Inspect, open-source framework for testing AI models (inspect.aisi.org.uk)
- Mistral AI, models overview, October 2026 (docs.mistral.ai)