Model tests
Reconciling settlements (Test 2)
What happened when 11 models reconciled a month of card and payout settlements.
- Claude Opus 5.5 missed about $13 a run. It found every error in three of its six runs, and its worst run missed $43. It cost $0.58 a run on the smaller test and $0.90 on the larger one.
- Most cheaper models missed hundreds of dollars. On the larger test, DeepSeek V4.1 Flash saved 86 cents a run against Opus 5.5 and missed about $1,400 more. On Test 1, reading card statements, cheap models came close to the best. On reconciliation they fell well behind.
- Grok 4.7 was the value pick on the larger test. It missed $58 in each of its three runs at $0.37 a run, about half the price of GPT-5.5, which missed $146 on average. Kimi K3, an open-weights model that can run on European hosting, sat with GPT-5.5 and Sonnet 4.6 on the smaller test at $0.26 a run.
- Worst runs matter as much as averages. Gemini 3.1 Pro, GLM 5.3, DeepSeek and Gemini 3.8 Flash each had a run that missed about $2,600 or more. DeepSeek's best run on the larger test found every error.
- Mistral Large 4 gave no answer. It kept checking its own working until it ran out of output, in 13 runs on three settings. It's a public preview released on 6 October 2026.
The Test 2 leaderboard ranks nine AI models on reconciling a month of files for an invented merchant group with a UK company and a Spanish company, with 17 planted errors worth $4,832. This article covers what the results mean: why the cheapest models fell behind, which errors the models missed and why, what happened to Mistral Large 4, and what a finance team should take from it.
The test
ScalePoint Research built an invented merchant group of restaurants, shops and online stores, with a UK company and a Spanish company, and a month of its files for August 2026: settlement reports from two acquirers, bank statements, the group ledger, fee invoices, payout reports and the contracts. Every error in the files was planted, so the right answer was known to the penny.
Each model got the files and the same prompt, and had to find every error, value it and say what to do about it, in one reply with no tools. It ran three times on each of two sizes of the same month: Batch, with daily batch totals, and Slice, which adds every card transaction at the Barcelona restaurant. The test ran on Inspect[2], the open-source test framework from the UK AI Security Institute, and code marked every answer against the answer key in pounds missed. The leaderboard has the rest of the method.
The findings
Money missed per run. Dot = average of three runs, ring = worst run, dashed line = everything missed ($4,832 Batch, $4,853 Slice). Under each model: cost per run, Batch / Slice.
Bar = average of three runs, tick = worst run. Scale $0 to $5,100; everything missed is about $4,850.
Claude Opus 5.5 was the only model to stay under $50 missed in every run. It missed $15 on Batch and $12 on Slice on average, and found every error in three of its six runs. GPT-5.5 and Grok 4.7 came next, then Kimi K3 and Claude Sonnet 4.6. Every other model missed hundreds of dollars on average. The chart shows each model's average and worst run.
The cheapest models fell well behind. DeepSeek V4.1 Flash costs 4 cents a run. It missed $508 on Batch and $1,422 on Slice on average, and $3,132 in its worst run, although its best Slice run found all 18 errors. GLM 5.3 found about as many errors as GPT-5.5 on Batch but missed eight times as much money, because all three of its runs missed the £705 settlement that was never paid.
Grok 4.7 was the value pick on the larger test. It missed the same $58 in all three Slice runs at $0.37 a run. On Batch one run missed the £450 fee that isn't in the contract, which pulls its average there to $265.
Some models gave no answer at all. Gemini 3.8 Flash ran out of output before answering in three of its six runs. When it did answer, it missed $86 to $667. Mistral Large 4 gave no answer in any run, which the next section covers.
False alarms were rare. There were 8 in 54 runs, and six came from Gemini 3.1 Pro, mostly about Acquirer B fees in the ledger.
The errors missed most were small and needed two files read together. Size didn't predict difficulty: the €1,704 misposting was found in 25 of 27 Batch runs, while a €50 Bizum payment with no ledger entry was found in 14 and a £16 margin overcharge in 12.
The six errors found least often: runs that found each one, out of 27 runs by the nine models that answered. Batch Slice. The debit-rate error is in Slice only.
- Bizum payment with no ledger entry. The error is an entry that isn't there. To find it, a model has to start from Acquirer B's Bizum lines and look for each one in the ledger.
- Margin above contract at the UK restaurant. The invoice shows only the service charge. The model has to work out the contract charge, 0.25% of sales plus 2p per sale, and compare. The same error at the UK shop was found in 26 of 27 runs. The restaurant's invoice also carries the £450 fee that isn't in the contract, and models may stop once they've flagged that. That idea hasn't been tested.
- Debit cards charged the credit rate. The model has to check the fee on each card line against the card type, across a week of transactions.
- Short weekend settlement. Acquirer B paid three batches, including the short Saturday one, in one bank credit. The model has to add up the three batches before comparing with the bank.
- Settlement never paid. Found in 22 of 27 Batch runs and only 15 of 27 Slice runs. The extra transaction lines in Slice seem to push it out.
Mistral Large 4
Mistral Large 4 gave no answer in 13 runs on three settings. With thinking capped at 16,000 tokens, all six runs thought to the 64,000-token output limit. On reasoning effort medium, five of six hit the limit and one stopped with no answer the scorer could read. A single run on effort low also hit the limit, after 18 minutes.
Its reasoning wasn't going round in circles. In the low-effort run it was on a final check of errors it had already found, and the five amounts it quoted all match the answer key. It went back over each one several times and ran out of output before writing the answer. Changing the setting made no difference to how long it thought, so on OpenRouter its thinking can't be capped. Artificial Analysis measured about 52,000 reasoning tokens per task for Large 4[1].
Mistral Large 4 is a public preview, released on 6 October 2026, ten days before these runs. ScalePoint Research will test it again once Mistral releases the final version or the open weights.
Mistral Large 3 (v25.12[3]) answered its single check run on Batch and found 5 of 17 errors. The full run stalled on repeated "too many requests" errors from Mistral's endpoint on OpenRouter, so it's recorded as not completed because of provider rate limits.
What this means in practice
For reconciliation, the choice of model matters far more than it did for reading statements. One missed settlement of £705 (about $945) pays for about a thousand runs of Claude Opus 5.5. Hence the price per run should be weighed against the money at stake, and the worst run looked at as well as the average.
Code can do much of the matching. The plain-code reconciler written to check the answer key found every error in seconds with no model. It was written knowing what was planted, so it isn't a fair comparison. It does show that the matching rules here are fixed, and a model is probably better used on the exceptions: explaining each one and deciding what to do. Rules, machine learning or a model covers that split across the processes on the map.
Questions this test leaves open
- A hybrid set-up. How well does a model do when code does the matching and the model only explains each exception and decides what to do?
- A full month. Do the results hold on the full 30-site month, split across several calls?
- Tools. Would a model that can run code or use a spreadsheet do better on the arithmetic?
- Routes. Test 2 ran on OpenRouter's default routes. Do the results change on the zero-retention routes a regulated firm would have to use?
ScalePoint Research is happy to talk through the method with anyone planning a test like this on their own reconciliation. Contact Martin Koderisch at martin@scalepointpartners.com.
How the test was built
This part holds the merchant group and its files, the planted errors and the detailed results. Every model's result is on the leaderboard, and the method every test shares is on the Method page.
The merchant group and its files
The month is August 2026, which ends on a UK bank holiday.
- Acquirer A serves the UK restaurants and shops. It prices on interchange plus plus, pays each day's sales gross the next working day and sends a fee invoice at month end.
- Acquirer B serves the online stores and Spain. It charges blended rates, pays net two working days later and combines all batches for one settlement account into one bank credit per value date. It also handles Bizum, pay by bank and the group's payouts.
- Concessions are other brands trading inside the UK shops. The group collects their card sales and pays them out weekly, less a commission set in their agreement.
| File | What it holds |
|---|---|
| Acquirer A batches | One row per site per day: sales, refunds and amount settled |
| Acquirer A refund lines | Every refund, with the original payment it refers to |
| Acquirer A invoice | Monthly fees per merchant ID |
| Acquirer B settlement summary | One row per batch, with the payment reference that ties it to the bank |
| Acquirer B exception lines | Refunds, chargebacks, chargeback reversals, Bizum and pay by bank payments |
| Acquirer B payout report | Concession payouts and bank-transfer refunds |
| Bank statements | UK Ltd in GBP and Spain S.L. in EUR, each in its bank's own format |
| Group ledger | Journal lines for takings, refunds, chargebacks and payouts |
| Chargeback notices | Notices the acquirers sent during the month |
| Contracts | Pricing for both acquirers and the concession agreements |
| FX reference rates | Daily EUR to GBP rate |
Slice also includes every card transaction for the Barcelona restaurant, with the fee Acquirer B charged on each one.
The planted errors
The month holds 17 planted errors worth $4,832 in Batch, and one more in Slice that needs the transaction lines. They fall into six types:
- Money that never arrived or arrived short, such as a settlement that was never paid.
- Fees charged above the contract, such as a fee that isn't in the contract at all.
- Refunds and chargebacks handled wrongly, such as a refund larger than the one agreed.
- Payouts to third parties paid wrongly, such as a commission at the wrong rate.
- Ledger entries missing, doubled or booked to the wrong company.
- Currency and account errors, such as a euro settlement paid into the sterling account.
They range from a four-figure misposting to a margin overcharge of about £12. The files also hold items that look like errors and aren't, such as settlements due after month end, a settlement paid late but in full, and small rounding differences between batch fees and line fees. Flagging any of them counts as a false alarm. The largest error is about 40% of the money at stake, so errors found is reported next to money missed.
The full list of errors, where they sit and their amounts stays private, so future models can't learn it and the test still measures reasoning when it's rerun.
Results in detail
Every model's average and worst run, the errors it found on each file size, its false alarms, its cost per run and its price are on the Test 2 leaderboard, with a ± figure for how much each result moved from run to run.
Limits
- The files are clean. Every file is a tidy CSV or text file. Real acquirer reports and bank exports are messier, and some arrive as PDFs.
- The errors and the tolerances were chosen by the author. Four errors were added after the first pilot runs because the test was too easy. A different set of errors would give different scores.
- One reply, no tools. A model that can run code or use a spreadsheet may do better on the arithmetic.
- Three runs per model. That's enough to show which models are inconsistent, and too few to rank models that sit close together, such as GPT-5.5 and Kimi K3 on Batch.
- Real months are bigger. The full invented group has 30 sites, and its month comes to about 800,000 tokens. That won't fit in one reply for any of these models, so a real set-up would split the work.
- List prices. Costs are OpenRouter list prices. Direct contracts and batch pricing can be lower.
- Fixes during the build. After the check runs, four models flagged a concession column whose label was ambiguous. The label was fixed before the full runs. The way the scorer matches answers to errors was also tightened, and every saved run was rescored with the final version.
Sources
- eesel, "Mistral Large 4: specs, pricing, benchmarks, and who should use it", October 2026, citing Artificial Analysis (eesel.ai)
- UK AI Security Institute, Inspect, open-source framework for testing AI models (inspect.aisi.org.uk)
- Mistral AI, models overview, October 2026 (docs.mistral.ai)
This is the article behind Test 2. The results, the method and every model's figures are on the Test 2 leaderboard. The article behind Test 1 is Reading card statements.