Judgement calls in payments operations: Claude Opus 5.5 and Mistral Medium 3.5 across nine processes (S3)

Background and context

The ScalePoint Research process map sorts 51 processes into 11 clusters. The first two studies, Reading card statements (S1) and Reconciling settlements (S2), covered the first two clusters. This study, S3, covered the other nine clusters, with one process from each, as set out under Scope.

S1 and S2 each went deep on one process. S1 compared 34 models on reading card statements, and S2 compared 11 on reconciling settlements. Across most of the skills those studies covered, the models came out close to each other. In S1, with the same plain prompt for every model, the best low-cost models got less than 0.5% of fees wrong, against almost none for Claude Opus 5.5. Where the models did pull apart, it was on cases that needed judgement or reasoning across several files, as in S2. Hence, for most of the work in those two processes, the low-cost models were close enough that the frontier model's extra cost bought little.

Hence this study focused on judgement. It was built to stretch the models on the decisions that rules and machine learning leave to a person, such as whether to fight a chargeback or clear a sanctions alert, and to see whether a real difference in capability showed up. In most firms these decisions are still made by staff, with software doing the matching, lookups and sums around them, so the study tested only that judgement step.

The key questions. The study set out to answer three questions, each with a working assumption going in:

1. On judgement steps, is there a substantial gap between the best model available and a low-cost open-weight model? The study expected the frontier model to be clearly more accurate, as S2 suggested.
2. Can more reasoning close that gap? Most models can be set to think a case through step by step before answering, at extra cost. The study expected more reasoning to close some of the gap, so that the low-cost model with reasoning on could get close to the frontier model for less money, and high effort would lift the frontier model on the hardest cases.
3. Once every mistake is costed in money, does the low-cost model still save money, or does the frontier model pay for itself?

Study design and set-up

Scope

Nine processes were in scope, one from each of the nine clusters not covered by S1 and S2. Each was set at one type of firm:

• Merchant: chargebacks, payment performance and accounts payable.
• EMI: payment exceptions, sanctions alert review and treasury prefunding.
• Acquirer: deal pricing.
• Issuer or bank: scheme changes.
• Marketplace: seller onboarding.

The unit of measurement was the case. In this study, a case was defined as one judgement-based decision the model had to make, such as whether to fight one chargeback or clear one sanctions alert. Every process had 22 cases, so 198 across the nine processes. In each process, the 22 cases came in three levels of difficulty:

• Five easy cases, where the records pointed clearly to one answer.
• Five medium cases, with a twist such as two lookalike records or a rule that changes the answer.
• Twelve hard cases, with missing information, sources that conflicted, rules that didn't quite fit, or a lot of money at stake.

Test procedure

Each case ran in four steps:

1. A Python script built the case pack. Each case was first written as a detailed brief holding every fact, figure and the right answer. The script, plain Python with no AI model, turned the brief into the documents a person would look at before deciding, with sums and deadlines already worked out and every document given an ID. For a chargeback, for example, the pack held the notice, the original transaction, the order, the delivery record, any refunds and the customer's emails.
2. The model read the pack. It got the firm's manual as its instructions, so it worked to the same rules a member of staff would.
3. The model made one decision and cited its evidence. It picked from a fixed list of decisions. For a chargeback, for example, the choices were to fight it, accept it, accept part of it, escalate it to a person or take no action. With the decision it gave a short reason, the IDs of the documents it relied on, any problems it noticed in the pack, such as missing evidence or records that disagreed, and how confident it was, from 0 to 100%. An illustrative chargeback answer: escalate, because the checkout record and the order system give different delivery postcodes; relied on the checkout record and the courier label; problem noticed: the two addresses don't match; confidence 80%. Every model answer from the study is available on request, as set out under Model answers on request.
4. The script scored the decision. It checked the decision against an answer key and costed any mistake in money, or in points where a process had no natural money figure.

All the data in the case packs was synthetic. The firms, customers, payments, emails and rulebooks were invented for the study, and no real or confidential data was used.

Model choice

The study compared one frontier model with one low-cost open-weight model.

Claude Opus 5.5 from Anthropic was the frontier model. It was the most accurate model in S2 by a wide margin, so it set the ceiling. At OpenRouter's list prices in October 2026, it costs $4 per million input tokens and $20 per million output tokens.

Mistral Medium 3.5 from Mistral AI was the low-cost model. Its weights are open, so a firm can host it in the EU on its own servers, and it costs well under half as much as Opus: $1.50 per million input tokens and $7.50 per million output tokens at OpenRouter's list prices in October 2026.

Medium 3.5 is Mistral's mid-tier model. It was chosen over Mistral Large 4, Mistral's larger and newer model. Large 4 was still a public preview, with its open weights yet to be released, and in S2 it gave no usable answer in any of its 13 runs and ignored the reasoning and output limits it was sent. Hence this study doesn't show how Mistral's strongest model, or other leading open-weight models, would do on these decisions. Grok 4.7 was also considered as a cheaper frontier model and ruled out, because its S2 results swung widely between runs and, interestingly, at the time it didn't offer a setting with reasoning off.

Reasoning settings

Most current AI models can be set to reason before they answer. With reasoning on, the model works through a chain of thought, setting out its thinking step by step before it reaches a decision.

The idea isn't new. In 2022, researchers at Google showed that asking a model to work through a problem step by step made it more accurate on maths and logic questions. From late 2024, starting with OpenAI's o1, model makers began training models to do this on their own before answering, and most major model makers now offer it as a setting. The reason is simple: a model that spends more effort on a hard question gets more of them right, especially questions that take several steps.

More reasoning costs more. Models are billed per token, the small chunks of text they read (input) and write (output), and the thinking counts as output, which is the more expensive of the two. It also takes longer, which matters for decisions that have to be made in seconds.

More reasoning doesn't always mean better answers. A model can underthink, answering a hard case too quickly and missing the detail that changes the answer. It can also overthink, talking itself out of a right answer or seeing a problem in a case that doesn't have one. It can even run out of its thinking budget and give no answer at all.

Testing two settings per model answers the second key question, whether more reasoning closes the gap.

Each model ran at its lowest available setting and at a high one:

• Claude Opus 5.5 ran at low and high effort. Low effort is the cheapest setting OpenRouter offers for Opus.
• Mistral Medium 3.5 ran with reasoning off, its lowest setting, and with reasoning on at its high level. Both settings were checked to be working before any paid runs. The rest of this study calls them reasoning off and reasoning on.

Number of answers

Each model ran every one of the 22 cases per process set out under Scope twice at each of its two settings. In each process, that gave:

• 44 answers per setting (22 cases, each run twice),
• 88 answers per model (two settings),
• 176 answers in all (two models).

In full: 22 cases × 2 runs × 2 settings × 2 models × 9 processes = 1,584 answers, less 8 for the excluded payment performance case, gives 1,576.

The results compare four combinations of model and setting: Opus at low effort, Opus at high effort, Mistral with reasoning off and Mistral with reasoning on. Each combination gave 394 answers across the nine processes. Hence the results are given out of 44 answers per combination in each process, or 42 for payment performance, which scored 21 cases for the reason given under Fixes and re-runs.

Key findings

Section 1 set out three key questions. There was a big gap between the models on judgement, as the study expected. More reasoning didn't close that gap, which went against the working assumption. Once its mistakes were costed in money, the low-cost model didn't save money on these decisions. The sections below go through each finding, based on 394 answers from each of the four combinations of model and setting across the nine processes.

There is a big gap on judgement

Claude Opus 5.5 at low effort got 385 of its 394 answers right. Most of its misses were grey cases where it read the rules differently from the expert who set the answers. Mistral Medium 3.5 with reasoning off got 251 right. Mistral got fewer right than Opus in all nine processes, as Figure 1 shows. The gap was smallest in scheme changes and widest in treasury prefunding, deal pricing and accounts payable. Mistral's misses included clearing sanctions alerts that the rules say must go to an analyst, paying an invoice from a lookalike sender, approving sellers that the policy says a person must check, and getting every treasury prefunding amount wrong, though most of those by small amounts.

The gap grew with difficulty

Each process had five easy, five medium and twelve hard cases, so each combination of model and setting gave 90 answers on easy cases, 90 on medium cases and 214 on hard ones. Right answers by difficulty were:

• Opus at low effort: 90 easy, 90 medium and 205 hard. At high effort, 90, 90 and 207.
• Mistral with reasoning on: 89 easy, 84 medium and 171 hard.
• Mistral with reasoning off: 70 easy, 61 medium and 120 hard.

Opus got every easy and medium case right at both settings, so all of its misses were on hard cases. Mistral with reasoning on was close to Opus on easy cases and fell behind on medium and hard ones. Without reasoning, Mistral also got 20 of the 90 easy answers wrong, so its problems weren't limited to the hard cases. Its confidence on wrong answers stayed between 96 and 100% at every level of difficulty.

Mistral didn't know when it was wrong

Each model gave a confidence score with every answer. Mistral's wrong answers averaged 98% confidence with reasoning off and 97% with it on, about the same as its right answers at 99%, as Figure 2 shows. Hence a firm couldn't use Mistral's own confidence to catch its mistakes, for example by sending cases it was unsure about to a better model or a person. Routing rules for a model like Mistral have to use features of the case instead, such as the amount, conflicting sources, missing data and the stakes. Opus was less sure when it was wrong, at about 70% against 86% on its right answers, which is worth testing as a routing signal in the next round.

More reasoning didn't close the gap

Opus at high effort got 387 of 394 right, two more than at low effort. It gained one answer in two processes and none in the others, and cost 20 to 65% more per case. Mistral with reasoning on got 344 right, up from 251, but still lost clearly more money than Opus at low effort in six of the nine processes. Ten times it ran out of thinking budget and gave no answer at all. Three of those were in treasury prefunding, where a missing answer counts as sending nothing, which would have left $1.2 million of payouts unfunded. Its reasoning also used up most of its price advantage: it cost more per case than Opus at low effort in eight of the nine processes. Hence paying for more reasoning makes sense only on steps where a test shows it changes the answer.

The cheapest model cost the most

The study's economic evaluation counted two kinds of cost per case.

• Token cost. What the model provider charges to read the case pack and write the answer, including any reasoning, at list prices. Opus at low effort cost $0.012 to $0.033 a case and at high effort $0.016 to $0.051. Mistral cost $0.004 to $0.009 with reasoning off and $0.013 to $0.084 with it on.
• Error cost. The money a wrong decision would lose the firm, at the rates set in each firm's manual: for example failed payouts and idle cash in treasury prefunding, four weeks of lost sales in payment performance, and margin over a deal's first year in deal pricing. Per pass of 22 cases, Opus at low effort lost under $1 in five processes and $16 to $533 in the other four. Mistral with reasoning off lost $62,000 in payment performance, $16,300 in payment exceptions, $12,300 in accounts payable, $5,700 in seller onboarding and $5,000 in deal pricing. Treasury prefunding is counted differently: the manual values a payout that can't be paid on the day at its full amount, so its figure measures payouts left unfunded, $1.7 million across the 22 cases for Mistral with reasoning off, rather than money gone for good.

Mistral with reasoning off saved between about 0.8 and 2.4 US cents a case in token cost against Opus at low effort. Once error cost was added, it cost more in full in six of the seven processes scored in money. Figure 3 shows token cost and error cost added together for the six processes where error cost is money a firm would actually lose. One wrong treasury day, sanctions clearance or supplier payment wipes out that saving across millions of cases. Chargebacks was the exception, where Mistral's misses happened to be on low-value disputes. Hence, used out of the box, Mistral Medium 3.5 was a false economy on these judgement steps.

The gap depends on the process

The gap between the models varied widely by process. It was small in scheme changes, where Mistral got 42 of 44 right at both settings, and very large in treasury prefunding and accounts payable. Hence a firm should test the specific decision it plans to hand over, on its own cases, before choosing a model. Model and Vendor Check, one of ScalePoint Research's executive briefings, runs that test. Appendix A gives the detail for each of the nine.

Table 1 shows right answers out of 44 for each combination of model and setting. Payment performance scored 21 cases, so 42 answers, for the reason given under Fixes and re-runs.

Table 1. Right answers by process, model and setting

Discussion and next steps

Discussion

• More reasoning had mixed results. In this study, turning reasoning on fixed 112 of Mistral's answers and broke 19 others: 15 it got right with reasoning off and wrong with it on, and four where it ran out of thinking budget. For Opus, high effort fixed four answers and broke two. Hence the effect of reasoning depended on the model. It helped Mistral a lot and Opus hardly at all, and it broke some answers for both. A firm should test reasoning settings on its own decisions before switching them on everywhere, since that's hard to know without a test.
• The cases didn't stretch Opus. Making them hard took real effort. After the first runs, the hard cases in each process were rebuilt as judgement cases, with missing evidence, sources that conflicted, rules that didn't quite fit, look-alike cases and high stakes, and the expert answered a sample of them before seeing any model's answers. At low effort it already got 385 of 394 right, and most of the nine it missed were grey cases where it read the rules differently from the expert. Hence high effort had little left to fix, and this study can't say where Opus starts making costly mistakes.
• A low-cost model might handle the easy cases. Mistral with reasoning on got 89 of 90 easy answers right, close to Opus, as section 3.2 shows. Hence a low-cost model with reasoning on might handle the easiest cases in a process, as long as a firm can tell those cases apart before the model sees them. The next piece, on routing, looks at that.
• The money figures assume nobody checked. The error cost assumes every wrong decision went through unchecked. The study didn't assume that AI runs any of these processes end to end. A person reviewing the model's decisions would catch some of those mistakes, at the cost of their time, and in most processes the model could also hand a case to a person by escalating or referring it. The study didn't model the cost of that checking.

Next steps

• Harder cases. Rebuilding the cases for judgement didn't stretch Opus, so the next round will try work closer to the real job. The working view is that three things are most likely to stretch it: long case packs with many documents to reason across, where the models pulled apart in S2; tasks that take several steps with tools, where an early mistake carries through to the end; and real records, which are messier than invented ones.
• Other models. S3 tested one frontier model and one low-cost model. Adding GPT, Gemini and other open-weight models will show whether the results hold beyond these two.
• Fine-tuning. Mistral was tested as released. A later study will test whether fine-tuning a low-cost model on a firm's past decisions narrows the gap enough to cover the cost of doing it.
• Escalation quality. S3 scored whether a model escalated when it should have. A separate article will use the run logs to score whether the note it handed over was good enough for a person to decide quickly.
• Routing. A firm doesn't have to send every case to the same model. A follow-up article will build a fuller economic model on these results. It will work out what different splits between the models would cost at a firm's own volumes, for example sending high-value cases to Opus and the rest to Mistral, including the cost of a person checking the cases that get escalated.

Notes on this study

The Method page sets out how every ScalePoint Research study is built and scored. These notes cover only what's specific to this study.

• About the pilots. The study was designed to give an initial sense of how big the gap is and where in payments operations it matters most. To do that, it ran a lighter pilot in each process, with the same method and the same pair of models in all nine processes, so that a pattern across the nine could show. Over time, more models may be tested on the same tasks and added to the study, and some processes will get a deep-dive article of their own, linked from their section in Appendix A once it's published.
• Model answers on request. Every answer from every run is kept in the study's run logs, with the model's reason, the documents it cited, the problems it flagged and its confidence. The full set for all nine processes is available on request from martin@scalepointpartners.com.
• Answer keys. Some answers were refined in discussion: the procedure rules were aligned to the expert's calls in payment performance, scheme changes and accounts payable, and two sanctions answers were revised after hearing the draft key's reasoning. In one chargeback case two answers counted as right: accepting and recalling the refund, which the expert prefers, and escalating, which costs $34 more.
• Fixes and re-runs. Three cases had flaws in their case packs and were fixed after the main run. Each fixed case was then re-run on its own at every setting: one chargeback case, where a portal export wrongly showed a notice as still open; one scheme changes case, where the bank's profile was wrong; and one payment performance case, where the rule text gave an order size that didn't match the data. A second payment performance case was excluded. It was built around a drop in approvals that turned out to be too small to stand out in the data. After the drop was enlarged and the case re-run, every model at every setting raised it with the acquirer, which is a fair reading of the evidence, so the case couldn't tell a right answer from a wrong one. Hence payment performance scored 21 cases.
• Money. Losses were costed in pounds and are shown here in US dollars at $1.34 to the pound, as in S1. Payment performance includes a shop that trades in euros, converted to pounds at 0.85.
• Reading small gaps. Each combination of model and setting gave 44 answers per process, which is enough to show a consistent pattern across the nine processes. A gap of one or two answers between two settings in one process shouldn't be read as one being better than the other.

Appendix A: Results by process