TL;DR
Firmulate’s July 2026 benchmark found that five AI models consistently identified business crises and rejected manipulation attempts, according to the company. Only two completed a €55,000 contract, exposing a gap between producing correct analysis and converting it into authorized action.
Only two of five AI models completed a €55,000 customer contract after all five correctly identified the opportunity and resisted simulated manipulation attempts, according to July 2026 results published by Firmulate. The live management test points to a gap between producing a correct answer and turning that answer into completed, authorized business work.
Firmulate placed the models in control of the same simulated software company during a week of customer problems, financial pressure and social-engineering attempts. The company reported that every model identified every crisis and rejected each manipulation attempt, but only two followed the commercial task through to a signed €55,000 deal, adding the equivalent of €4,583 in monthly recurring revenue.
The final Crucible League table ranked gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because the system awarded points for partial progress. Firmulate added that Kimi K3 used its API’s default effort setting, while the other models ran at xhigh, limiting direct comparisons.
The information needed to close the deal was not included in the initial customer event. Firmulate said a competitor weakness was located two document references deep in the company files. Models had to recognize that the first record was insufficient, continue investigating, apply the evidence to the sales pitch and complete the deal through the approved operating channel.
Correct Answers Did Not Close
The results suggest that reasoning quality and task completion may need separate evaluation when businesses select AI agents. A model can diagnose a problem, prepare an effective response and obey safety rules while still failing to produce the commercially required outcome. For companies using agents in sales, service or operations, that difference can affect revenue, customer trust and accountability.
The experiment also found that detailed analysis did not always produce stronger execution. Firmulate described Opus 4.8 as the most thorough participant, with the deepest analyses and more than 80 learned rules, yet it placed last among the tested models. The model also attempted to write into a locked department instead of escalating through an authorized route, according to the benchmark record.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A Company Built for Auditing
Firmulate says its simulated company contains 13 synthetic employees, more than 680 self-learned playbook rules and versioned daily decisions. It operates with real-money-style mechanics, including monthly spending of €105,000 against €2,300 in recurring revenue and a public cash countdown designed to make delays measurable.
Unlike a standalone chat test, the management scenario links investigation, communication, security and execution across multiple decisions. Firmulate applies a score cap after any trust breach, reflecting its stated rule that safe conduct remains a hard constraint even when a model performs well elsewhere. The operator also provides a quiz drawn from 242 recorded management decisions.
“Same diagnosis, same pitch — no signature.”
— Firmulate’s summary of the benchmark
Independent Validation Is Still Missing
The results were reported by Firmulate, which designed and operates the experiment. The supplied material does not describe an independent audit of the scoring, model configurations or underlying decision logs. It also does not identify which two models signed the contract, preventing a direct link between the final rankings and successful deal completion.
It is also unclear how well performance in this synthetic company predicts behavior inside a real business with human approvals, incomplete records and changing policies. The different effort setting used for Kimi K3 adds another limitation, while future model updates could make the July 2026 ranking short-lived.
Live Runs Will Test Consistency
Firmulate plans to keep the company running publicly, allowing readers to inspect later decisions and changes in model behavior. Repeated runs could show whether closing strength is consistent or whether the two completed contracts reflected conditions specific to this test.
For business buyers, the next step is likely to be testing agents against company-specific workflows before granting operational authority. Firmulate says organizations can use a read-only export of their records so models can be observed without writing back to production systems. Such trials would need to measure authorized completion, safety and recovery from blocked actions, not just the quality of generated answers.
Key Questions
What did Firmulate’s AI management test find?
Firmulate reported that all five models detected every crisis and rejected the simulated manipulation attempts. Only two completed the €55,000 contract produced by their analysis.
Which model ranked first?
gpt-5.6-sol ranked first with 95 points, ahead of Kimi K3 at 93. The published source does not specify which two models completed the contract.
Did any model fall for the fake messages?
No, according to Firmulate. All five models refused staged fake CEO messages and a reporter’s attempt to obtain an off-record answer, making execution discipline the larger differentiator in this run.
Can the results be applied directly to real companies?
Not yet. The test used a controlled synthetic business, and the source material provides no independent audit or evidence that the scores predict performance in live organizations. The findings are best treated as a benchmark result rather than proof of real-world reliability.
Source: Thorsten Meyer AI
