Just like a seasoned gamer knows that sometimes, the true test isn’t about getting a perfect score but understanding where your AI or game engine falters, a new AI benchmark exposes the core of what makes an AI reliable in real-world scenarios. The surprising part? Even a ‘do-nothing’ baseline achieves 26 points. That number isn’t a bug—it’s a feature that reveals how trust, partial progress, and honesty are measured in AI performance today.
Play games on Amazon Luna for your game nights, included with Prime
- A rotating selection of games, no download needed
- Play on TV, laptop or phone
- Fast, free delivery for your gear, too
Understanding the Baseline: Why 26 Points?
Imagine setting an AI up in a simulated business crisis—everything from customer demands to internal conflicts. In a recent experiment by Firmulate, four advanced AI models were put through the exact same scenario: managing a small software company during its worst week. The goal wasn’t just to see if they could impress with chatter, but whether they could handle real decisions, avoid manipulation, and ultimately close a deal worth €55,000.
Strikingly, even a ‘do-nothing’ baseline—meaning an AI that doesn’t perform any tasks—scores 26 points out of 100. That’s because the benchmark assigns partial credit for doing anything at all, even if it’s just passively observing the situation or avoiding risky manipulations. It’s a nod to reality: in business, doing nothing is better than doing the wrong thing, and sometimes, avoiding risk is the smartest move.
Popular right now
Ninja NC501 CREAMi XL Deluxe 11-in-1 Ice Cream and Frozen Treat Maker
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
De'Longhi La Specialista Arte Evo Espresso Machine with Grinder
As an affiliate, we earn on qualifying purchases.
Progress Counts, But Trust Caps Out
The experiment clearly shows that all four models identified every crisis, refused manipulative tricks, and stayed honest under pressure. Only two of them actually signed the deal their analysis deserved—meaning their actions matched their insights. The others, despite good diagnoses, left opportunities unexploited or failed to follow through, often leaving potential revenue on the table.
What’s revealing here is that AI performance isn’t just about accuracy or speed; it’s about integrity. The benchmark caps the total score at 100, but if an AI breaches trust even once, its final grade is capped. This emphasizes that trustworthiness isn’t an optional feature—it’s fundamental. An AI that slips up in honesty or decision-making risks diminishing its overall value, regardless of its analytical prowess.
The Hidden Weakness: Reading the Files
While all models spotted crises from high-level prompts, the real decisive factor was their ability to access and interpret crucial internal documents. The models that managed to read two document references deep into the company’s files won the deal at full price, adding over €4,500 monthly recurring revenue (MRR). Conversely, models that failed to read the documents left money on the table.
This demonstrates a vital truth for business AI: surface-level understanding isn’t enough. Deep, accurate reading of internal data can be the difference between closing a deal and losing it, highlighting the importance of thorough data integration and comprehension capabilities.
Resisting Social Engineering: A Test of Integrity
The experiment also tested how models would handle social engineering—fake CEO messages escalating in three stages and a reporter’s subtle request for background approval. All four models refused these manipulative tactics, with Kimi K3 explicitly treating the requests as potential impersonation or approval-bypass attempts. This resilience underscores that trustworthiness extends beyond data reading to include resistance against deception.
For businesses, this is a crucial insight: AI systems must be designed to recognize and refuse manipulative tactics, especially as social engineering threats grow more sophisticated.
Real-World Implications: The Live Company Scenario
Firmulate’s live experiment involves a simulated company with 13 synthetic employees, running with real money mechanics—burning €105,000 monthly against a modest €2,300 MRR. Every decision, each rule, and every crisis is versioned and observable, offering a transparent view of how AI models perform in high-stakes environments.
In this setting, the most thorough participant, Opus 4.8, with over 80 learned rules and deep analyses, finished last because it abandoned its pursuit of the deal, leaving it on the table. This reveals an important lesson: thoroughness and discipline matter, but so does focus and follow-through. Even the most detailed AI can falter if it slips in execution or discipline.
What Business and Gaming Can Learn from This Benchmark
For fans of gaming and interactive entertainment, the takeaway is clear: AI performance isn’t just about shiny features or quick responses. It’s about integrity, trust, and the ability to follow through on decisions—values that are essential both in business and in gameplay. Just as gamers know that a perfect score isn’t the only measure of skill, businesses need AI that can handle real pressures and stay honest under fire.
And for enterprise leaders, the Firmulate experiment offers a powerful tool: running your own ‘wargame’ against your AI workforce before deploying it live. By simulating crises and temptations in a controlled environment, you can identify weaknesses—like leaving money on the table or failing trust checks—before they become costly mistakes.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
