Anyone who has built a tiny house knows the type — maybe you are the type. The builder who drafts seventeen versions of the loft ladder, who reads the local code front to back twice, whose material list has annotations in three colors. That builder produces the most beautiful plans in the county. And sometimes the house never gets finished, because while they were perfecting the joinery, the season turned and the inspection window closed.
Get compact home essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
It turns out AI models can fail the exact same way. And there is now a live, watchable experiment that proves it.
A Stress Test for AI Managers
The experiment is called Firmulate, and its premise is disarmingly simple: stop testing how well AI chats, and start testing how well it manages. Four frontier AI models were each handed the same job — running the same small software company through its worst possible week. Same customers, same crises, same temptations to cut corners. Only the model changed. Every decision was versioned and auditable, so nothing is judged on vibes.
Think of it as a wind-load test for AI judgment, the way you’d rack a wall frame before trusting it with your roof.
Products Worth Considering
Roborock Q7 L5 Robot Vacuum and Mop Combo
TCL 55-Inch T7 Series 4K QLED Smart Google TV (55T7)
LEVOIT Vital 200S-P Smart Air Purifier for Large Rooms, White
As an affiliate, we earn on qualifying purchases.
The Final League Table
By the end of the Crucible League run in July 2026, the standings were: gpt-5.6-sol in first with 95 points, Kimi K3 second at 93, Sonnet 5 third at 88, a second Sonnet configuration at 77 — and Opus 4.8 in last place at 73. For context, doing nothing at all would have scored 26, because partial progress counts. One caveat worth noting in fairness: K3 ran without an effort parameter while the others ran at maximum effort, which makes its second-place finish even more striking.
Everyone Saw the Storm. Only Some Closed the Door.
Here’s the finding that matters. All five models that ran — the league results and profile data cover five participants — spotted every crisis and refused every manipulation attempt. That part is genuinely reassuring. When a fake CEO message escalated over three stages, and when a reporter tried the classic “just one yes/no, on background” trick, every single model declined. Kimi K3 even put its reasoning on record: treat the request as a suspected approval-bypass or possible impersonation.
But only two models actually finished the job. There was a €55,000 deal on the table — a deal their own analysis had earned. Most of them diagnosed it, pitched it, and then… never signed. As the experiment’s summary puts it: same diagnosis, same pitch, no signature.
If that’s not a tiny-house story, I don’t know what is. It’s the builder who correctly calculates the load rating for the cantilevered reading nook, tells everyone about the cantilevered reading nook, and never bolts the thing down.
The Buried Fact
There’s a twist, too. The decisive weakness in the competitor — the fact that justified closing at full price, worth +€4,583 in monthly recurring revenue — wasn’t in the customer meeting at all. It sat two document references deep in the company’s own files. The models that actually read their own paperwork won the deal. The ones that skimmed didn’t.
Again: the parallel to our world is almost uncomfortable. The best tiny-house builds come from people who read the county’s actual code documents, not the forum summary of the code documents.
The Most Thorough Builder Came Last
And now the character study at the heart of this. Opus 4.8 was, by volume, the most impressive participant in the field. It learned 80 new playbook rules during its run — the most of any model — and produced the deepest analyses of anyone. On paper, it was the valedictorian.
It finished last anyway. The €55,000 close was left on the table, and discipline slipped at a crucial moment: when it hit a locked department, it kept attempting writes instead of escalating the problem to someone who could unlock it. Diligence, in other words, wasn’t the issue. Converting diligence into impact was.
To be fair — and the experiment is scrupulous about this — the same weakness showed up in all four models, just weaker. Nobody in the field was immune to mistaking thoroughness for completion.
Why It’s Watchable
What makes Firmulate more than a one-off paper is that it’s ongoing. The live company runs 13 synthetic employees on real money mechanics — burning €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown and 680+ self-learned playbook rules, every workday versioned. The site rebuilds itself twice a day as new runs finish. There’s also a genuinely fun artifact: a “guess the model” quiz built from 242 real, unedited management decisions. Try it — it’s harder than you’d think, and it teaches you to recognize management fingerprints the way you’d recognize a builder’s.
For enterprises, there’s a pilot program: run the same wargame against a read-only export of your own business, with nothing ever writing back to real systems.
The tiny-house movement was built on a rejection of more-is-better — smaller footprint, smarter choices, finished projects over sprawling ones. The Firmulate results suggest AI needs the same philosophy. The winning models weren’t the ones that analyzed the most or accumulated the most rules; they were the ones that read what mattered, escalated what was blocked, and signed what they’d earned.
Whether you’re sizing a water heater for 200 square feet or choosing an AI agent to touch your CRM, the question is the same: not “how much can it do,” but “will it finish what it starts?” Opus 4.8, with its 80 rules and its empty signing hand, is the reminder none of us needed but all of us deserved.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
