Summary
Microsoft released ThinkingBox, an open-source, MCP-compatible sandbox and paired benchmark (ThinkingBox-Bench) that evaluates AI agents by the database state they actually change rather than what they claim to do. Detailed in an August 19, 2026 Microsoft engineering post and on GitHub, it tested 12 models across 507 policy-conditioned business workflows, finding the best model scored 65% pass@1 but only 25% when required to succeed across 20 repeated runs.
What changed
Microsoft published ThinkingBox and ThinkingBox-Bench on GitHub (github.com/microsoft/thinkingbox): isolated MCP tool sessions with full execution traces and outcome evaluation over terminal backend state, plus 507 tasks across retail, hospitality, auto insurance, neobank IT, and consulting HR/IT support scenarios.
Why it matters
It reframes agent evaluation from single-shot task success to reliability under repetition, exposing that strong tool-use scores do not equal dependable work completion. As enterprises move agents into stateful business systems, outcome-based, repeat-run benchmarks like this set a harder bar for 'production-ready' claims.
Evidence excerpt
Instead of grading agents on what they say they did, ThinkingBox checks what they actually changed in a database; the strongest model achieves 65.36% pass@1 but only 25.25% pass^20 across 507 policy-conditioned workflows in five business domains.