Microsoft and Hugging Face use ThinkingBox database checks as RL rewards to improve Qwen3.8-27B agent reliability

Microsoft and Hugging Face released ThinkingBox-Training on GitHub and showed how ThinkingBox’s executable final-database-state checks can serve as RL rewards for agent post-training. After 50 GRPO steps on training tasks outside the 507-task benchmark, Qwen3.8-27B improved pass@1 from 51.70% to 60.78% and observed 20/20 from 38 to 107 tasks, while pass@20 fell from 453 to 446 tasks.

For teams building database-writing agents, the reported result supports turning production-like executable checks into reward functions rather than relying only on transcript grading. The trade-off is narrower coverage: repeatable success rose, but at-least-once success declined slightly, and the evidence comes from one 27B model and a fixed offline-selected task pool, so generalization to other models or task-selection methods remains unanswered.