r/mlscaling • u/PotentialFirm7721 • 46m ago
ZipGate v2.2 Spatial Reasoning Benchmark Pilot Results - Can your model pass the 5-task challenge?
Title: IKIN ZipGate v2.2 Pilot Benchmark: Chain-of-Thought vs. Zero-Shot Performance on Spatial Reasoning
Body:
I just wrapped up the pilot evaluation of ZipGate v2.2, a new benchmark designed specifically to test complex spatial reasoning on 2D triangular lattices.
We tested 5 frontier models across 4 tiers of difficulty:
Bullet Tier 1: Axial Kinematics
Bullet Tier 2: Internal State Tracking
Bullet Tier 3: Gate Arbitration (Rule following)
Bullet Tier 4: Long-Horizon Rollout
Key Takeaway: Explicit Chain-of-Thought (CoT) or extended thinking modes are mandatory for high performance here. While CoT models (Claude 3.5, o1, Gemini 1.5 Pro) cleared the 5-task pilot set with 100% accuracy, standard zero-shot models struggled significantly with Tier 3 (rule tie-breakers) and Tier 4 (multi-step rollout drift).
Full scorecard attached. The 5-task pilot dataset is available on GitHub if you want to test your own models.
🔗 GitHub: https://github.com/jadeikin-ZIPGATE/ikin-zipgate-benchmark