API settings nearly tripled GPT-5.6 Sol's score
GPT-5.6 Sol encountered difficulties with ARC-AGI-3, a benchmark in which models must learn unfamiliar two-dimensional games without instructions. An investigation revealed that the problem was related not only to the model’s capabilities but also to the design of the testing environment: after each move, it discarded the model’s reasoning, and as the context filled up, it deleted the action history.
OpenAI created a new testing harness based on the Responses API and enabled reasoning persistence along with context compaction. This allowed the model to use its accumulated experience instead of repeatedly analyzing everything from scratch. Its score on the public ARC-AGI-3 set increased by 188%, while output token usage fell sixfold.
The experiment demonstrates that benchmark performance depends not only on the model itself, but also on API settings, the structure of the testing harness, and how prompts are constructed. For long-running AI agents, preserving reasoning and maintaining a compact context are especially important.
Why it matters
- —AI benchmark results can depend significantly on the testing environment and API settings, not just on the model's capabilities.
- —Preserving reasoning helps long-running agents use accumulated experience.
- —Context compaction both improved test performance and reduced token usage.
Key facts
- ARC-AGI-3 tests models' ability to master unfamiliar two-dimensional games without instructions.
- The standard environment deleted reasoning after each turn and gradually cleared the action history.
- OpenAI enabled reasoning preservation and context compaction through the Responses API.
- GPT-5.6 Sol's score on the public set increased by 188%.
- The model used six times fewer output tokens.
The full text is in the original source. Here we provide a brief summary and key facts.