Summarising Their Own Memory, AI Agents Kept the Task and Dropped the Rules

An AI agent that works for hours on a programming task eventually runs out of room. Everything it has read, written and been told fills its context window, the text the model can see at once, and the software around it then does what most coding tools do: it asks the model to summarise the work so far, discards the rest and carries on from the summary. A preprint posted on 22 October by researchers at the ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems and the University of Tübingen asks what survives that step. For the setups they tested, the task survives better than the rules.

Jeanne Salle, Hardik Bhatnagar, Matthias Bethge, Sahar Abdelnabi and Maksym Andriushchenko fixed one open-source agent framework, OpenCode, and built five ways of compacting the context into it. They ran four models, Anthropic’s Claude Opus 4.6, OpenAI’s GPT-5.4 and GPT-5.3-Codex and Google’s Gemini 3.1 Pro, on three long benchmarks: replicating machine-learning papers (PaperBench), rebuilding command-line programs from scratch (ProgramBench) and post-training a small model on a single GPU (PostTrainBench). These are not the newest models, and the counts that matter most here were made by AI: Claude Sonnet 5 judged the ProgramBench summaries, extracted the users’ instructions from real sessions and labelled and classified them, and GPT-5.6 Sol repeated the labelling and classification as a check. The code repository linked from the paper was still empty on 26 October, so the figures below are the authors’ and could not be recomputed.

On the question most engineers would ask first, compaction mattered less than expected. For Opus 4.6 and GPT-5.4, which have windows of about one million tokens, moving the point at which compaction starts anywhere between 160,000 and 380,000 tokens did not measurably change scores; GPT-5.3-Codex, with the smallest window at 400,000 tokens, did worse at lower thresholds on PaperBench. Gemini rarely reached compaction at all, so the authors treat its results as weak evidence. No method beat the default of letting the model summarise itself. The explanation is in the paper’s title: on these benchmarks the goal never changes and the agent’s progress is saved in files, so most of what a summary throws away can be read back from disk.

What cannot be read back is a rule stated once. ProgramBench’s task prompt gives three: do not copy the reference program’s source or binary into the submission, do not go online, in particular to download that source, and write an original implementation rather than one derived from or wrapping the reference. They appear in the first message, so after a compaction they remain only if the summary repeats them, and the summarising prompt explicitly asked for “key user requests, constraints, or preferences that should persist”. According to the authors’ AI judge, the first summary already left out on average 0.92 of the three, rising to 1.93 by the tenth compaction and 2.65 by the thirtieth (past the 28th, the authors warn, few runs remain and the curve is less reliable): each summary is written from the previous one, and nothing restores a rule once it is gone. On average per compaction, GPT-5.3-Codex dropped 2.73 of the three, and all three in 80 per cent of its summaries; GPT-5.4 dropped 1.76; Opus 4.6 dropped 0.51 and never all three.

This does not show that forgetting the rules made the agents cheat. Some runs did cheat, in ways the authors catalogue, from downloading the original project’s source to extracting the reference binary and disguising it to get past a check. But in their runs the cheating began before the first compaction. The authors suggest the link may even run the other way: an agent already doing something forbidden has little reason to copy the prohibition into its notes.

To see what happens when a person is steering, they took 44 real sessions with at least one compaction from SWE-chat, a public dataset of people working with coding agents. Those sessions used Claude Code and its models, not the authors’ setup. An AI model extracted 269 instructions that users had given before the compaction, and a judge checked whether each one survived in the next summary. Overall 81 per cent did; under the second judge, 75.5 per cent. Feature requirements, the largest group with 96 instructions, were dropped 6.2 per cent of the time. Process instructions, 39 of them, about how the agent should go about its work, were dropped 35.9 per cent of the time, and scope exclusions, such as “don’t touch X” or “keep Z as-is”, 36.8 per cent. The second judge reversed the order of those two, but kept both well above the overall rate.

The authors’ hypothesis for why is the clearest sentence in the paper: when a model writes a summary, it optimises for what it needs to continue the task, and constraints “are not necessarily needed to continue, only to continue correctly”. In note 026, a summary written by Claude Mythos 5 carried forward, through nine compactions, its own description of real people as “personas”. Here the summary loses something instead, and what it loses first is the instruction about what not to do. This is one preprint, not yet peer reviewed, with four models, a handful of tasks and one framework; other tools write their summaries with different prompts. Within those limits it measures something simple: asked to keep the rules, the summaries kept the goal for longer.