This context limit naturally limits the dimensions of a codebase a LLM can process at one time, and in case you feed the AI model a number of huge code files (which must be re-evaluated by the LLM each time you send one other response), it might burn up token or usage limits pretty quickly.
Tricks of the trade
To get around these limits, the creators of coding agents use several tricks. For instance, AI models are fine-tuned to write down code to outsource activities to other software tools. For instance, they may write Python scripts to extract data from images or files quite than feeding the entire file through an LLM, which saves tokens and avoids inaccurate results.
Anthropic’s documentation notes that Claude Code also uses this approach to perform complex data evaluation over large databases, writing targeted queries and using Bash commands like “head” and “tail” to investigate large volumes of information without ever loading the total data objects into context.
(In a way, these AI agents are guided but semi-autonomous tool-using programs which might be a significant extension of an idea we first saw in early 2023.)
One other major breakthrough in agents got here from dynamic context management. Agents can do that in a couple of ways in which are usually not fully disclosed in proprietary coding models, but we do know crucial technique they use: context compression.
The command-line version of OpenAI Codex running in a macOS terminal window.
Credit:
Benj Edwards
When a coding LLM nears its context limit, this system compresses the context history by summarizing it, losing details in the method but shortening the history to key details. Anthropic’s documentation describes this “compaction” as distilling context contents in a high-fidelity manner, preserving key details like architectural decisions and unresolved bugs while discarding redundant tool outputs.
This implies the AI coding agents periodically “forget” a big portion of what they’re doing each time this compression happens, but unlike older LLM-based systems, they aren’t completely clueless about what has transpired and might rapidly re-orient themselves by reading existing code, written notes left in files, change logs, and so forth.
