Breakdown on Codex

The Four Layers of Memory

This article explores OpenAI Codex's memory mechanism and how it preserves long-term memory.

There are three main layers of memory, along with an occasional fourth layer generated by the AI as an ad-hoc note.

• The First Layer is actually a model-facing index made of two files: memory_summary.md and MEMORY.md. memory_summary.md is the compressed overview of the user, preferences, stable workflows, and important local paths. MEMORY.md is the larger searchable registry that groups previous tasks by project, topic, keywords, reusable knowledge, and failure notes, and links to the more detailed second-layer rollout summaries.
• The Second Layer is a detailed documentation of how the task was completed, including the specifications, issues faced, and solutions. The markdown part contains almost all information except tool call results. There is also a JSONL document with complete tool call results and user instructions, which the model can reference if needed.
• The Third Layer is a raw JSON process document of the session. It contains the JSONL document and a SQLite database with timestamps, exact tokens, raw results, environments, and document paths. The model reads the SQLite database via the command line if the JSONL document is insufficient.
• The Fourth Layer is called ad hoc notes, which are customized notes generated when a user emphasizes a point or tells the AI to remember something. Codex creates a small markdown file using the time and title as its file name, containing the context and scope of the instructions. When the user mentions a task similar to one in the main memory file, the AI reads both the second-layer note and the ad-hoc note.

It is important to note that there is also a file called raw_memories.md which has the details of a task in it, but also keywords which are very easy for the models to search with rg, a method very convenient in the terminal.

One obvious benefit of this mechanism is that your preferences are never forgotten. The AI avoids repeating mistakes because it tracks memory files and detailed descriptions of how to solve past problems.

However, the downsides of this mechanism are outweigh the benefits. The most damaging downside is overfitting. The AI overfits to your requests and generalizes them too broadly to other tasks. In some cases, the AI restricts its capabilities to tailor to needs you do not actually have.

Another serious problem is unconscious recording. The AI sometimes interprets actions as a preference and saves it without your knowledge. Furthermore, when the AI acts lazy to avoid heavy work, it records that habit into the file, leading to more loafing on the job.