How to Manage Context in AI Agents: Three-Layer Compression Strategy

Mar 18, 2026

I hit a wall. My AI agent crashed mid-task with a dreaded error:

Error: context_length_exceeded - This conversation has exceeded the maximum context length of 200000 tokens. Please reduce the length of your messages or conversation history.

I was debugging a complex codebase, had read dozens of files, and suddenly—boom. Context overflow. All that exploration, all those tool results, gone.

The real problem? A single file read can cost 4000+ tokens. After exploring a codebase for 30 minutes, context fills up fast. Without a strategy to compress, agents hit token limits and fail at the worst possible moment.

Here’s what I learned about managing context in AI agents.

The Token Cost Problem

Every interaction burns tokens. Let me show you what happens when you’re not paying attention:

Action                          Tokens
-----------------------------------------
Read a 500-line file            ~4,000
Read a 2000-line file           ~16,000
List directory contents         ~500
Grep search results             ~1,000
Tool call overhead              ~200 each
-----------------------------------------
Total after 10 file reads       ~50,000+

I was naive. I thought 200K tokens was plenty. Then I built an agent that:

Explored a codebase structure
Read relevant files
Made tool calls to find patterns
Stored conversation history

Within an hour, I was at 180K tokens. The agent was one file read away from crashing.

The Three-Layer Solution

The key insight: context is finite, but work is infinite. You need a strategy to make room.

Here’s the architecture:

Every turn:
+------------------+
| Tool call result |
+------------------+
        |
        v
[Layer 1: micro_compact]        (silent, every turn)
  Replace tool_result > 3 turns old
  with "[Previous: used {tool_name}]"
        |
        v
[Check: tokens > 50000?]
   |               |
   no              yes
   |               |
   v               v
continue    [Layer 2: auto_compact]
              Save transcript to .transcripts/
              LLM summarizes conversation.
              Replace all messages with [summary].

Let me break down each layer.

Layer 1: micro_compact (Silent, Every Turn)

This runs automatically after every turn. It’s invisible to the user but essential.

KEEP_RECENT = 3  # Keep last 3 tool results

def micro_compact(messages: list) -> list:
    """Replace old tool results with placeholders."""
    tool_results = []
    for i, msg in enumerate(messages):
        if msg["role"] == "user" and isinstance(msg.get("content"), list):
            for j, part in enumerate(msg["content"]):
                if isinstance(part, dict) and part.get("type") == "tool_result":
                    tool_results.append((i, j, part))

    # Keep recent results, compress older ones
    if len(tool_results) <= KEEP_RECENT:
        return messages

    for _, _, part in tool_results[:-KEEP_RECENT]:
        if len(part.get("content", "")) > 100:
            tool_name = part.get("tool_name", "unknown")
            part["content"] = f"[Previous: used {tool_name}]"

    return messages

Why this works:

Old tool results are often irrelevant to current task
Placeholder preserves the fact that a tool was used
Reduces tokens by 90%+ on large outputs

Example transformation:

BEFORE (5,000 tokens):
{"type": "tool_result", "content": "def process_data(data):\n    # ... 200 lines of code ...\n    return result"}

AFTER (10 tokens):
{"type": "tool_result", "content": "[Previous: used read_file]"}

I initially tried keeping ALL tool results. Bad idea. The context bloated immediately. Then I tried keeping just the last result. Also bad—I lost critical context mid-task. Keeping the last 3 results was the sweet spot.

Layer 2: auto_compact (Threshold Triggered)

When tokens exceed a threshold (e.g., 50,000), the agent triggers a full summarization.

THRESHOLD = 50000  # Trigger compression at 50K tokens

def auto_compact(messages: list) -> list:
    """Summarize conversation when threshold exceeded."""
    # Save transcript for recovery
    transcript_path = TRANSCRIPT_DIR / f"transcript_{int(time.time())}.jsonl"
    with open(transcript_path, "w") as f:
        for msg in messages:
            f.write(json.dumps(msg, default=str) + "\n")

    # LLM summarizes
    response = client.messages.create(
        model=MODEL,
        messages=[{
            "role": "user",
            "content": "Summarize this conversation for continuity. Focus on:\n"
                      "1. Current task and progress\n"
                      "2. Key findings and decisions\n"
                      "3. Files explored and their purposes\n"
                      "4. Next steps to take\n\n"
                      + json.dumps(messages, default=str)[:80000]
        }],
        max_tokens=2000,
    )

    # Replace with compressed version
    return [
        {"role": "user", "content": f"[Compressed]\n\n{response.content[0].text}"},
        {"role": "assistant", "content": "Understood. Continuing."},
    ]

Key insight: transcripts preserve full history on disk. Nothing is truly lost—just moved out of active context.

This saved me multiple times. When I needed to recover something from 50 turns ago, I just opened the transcript file:

.transcripts/
  transcript_1710745200.jsonl
  transcript_1710748900.jsonl
  transcript_1710752600.jsonl

Each .jsonl file contains the full message history:

{"role": "user", "content": [{"type": "text", "text": "Read the main.py file"}]}
{"role": "assistant", "content": [{"type": "text", "text": "I'll read the file."}, {"type": "tool_use", "name": "read_file", "input": {"path": "main.py"}}]}
{"role": "user", "content": [{"type": "tool_result", "content": "def main():\n..."}]}

Layer 3: Manual Compact (On-Demand)

Sometimes the model needs to request compression explicitly. This happens when:

The model detects it’s running low on context
The user requests a “fresh start” while keeping key info
A major task phase completes

def check_manual_compact_needed(current_tokens: int, max_tokens: int) -> bool:
    """Check if model should request compression."""
    # Leave 20% buffer for response generation
    return current_tokens > max_tokens * 0.8

# Model can emit a special tool call:
{
    "type": "tool_use",
    "name": "compact_context",
    "input": {
        "reason": "Approaching context limit before complex task",
        "preserve_topics": ["current_file_being_edited", "error_we_are_fixing"]
    }
}

Putting It All Together

Here’s how the layers work in sequence:

def manage_context(messages: list, max_tokens: int = 200000) -> list:
    # Layer 1: Always run micro_compact
    messages = micro_compact(messages)

    # Check current token count
    current_tokens = count_tokens(messages)

    # Layer 2: Auto-trigger if threshold exceeded
    if current_tokens > THRESHOLD:
        messages = auto_compact(messages)

    # Layer 3: Signal if approaching hard limit
    if current_tokens > max_tokens * 0.8:
        # Model can request manual compact
        return messages, "compact_suggested"

    return messages, "ok"

The results speak for themselves:

Without compression:
  - Crashes at ~50 turns
  - Lost all context at crash
  - No recovery possible

With three-layer compression:
  - Runs indefinitely
  - Maintains critical context
  - Full history recoverable from transcripts

Lessons Learned

Don’t wait until you hit the limit. Start compressing early with micro_compact. It’s painless and automatic.
Keep transcripts. They’re your safety net. I’ve recovered from crashes by feeding transcripts back into fresh sessions.
The threshold matters. Set it too low, and you compress too often (expensive). Set it too high, and you risk hitting the hard limit. 50K tokens worked well for a 200K context window.
Preserve what matters. The summarization prompt should focus on:
- Current task state
- Key decisions made
- Files explored and their purposes
- Next steps
Don’t compress everything. Some tool results are critical. Keep recent ones, compress older ones.

Context Window: The maximum number of tokens an LLM can process in a single request. Claude’s context window ranges from 100K to 200K tokens depending on the model.

Summarization: Using an LLM to condense a long conversation into key points. Essential for context management.

Transcript Persistence: Saving full conversation history to disk before compression. Enables recovery and debugging.

References

Claude Code Context Compact Documentation - Official docs on context management
LangChain Memory - Memory patterns for agents
MemGPT - Virtual context management for LLMs

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Context will fill up. That’s a guarantee. But with three layers of compression—silent micro-compression, automatic summarization, and manual triggers—your agents can run indefinitely. The key insight: nothing is truly lost, just archived. Your transcripts hold the full history, while your active context stays lean and focused.