Skip to content
BSWEN
Blog

Index vs Dump: How to Organize AI Agent Memory for Maximum Effectiveness

Cowrie
Cowrie
Dev @ Bswen

Problem

My AI agent’s memory became unusable after three months of development.

I had been dumping everything into a single MEMORY.md file. Every conversation, every decision, every project detail—all appended to one growing document. When I started, it worked fine. A few hundred lines, easy to search, quick to load.

Then it hit 10,000 lines. Then 30,000. Then 50,000.

Here’s what happened when I asked my agent a simple question:

Terminal Output
User: What did we decide about the database schema for the user service?


Agent: Let me search my memory...
[Loading MEMORY.md - 52,847 lines]
[Context window exhausted after 50,000 lines]
[Error: Cannot find relevant information in loaded context]

The agent couldn’t find a decision we made two weeks ago because it had to load 50,000 lines of mixed content. The context window filled with irrelevant information before reaching the relevant section.

I realized I had made a fundamental mistake: I was treating memory as a log file, not a library.

What Happened?

I searched for best practices on AI agent memory and found a Reddit discussion about OpenClaw. The most critical warning jumped out immediately:

“Do not let memory become one giant file.”

The thread explained the difference between two approaches:

  1. Dump approach: Append everything to one file
  2. Index approach: Use a lightweight index pointing to structured files

I was using the dump approach. Here’s what my memory structure looked like:

My Broken Memory Structure
memory/
└── MEMORY.md (52,847 lines of chaos)

This structure has several fatal flaws:

  • Context overload: Loading memory means loading everything
  • No lazy loading: Can’t selectively load relevant sections
  • Maintenance nightmare: Where do you add new information?
  • Performance degradation: Slower with each addition
  • Lost information: Important details buried in noise

How I Fixed It

I restructured my memory using the index-based approach. Here’s what I learned.

Step 1: Create a Lightweight Index

The MEMORY.md file should be an index, not a dump. It points to files, it doesn’t contain them.

memory/MEMORY.md
# Memory Index


## Active Projects
- [[projects/ai-assistant-refactor]] - Refactoring the main assistant
- [[projects/customer-onboarding]] - New user flow implementation


## Key People
- [[people/john-doe]] - Backend lead, prefers async communication
- [[people/jane-smith]] - Product manager, weekly syncs


## Recent Decisions
- [[decisions/2024-01-architecture-choice]] - Why we chose PostgreSQL
- [[decisions/2024-02-auth-strategy]] - OAuth2 implementation approach


## Daily Logs
- [[logs/2024-03-01]] - Initial planning session
- [[logs/2024-03-02]] - Database schema review

This index is small—maybe 50 lines. It loads instantly. It tells the agent where to find specific information.

Step 2: Create Structured Folders

Each category gets its own folder with focused files:

Correct Memory Structure
memory/
├── MEMORY.md           # Lightweight index only (50 lines)
├── people/
│   ├── john-doe.md     # Individual context per person
│   └── jane-smith.md
├── projects/
│   ├── ai-assistant-refactor.md
│   └── customer-onboarding.md
├── decisions/
│   ├── 2024-01-architecture-choice.md
│   └── 2024-02-auth-strategy.md
└── logs/
    ├── 2024-03-01.md   # Raw daily journals
    └── 2024-03-02.md

Each file has a single purpose. Adding information is straightforward—you create or update the appropriate file.

Step 3: Implement Lazy Loading

The agent now follows this loading strategy:

  1. Always load MEMORY.md (small, fast)
  2. Parse index to identify relevant sections
  3. Load specific files only when needed

Here’s the code I wrote to implement this:

memory.py
from pathlib import Path
from typing import Optional, List
import re


class AgentMemory:
    """Index-based memory with lazy loading."""


    def __init__(self, memory_dir: str = "memory/"):
        self.memory_dir = Path(memory_dir)
        self.index_path = self.memory_dir / "MEMORY.md"


    def load_index(self) -> str:
        """Load lightweight index - always fast."""
        if not self.index_path.exists():
            return "# Memory Index\n\nNo memories yet."
        return self.index_path.read_text()


    def load_file(self, category: str, filename: str) -> str:
        """Load specific file only when needed."""
        path = self.memory_dir / category / filename
        if not path.exists():
            return f"Memory file not found: {category}/{filename}"
        return path.read_text()


    def add_memory(self, category: str, filename: str, content: str) -> None:
        """Add to structured location."""
        category_dir = self.memory_dir / category
        category_dir.mkdir(parents=True, exist_ok=True)


        path = category_dir / filename
        path.write_text(content)


        self._update_index(category, filename)


    def _update_index(self, category: str, filename: str) -> None:
        """Keep index in sync - lightweight updates."""
        index_content = self.load_index()


        # Check if entry already exists
        link = f"[[{category}/{filename}]]"
        if link in index_content:
            return


        # Find the right section and add entry
        lines = index_content.split('\n')
        section_header = f"## {category.replace('-', ' ').title()}"


        new_lines = []
        added = False


        for i, line in enumerate(lines):
            new_lines.append(line)
            if line.strip() == section_header and not added:
                # Add entry after section header
                new_lines.append(f"- [[{category}/{filename}]]")
                added = True


        if added:
            self.index_path.write_text('\n'.join(new_lines))


    def find_relevant_files(self, query: str) -> List[str]:
        """Parse index to find relevant files based on query."""
        index = self.load_index()
        relevant = []


        # Simple keyword matching
        query_lower = query.lower()
        for line in index.split('\n'):
            if '[[' in line and ']]' in line:
                # Extract file path from link
                match = re.search(r'\[\[([^\]]+)\]\]', line)
                if match:
                    file_path = match.group(1)
                    if query_lower in line.lower() or query_lower in file_path.lower():
                        relevant.append(file_path)


        return relevant


    def get_context_for_query(self, query: str) -> str:
        """Load index + relevant files for a query."""
        context_parts = [self.load_index()]


        relevant_files = self.find_relevant_files(query)
        for file_path in relevant_files:
            parts = file_path.split('/')
            if len(parts) == 2:
                category, filename = parts
                context_parts.append(f"\n---\n# {file_path}\n\n{self.load_file(category, filename)}")


        return '\n'.join(context_parts)

Step 4: Add Semantic Search (Optional Enhancement)

For larger memory systems, I added semantic search using vector embeddings:

hybrid_memory.py
from sentence_transformers import SentenceTransformer
import faiss
import numpy as np
from pathlib import Path
from typing import List, Tuple


class HybridMemory(AgentMemory):
    """Memory with semantic search capability."""


    def __init__(self, memory_dir: str = "memory/"):
        super().__init__(memory_dir)
        self.encoder = SentenceTransformer('all-MiniLM-L6-v2')
        self.vector_index = faiss.IndexFlatL2(384)  # Vector dimension
        self.file_paths: List[str] = []


    def build_search_index(self) -> None:
        """Build vector index from all memory files."""
        self.vector_index = faiss.IndexFlatL2(384)
        self.file_paths = []


        categories = ['people', 'projects', 'decisions', 'logs']
        for category in categories:
            category_dir = self.memory_dir / category
            if not category_dir.exists():
                continue


            for file_path in category_dir.glob('*.md'):
                content = file_path.read_text()
                if content.strip():
                    embedding = self.encoder.encode([content])
                    self.vector_index.add(embedding.astype('float32'))
                    self.file_paths.append(f"{category}/{file_path.name}")


    def find_relevant_semantic(self, query: str, k: int = 5) -> List[Tuple[str, float]]:
        """Find top-k relevant memory files using semantic search."""
        if self.vector_index.ntotal == 0:
            self.build_search_index()


        if self.vector_index.ntotal == 0:
            return []


        query_embedding = self.encoder.encode([query])
        distances, indices = self.vector_index.search(
            query_embedding.astype('float32'), min(k, self.vector_index.ntotal)
        )


        results = []
        for idx, dist in zip(indices[0], distances[0]):
            if idx < len(self.file_paths):
                results.append((self.file_paths[idx], float(dist)))


        return results


    def get_enhanced_context(self, query: str) -> str:
        """Load index + semantically relevant files."""
        context_parts = [self.load_index()]


        semantic_results = self.find_relevant_semantic(query, k=3)
        for file_path, distance in semantic_results:
            if distance < 1.0:  # Only include if reasonably relevant
                parts = file_path.split('/')
                if len(parts) == 2:
                    category, filename = parts
                    content = self.load_file(category, filename)
                    context_parts.append(f"\n---\n# {file_path} (relevance: {1-distance:.2f})\n\n{content}")


        return '\n'.join(context_parts)

Why This Matters

After switching to the index-based approach, I saw immediate improvements:

Scalability: My index stayed at ~50 lines while my memory grew to hundreds of files. Performance remained consistent.

Context Efficiency: My agent now loads only what’s relevant. Working on a project? Load that project file. Need to recall a decision? Load that specific decision file.

Maintainability: Each memory file has a single purpose. I know exactly where to add new information.

Human Readability: Markdown files are easy for me to review and edit. I can debug agent behavior by reading the memory files.

Common Mistakes I Made

Mistake 1: Dumping Everything in MEMORY.md

WRONG: Giant memory dump
# MEMORY.md (50,000 lines)


## 2024-01-01
Today I worked on... [500 lines of details]


## 2024-01-02
Met with John about... [300 lines of details]
# ... continues for months

This becomes unusable quickly. The agent loads everything, context fills up, and finding specific information becomes impossible.

Mistake 2: No Structure at All

WRONG: No structure
memory/
└── everything.md  # All memories in one file

Without structure, there’s no way to selectively load context.

Mistake 3: Over-Engineering the Index

WRONG: Index with full content
## Projects
- AI Assistant Refactor
  Status: In Progress
  Team: John, Jane
  Last Update: 2024-03-01
  Details: [100 lines of project details]

The index should point, not contain. Keep it lightweight.

While structure is important, don’t ignore the power of semantic search. A hybrid approach—structured files + search capability—provides the best of both worlds.

Comparison: Before vs After

Memory Access Comparison
BEFORE (Dump Approach):
1. Load MEMORY.md (52,847 lines)
2. Context window fills
3. Agent cannot find relevant information
4. Agent gives up or hallucinates


AFTER (Index Approach):
1. Load MEMORY.md (50 lines)
2. Parse index, find relevant section
3. Load specific file (e.g., decisions/2024-01-architecture-choice.md)
4. Agent has exact information needed
5. Total context: ~200 lines

The difference is dramatic. What used to fail now succeeds consistently.

When to Use Each Approach

The index approach works best when:

  • Your agent runs for weeks or months
  • You accumulate many decisions and conversations
  • Multiple projects or domains are involved
  • You need to find specific past information

The dump approach might be acceptable when:

  • Your agent has short-lived sessions
  • Memory is primarily for recent context
  • You don’t need to search past information
  • Total memory stays under a few thousand lines

Summary

In this post, I showed how to organize AI agent memory using an index-based architecture. The key point is using a lightweight MEMORY.md index pointing to structured folders, with lazy loading and optional semantic search.

The transformation from dump to index approach fixed my agent’s memory problem. Instead of a 50,000-line monolith that couldn’t be searched, I now have a fast, scalable memory system that loads only relevant context.

Key takeaways:

  1. MEMORY.md is an index, not a dump
  2. Structured folders provide clear organization
  3. Lazy loading prevents context overload
  4. Semantic search enhances retrieval for large systems
  5. Each memory file has a single purpose

Your agent’s memory is its foundation. Treat it like a library, not a log file.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags: