How to add persistent memory to AI agents with markdown

Mar 3, 2026

Cowrie

Dev @ Bswen

Purpose

This post demonstrates how to add persistent memory to AI agents using markdown files.

Environment

Python 3.10+
memsearch (latest)
OpenAI or Anthropic API or Ollama (local)

The Problem with Traditional Agent Memory

When I built my first AI agent, I ran into a fundamental problem: the agent couldn’t remember anything between sessions. Each new conversation started from scratch. The agent had no idea what we discussed yesterday or last week.

This limitation comes from how LLMs work:

Context windows fill up during long conversations
Sessions are isolated from each other
Without manual state management, nothing persists

I tried several solutions:

LangChain Memory Stores:

Required learning the LangChain framework
Memory stored in opaque internal state
Hard to debug when things went wrong
Tied to LangChain’s architecture

Vector Databases (Chroma, Pinecone, Weaviate):

Memories stored as binary blobs
Can’t read or edit memory files directly
Vendor lock-in makes migration painful
Need separate tools to inspect what’s stored

MCP Servers:

Added unnecessary complexity
Required running separate services
Made deployment harder
Introduced more failure points

All these approaches shared the same problem: memory became opaque. I couldn’t just open a file, read what the agent “remembered,” and make corrections.

Markdown-First Memory Architecture

The solution I found is simple: use markdown files as the source of truth for agent memory. Vector databases become optional indexes that can be rebuilt anytime.

Here’s how the architecture works:

User Query
    ↓
[Embed Query Vector]
    ↓
[Milvus Vector DB]
    ↓
[Cosine Similarity Search]
    ↓
Top-K Results + Scores
    ↓
Inject into LLM Context
    ↓
[Generate Response]
    ↓
[Save to Markdown]
    ↓
[Auto-Index to Vector DB]

The key insight: markdown files are the primary storage. The vector database is a derived index that can be regenerated if needed. This gives you:

Human-readable memory you can inspect and edit
Git version control for all memories
Zero vendor lock-in
Easy debugging when the agent retrieves wrong context
Simple migration between systems

Quick Start with memsearch

I used a library called memsearch to implement this. It handles the indexing and search while keeping markdown as the source of truth.

First, install it:

pip install memsearch[openai]  # For OpenAI embeddings
# OR
pip install memsearch[anthropic]  # For Anthropic embeddings
# OR
pip install memsearch[ollama]  # For fully local with Ollama

Basic usage is simple:

from memsearch import MemSearch

mem = MemSearch(paths=["./memory"])
await mem.index()
results = await mem.search("Redis config", top_k=3)
print(results[0]["content"], results[0]["score"])

The paths parameter tells memsearch where to look for markdown files. The index() call processes all markdown files and builds the vector index. The search() call finds relevant memories using semantic similarity.

Full Agent Implementation

Here’s a complete agent that remembers conversations across sessions:

import asyncio
from datetime import date
from pathlib import Path
from openai import OpenAI
from memsearch import MemSearch

MEMORY_DIR = "./memory"
llm = OpenAI()
mem = MemSearch(paths=[MEMORY_DIR])

def save_memory(content: str):
    p = Path(MEMORY_DIR) / f"{date.today()}.md"
    p.parent.mkdir(parents=True, exist_ok=True)
    with open(p, "a") as f:
        f.write(f"\n{content}\n")

async def agent_chat(user_input: str) -> str:
    # 1. Recall - search memory for relevant context
    memories = await mem.search(user_input, top_k=3)
    context = "\n".join(f"- {m['content'][:200]}" for m in memories)

    # 2. Think - generate response using retrieved memories
    resp = llm.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"You have these memories:\n{context}"},
            {"role": "user", "content": user_input},
        ],
    )
    answer = resp.choices[0].message.content

    # 3. Remember - save conversation to markdown
    save_memory(f"## {user_input}\n{answer}")
    await mem.index()

    return answer

async def main():
    while True:
        user_input = input("You: ")
        if user_input.lower() in ["exit", "quit"]:
            break
        response = await agent_chat(user_input)
        print(f"Agent: {response}")

asyncio.run(main())

The agent follows a simple pattern:

Recall: Search memory for relevant past conversations
Think: Generate a response using the retrieved context
Remember: Save the new interaction to markdown and re-index

I can see my memories in plain text:

## How do I configure Redis for caching?

Use these Redis settings for caching:
- `maxmemory-policy allkeys-lru`
- `save ""` (disable persistence)
- `timeout 300`
- Set appropriate `maxmemory` based on available RAM

## What's the best way to handle API rate limits?

Implement exponential backoff:
1. Start with 1 second delay
2. Double delay on each retry
3. Cap at 60 seconds
4. Use jitter to avoid thundering herd

Anthropic Claude Integration

The same pattern works with Anthropic’s Claude:

import asyncio
from datetime import date
from pathlib import Path
from anthropic import Anthropic
from memsearch import MemSearch

MEMORY_DIR = "./memory"
llm = Anthropic()
mem = MemSearch(paths=[MEMORY_DIR])

async def agent_chat(user_input: str) -> str:
    memories = await mem.search(user_input, top_k=3)
    context = "\n".join(f"- {m['content'][:200]}" for m in memories)

    resp = llm.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=1024,
        system=f"You have these memories:\n{context}",
        messages=[{"role": "user", "content": user_input}],
    )
    answer = resp.content[0].text

    save_memory(f"## {user_input}\n{answer}")
    await mem.index()
    return answer

Fully Local with Ollama

If you want to keep everything local, Ollama works well:

import asyncio
from datetime import date
from pathlib import Path
from ollama import chat
from memsearch import MemSearch

MEMORY_DIR = "./memory"
mem = MemSearch(paths=[MEMORY_DIR], embedding_provider="ollama")

async def agent_chat(user_input: str) -> str:
    memories = await mem.search(user_input, top_k=3)
    context = "\n".join(f"- {m['content'][:200]}" for m in memories)

    resp = chat(
        model="llama3.2",
        messages=[
            {"role": "system", "content": f"You have these memories:\n{context}"},
            {"role": "user", "content": user_input},
        ],
    )
    answer = resp.message.content

    save_memory(f"## {user_input}\n{answer}")
    await mem.index()
    return answer

This gives you full privacy and no API costs.

How Semantic Search Works

When you search memory, memsearch converts your query into a vector embedding and finds similar vectors in the database. The similarity is measured using cosine similarity, which returns a score between 0 (unrelated) and 1 (identical).

Here’s what happens under the hood:

Query: "Redis caching configuration"
    ↓
[Embedding Model: text-embedding-3-small]
    ↓
Vector: [0.0123, -0.0456, 0.0789, ...] (1536 dimensions)
    ↓
[Milvus Search - Cosine Similarity]
    ↓
Results:
  1. Score: 0.894 "Use these Redis settings for caching..."
  2. Score: 0.756 "Implement Redis connection pooling..."
  3. Score: 0.623 "Database caching strategies..."

The embedding model (text-embedding-3-small from OpenAI) converts text into a 1536-dimensional vector. Similar concepts end up with similar vectors, so the search finds semantically related content even if the exact words don’t match.

memsearch also supports hybrid search that combines:

Dense vectors (semantic similarity)
BM25 (keyword matching)
Reciprocal Rank Fusion (RRF) for reranking

This gives you the best of both worlds: semantic understanding plus exact keyword matching.

File Structure

Your memory directory structure looks like this:

.memsearch/
├── memory/
│   ├── 2026-02-09.md
│   ├── 2026-02-10.md
│   └── 2026-02-11.md
└── config.toml

Each markdown file contains memories for a day. The config.toml file stores settings like the embedding provider and index location.

Integration Patterns

I found memsearch works with most agent frameworks:

LangGraph: Create a custom tool that wraps mem.search()

from langchain_core.tools import tool

@tool
def search_memory(query: str) -> str:
    results = await mem.search(query, top_k=3)
    return "\n".join(m["content"] for m in results)

CrewAI: Create a custom tool class

from crewai.tools import tool

@tool("Search Memory")
def search_memory(query: str) -> str:
    """Search agent memory for relevant information."""
    results = await mem.search(query, top_k=3)
    return "\n".join(m["content"] for m in results)

The key is the same: use mem.search() to retrieve context before generating responses.

Production Considerations

When I moved this to production, I learned a few things:

Live file watching: Use watchdog to automatically re-index when memory files change:

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class MemoryHandler(FileSystemEventHandler):
    def on_modified(self, event):
        if event.src_path.endswith(".md"):
            asyncio.create_task(mem.index())

observer = Observer()
observer.schedule(MemoryHandler(), "./memory", recursive=False)
observer.start()

Deduplication: memsearch uses SHA-256 content hashing to avoid storing duplicate chunks. If you update a memory, it only re-indexes the changed parts.

Backend options: Milvus offers three modes:

Milvus Lite: Embedded, runs in-process
Milvus Server: Distributed, for production
Zilliz Cloud: Managed service

Cost optimization:

Batch embeddings when possible
Use local models (Ollama) for privacy
Cache frequently retrieved memories
Set appropriate chunk sizes (500-1000 tokens usually works well)

Comparison: memsearch vs Alternatives

Feature	memsearch	LangChain Memory	claude-mem
Storage	Markdown files	Internal state	SQLite + Chroma
Transparency	Full	Partial	None
Git-friendly	Yes	No	No
Setup	1 pip install	Framework required	Node stack
Recall method	Automatic hook	Manual tool call	Agent-driven
Vendor lock-in	Zero	Medium	High

The key difference is transparency. With memsearch, you can open any memory file and see exactly what your agent knows. With other systems, you’re guessing what’s stored in opaque databases.

Common Pitfalls

I ran into these problems and learned how to avoid them:

Chunking too small: Creates many tiny fragments that lack context. Aim for 500-1000 tokens per chunk.

Chunking too large: Makes it hard to find specific information. If a chunk is too big, search results return irrelevant content.

Using wrong embedding model: Make sure your embedding model matches your use case. text-embedding-3-small is good for general purpose; use domain-specific models for technical content.

Not deduplicating content: If you store the same information multiple times, it clutters search results. memsearch handles this with SHA-256 hashing.

Forgetting to re-index: When you edit memory files manually, run await mem.index() to update the vector database.

Choosing expensive cloud providers: For many use cases, local models (Ollama with nomic-embed-text) work just as well and cost nothing.

The Reason

I think the key reason this markdown-first approach works is separation of concerns:

Markdown files store the actual knowledge (what)
Vector database provides fast semantic search (how)

When you need to debug or edit memories, you work with markdown. When you need fast retrieval, the vector index handles it. This separation gives you the best of both worlds.

Summary

In this post, I showed how to add persistent memory to AI agents using markdown files as the source of truth. The key point is treating vector databases as rebuildable indexes rather than primary storage. This approach gives you human-readable, version-controlled memory with zero vendor lock-in.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 memsearch - A Markdown-first memory system
👨‍💻 OpenAI Embeddings API
👨‍💻 Anthropic Cookbook - Context Management
👨‍💻 LangGraph - Add Memory to Agents
👨‍💻 Milvus Documentation

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!