How AI Agents Work Internally: The LLM Loop Architecture Explained

Mar 25, 2026

Problem

I’ve been using Claude Code, Cursor, and Devin for months. They feel magical - I type a request, and they somehow know what to do. But I had no idea what was actually happening inside.

When things went wrong, I was stuck. Why did my agent suddenly get slow? Why did it cost so much more than expected? Why did it sometimes get stuck in loops?

I needed to understand what’s under the hood.

What I Thought Was Happening

Before I dug in, I imagined agents worked something like this:

User Request → AI Brain → Magic → Result

The AI would somehow “understand” my intent and execute commands directly. I assumed the LLM was executing the tools itself.

This mental model was completely wrong.

What’s Actually Happening

Simon Willison (Django co-creator) cut through the hype in September 2025:

An AI agent is an LLM in a loop calling tools until the goal is achieved.

That’s it. No magic. Just a loop.

OpenAI’s Lilian Weng formalized this further:

Agent = LLM + Planning + Memory + Tool Calling

The core architecture is a perception-thinking-action cycle:

+------------------+
|  User Request    |
+--------+---------+
         |
         v
+--------+---------+     +--------+---------+
|  LLM "thinks"    | --> |  Decides what   |
|  about the task  |     |  tool to call   |
+--------+---------+     +--------+---------+
         ^                        |
         |                        v
         |              +--------+---------+
         |              |  External code   |
         |              |  executes tool   |
         |              +--------+---------+
         |                        |
         |                        v
         |              +--------+---------+
         +--------------|  Result fed back |
                        |  as new message  |
                        +------------------+

The Key Insight: LLM Never Executes Tools

This is the part I got wrong. The LLM never executes tools. It only generates JSON describing what it wants to do.

Here’s the actual flow:

LLM receives conversation history (including available tools)
LLM outputs structured JSON: { "tool": "read_file", "args": { "path": "/src/app.py" } }
External orchestration code parses the JSON
External code executes the tool (reads the file)
Result is appended to conversation history
Loop back to step 1

The LLM is the decision-maker, not the executor. This separation is critical for safety.

Building a Minimal Agent Loop

Let me show you the actual code pattern. Here’s a minimal agent loop in Python:

from anthropic import Anthropic

client = Anthropic()

# The conversation history - this grows with each iteration
messages = [
    {"role": "user", "content": "Find all Python files in the src directory"}
]

# Tool definitions (simplified)
tools = [
    {
        "name": "list_files",
        "description": "List files in a directory",
        "input_schema": {
            "type": "object",
            "properties": {
                "path": {"type": "string"}
            },
            "required": ["path"]
        }
    },
    {
        "name": "read_file",
        "description": "Read a file's contents",
        "input_schema": {
            "type": "object",
            "properties": {
                "path": {"type": "string"}
            },
            "required": ["path"]
        }
    }
]

# The agent loop
max_iterations = 10
iteration = 0

while iteration < max_iterations:
    # Step 1: LLM decides what to do
    response = client.messages.create(
        model="claude-opus-4-20250514",
        max_tokens=4096,
        tools=tools,
        messages=messages
    )

    # Step 2: Check if LLM wants to use a tool
    if response.stop_reason != "tool_use":
        # LLM decided task is complete - no more tool calls
        print(response.content[0].text)
        break

    # Step 3: Extract tool call from response
    tool_use = next(block for block in response.content if block.type == "tool_use")
    tool_name = tool_use.name
    tool_args = tool_use.input

    print(f"LLM wants to call: {tool_name}({tool_args})")

    # Step 4: External code executes the tool (NOT the LLM!)
    if tool_name == "list_files":
        result = list_files(tool_args["path"])  # Your implementation
    elif tool_name == "read_file":
        result = read_file(tool_args["path"])  # Your implementation
    else:
        result = f"Unknown tool: {tool_name}"

    # Step 5: Feed result back to LLM
    messages.append({"role": "assistant", "content": response.content})
    messages.append({
        "role": "user",
        "content": [{
            "type": "tool_result",
            "tool_use_id": tool_use.id,
            "content": result
        }]
    })

    iteration += 1

if iteration >= max_iterations:
    print("Agent hit maximum iterations - safety limit triggered")

When I run this with a file listing task:

LLM wants to call: list_files({'path': 'src'})
LLM wants to call: read_file({'path': 'src/main.py'})
LLM wants to call: read_file({'path': 'src/utils.py'})
Found 2 Python files: main.py, utils.py

The main.py file contains the application entry point...

Why This Architecture Matters

Understanding this loop explains several behaviors I noticed:

Why Agents Get Slower Over Time

Each iteration adds to the messages array. After 10 tool calls, the LLM processes 10+ messages. After 50 calls, it processes 50+ messages. The context grows, and so does latency.

Iteration 1:  [user, assistant(1), tool_result(1)]  ~500 tokens
Iteration 5:  [user, assistant(1), tool_result(1), ..., tool_result(5)]  ~2,500 tokens
Iteration 20: [..., tool_result(20)]  ~10,000 tokens

Why Agents Get More Expensive

LLM pricing is per token. More messages = more tokens = higher cost. A long-running agent can cost 10x more than a simple query.

Why Context Engineering Replaced Prompt Engineering

The entire conversation history IS the context. Tool results, previous decisions, errors - all fed back. The “context window” (200K tokens for Claude) is your budget. Run out, and you hit errors.

Why Tool Schemas Matter

The LLM sees tool definitions as JSON schemas:

# What the LLM actually sees
tools = [
    {
        "name": "search_files",
        "description": "Search for files matching a pattern",
        "input_schema": {
            "type": "object",
            "properties": {
                "pattern": {
                    "type": "string",
                    "description": "Glob pattern to match files"
                },
                "path": {
                    "type": "string",
                    "description": "Directory to search in"
                }
            },
            "required": ["pattern"]
        }
    }
]

Vague descriptions lead to wrong tool calls. Clear schemas lead to correct behavior.

Common Misconceptions I Had

Misconception 1: LLM Executes Tools Directly

No. The LLM outputs JSON. External code parses and executes. This separation is intentional - it prevents the AI from running arbitrary code.

Misconception 2: Agents Have Hidden Intelligence

Agents don’t have secret capabilities. They have:

An LLM (same model you can call directly)
A loop
Tool definitions
Message history management

That’s the entire magic.

Misconception 3: More Tools = Better Agent

More tools mean more choices for the LLM, which can lead to confusion. I found that 5-10 well-defined tools work better than 50 vague ones.

Safety Boundaries

The loop architecture enables safety controls:

# Maximum iterations - prevent infinite loops
MAX_ITERATIONS = 50

# Token budget - prevent runaway costs
MAX_TOKENS = 100000

# Timeout - prevent hanging
TIMEOUT_SECONDS = 300

# Allowed tools - restrict what can be called
ALLOWED_TOOLS = ["read_file", "write_file", "search"]

# The loop checks these limits
def should_continue(iteration, total_tokens, elapsed_time):
    if iteration >= MAX_ITERATIONS:
        return False, "Maximum iterations reached"
    if total_tokens >= MAX_TOKENS:
        return False, "Token budget exceeded"
    if elapsed_time >= TIMEOUT_SECONDS:
        return False, "Timeout exceeded"
    return True, "Continue"

Claude Code, Cursor, Devin - they all implement similar safety limits. When an agent “times out” or “hits a limit,” this is what’s happening.

How Major Products Implement This

All major AI coding tools share the same skeleton:

Product	LLM	Loop	Tools	Key Difference
Claude Code	Claude Opus	Yes	File system, terminal, web	Deep integration with Claude models
Cursor	Claude/GPT-4	Yes	File system, terminal, search	IDE integration, context awareness
Devin	Custom	Yes	Full development environment	More tools, longer context
Manus	Various	Yes	Code execution, web	Evaluation-focused

The differences are in engineering precision, not fundamental architecture.

Summary

AI agents are elegantly simple in principle: an LLM loop with tool calling. The LLM decides what to do, generates JSON, external code executes the tool, and results feed back. This cycle repeats until the task is complete or a safety limit triggers.

Understanding this architecture explains why agents get slower and more expensive over time (growing context), why tool schemas need to be precise (LLM sees them as JSON), and why context engineering is now more important than prompt engineering.

The next time you use Claude Code or Cursor, remember: it’s just a very well-engineered loop.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 Simon Willison's Blog on AI Agents
👨‍💻 Lilian Weng's Agent Architecture Post
👨‍💻 Anthropic Tool Use Documentation

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!