Crawl4AI Tutorial: Turn Websites into LLM-Ready Markdown for RAG and AI Agents

Sep 2, 2026

crawl4ai logo

TL;DR

Crawl4AI is an open-source (Apache 2.0) Python web crawler built on Playwright. It turns real-world websites, including JavaScript-rendered pages, into clean, LLM-friendly Markdown and structured data.
Its biggest advantage is browser-rendered, content-filtered Markdown instead of raw HTML noise.
It is free to use: Apache 2.0 license, and basic crawling needs no third-party crawling API key.
The key difference from Firecrawl: Crawl4AI is a crawler framework you operate yourself; Firecrawl is a managed scraping API you consume.
It fits best when you want control, local processing, and no per-page API bill: RAG pipelines, AI agents, and local Ollama workflows.

My core judgment: Crawl4AI is attractive when you want maximum control, local processing and no per-page API bill. Firecrawl is usually easier when you prefer a managed scraping infrastructure and do not want to operate browsers, proxies and crawler workers yourself. This is not about one tool beating the other.

What Is Crawl4AI?

Crawl4AI converts JavaScript-rendered websites into clean LLM-ready Markdown for RAG and AI agents

Crawl4AI is an open-source, asynchronous, browser-based web crawler designed to transform websites into LLM-friendly Markdown and structured data.

A traditional scraper built with requests + BeautifulSoup downloads HTML and parses it. That works for simple static pages. Crawl4AI instead drives a real headless browser, so the page executes JavaScript first, then the rendered DOM goes through content filtering before becoming Markdown or structured output.

Website
   ↓
Playwright Browser
   ↓
DOM / rendered page
   ↓
Content filtering
   ↓
Markdown / structured extraction
   ↓
RAG / Agent / Data Pipeline

Why does this matter for LLM applications?

Lower token consumption: you stop sending nav menus, cookie banners, and scripts to the model.
Better embedding quality: embeddings are built from the actual article content, not page chrome.
Less RAG noise: fewer irrelevant chunks means fewer hallucination triggers and wrong retrievals.
Easier chunking: Markdown preserves heading hierarchy and list structure.
Easier agent reasoning: the agent reads a document, not a tag soup.

I will not claim a fixed token-saving percentage here. No reliable public benchmark covers all sites, and results vary a lot depending on how noisy the source page is.

Why LLM Applications Need a Different Kind of Web Crawler

Suppose I want to give an AI agent access to a documentation website. The obvious approach is:

download the HTML
strip tags
send the text to an LLM

The first time I tried this, the “text” I got was mostly garbage. Modern websites contain navigation menus, cookie banners, JavaScript-rendered content, repeated headers, tracking elements, and hundreds of irrelevant links. Raw HTML wastes context tokens, and retrieval over it produces poor results.

Crawl4AI tries to solve exactly this problem: converting real-world websites into clean Markdown or structured data that can be consumed directly by RAG pipelines and AI agents.

For context on the project’s momentum: the repo unclecode/crawl4ai has passed 80,000 GitHub stars as of September 2026 and is licensed under Apache 2.0. It is written in Python and built around Playwright, with LLM and RAG workloads as the primary target. Star count is background, not proof of quality.

Crawl4AI Quick Start

Install Crawl4AI

You need Python, pip, and a Chromium/Playwright environment. Install and verify like this:

pip install -U crawl4ai
crawl4ai-setup
crawl4ai-doctor

crawl4ai-setup downloads and configures the Playwright/Chromium browser that Crawl4AI drives.
crawl4ai-doctor checks your environment and reports missing dependencies. Run it whenever arun() fails to open a browser.

Crawl Your First Page

Here is the minimal runnable example:

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com"
        )
        print(result.markdown)

asyncio.run(main())

The key parts:

AsyncWebCrawler manages a browser instance. Using it as an async context manager means the browser is created on enter and closed on exit, so I do not leak processes.
arun() fetches the URL, renders the page, and converts the DOM to Markdown.
result.markdown holds the converted content (more on raw_markdown and fit_markdown below).

Crawl4AI also ships a CLI called crwl. A single page to Markdown is:

crwl https://example.com -o markdown

Understanding the Markdown Output

The output below is a labeled conceptual example, not a real dump from example.com, so I can show what the structure looks like:

# Example Article

This is the useful article content.

## Section One

- point one
- point two

## Section Two

A code block and a table would follow here.

A raw browser page contains navigation, scripts, a cookie banner, and ads. Crawl4AI returns the article structure above. That is the difference that matters for LLM workloads.

Why Crawl4AI’s Markdown Matters for RAG

Crawl4AI architecture showing where web crawling fits into a RAG pipeline

“Markdown is cleaner than HTML” is only half the story. Markdown preserves structure that matters for retrieval:

heading hierarchy (#, ##, ###) becomes document structure
lists and tables stay readable
links keep their context
paragraphs stay separated

That structure gives you:

Cleaner chunks. A chunker can split on headings and paragraphs instead of cutting through <div> soup.
Better retrieval context. A retrieved chunk carries its section heading, so the LLM knows where it came from.
Lower prompt overhead. No tag attributes, no scripts, no duplicated links in every chunk.
Easier debugging. I can read a stored Markdown chunk directly and tell whether the crawl captured the right content.

Important caveat I want to be honest about: clean Markdown does not automatically mean good RAG. Chunking, deduplication, canonical URLs, metadata, page selection, and freshness still determine retrieval quality. Crawl4AI improves the input; the indexing pipeline is still on me.

Raw Markdown vs Fit Markdown

Crawl4AI raw Markdown compared with filtered fit Markdown for RAG

Crawl4AI does not only convert HTML to Markdown. It can also filter the content so you keep what is actually relevant to the page topic.

raw_markdown: the full conversion, including nav menus, footers, and repeated boilerplate.
fit_markdown: filtered content produced by a content filter.

The two filters to know:

PruningContentFilter: removes low-information nodes based on text density, link density, and tag importance.
BM25ContentFilter: keeps content relevant to a user query, useful when I know what the page should be about.

Here is how I generate both at once:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
from crawl4ai.content_filter_strategy import PruningContentFilter

async def main():
    config = CrawlerRunConfig(
        markdown_generator=DefaultMarkdownGenerator(
            content_filter=PruningContentFilter(threshold=0.6)
        )
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://docs.example.com/install", config=config)
        if result.success:
            print("Raw markdown length:", len(result.markdown.raw_markdown))
            print("Fit markdown length:", len(result.markdown.fit_markdown))
        else:
            print("Crawl failed:", result.error_message)

asyncio.run(main())

The threshold in PruningContentFilter controls how aggressive pruning is. A lower threshold keeps more content; a higher one prunes more.

Conceptually, the difference looks like this:

Raw:  Home | Pricing | Login | Documentation | menu links ...
      <actual article content>
      Newsletter signup | Privacy Policy | Terms | footer links

Fit:  # Documentation
      <actual article content only>

For a RAG index, agent context, or technical documentation, I use fit_markdown in almost every case. Storing boilerplate in a vector database just pollutes retrieval.

Crawling JavaScript and SPA Websites

The most common reason requests.get(url) fails is that the server returns only an HTML shell:

<div id="app"></div>
<script src="app.js"></script>

The real content is rendered by JavaScript in the browser. Crawl4AI uses Playwright, so it executes the JavaScript first and crawls the rendered DOM after. That makes it suitable for:

React, Vue, and Next.js sites
dashboards and SPAs
lazy-loaded pages
dynamic documentation portals

One limit I have to point out: browser rendering is not an anti-bot bypass. Sites behind Cloudflare, CAPTCHAs, fingerprinting, or login walls can still require authenticated sessions, proxy infrastructure, cookies, and specialized browser configuration. Do not expect Crawl4AI’s stealth options to defeat every bot defense.

Extracting Structured Data

Markdown covers text-heavy content. When I need fields like product title, price, or description, Crawl4AI offers three levels of structured extraction.

CSS and XPath Extraction

CSS selectors work well when the page layout is stable. I define a schema and Crawl4AI extracts JSON:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, JsonCssExtractionStrategy

schema = {
    "name": "Product",
    "baseSelector": "div.product",
    "fields": [
        {"name": "title", "selector": "h2", "type": "text"},
        {"name": "price", "selector": ".price", "type": "text"},
    ],
}

async def main():
    config = CrawlerRunConfig(extraction_strategy=JsonCssExtractionStrategy(schema))
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://shop.example.com/items", config=config)
        print(result.extracted_content)

asyncio.run(main())

For more complex DOMs, XPath gives finer control than CSS selectors.

LLM-Based Extraction

When page structures are inconsistent across pages, rule-based selectors break. Crawl4AI can hand the content to an LLM for extraction, and it works with LiteLLM so you can plug in different providers, including local models.

The trade-off is straightforward:

Approach	Pros	Cons
CSS / XPath	Deterministic, fast, cheap	Breaks when layout changes
LLM extraction	Flexible, handles inconsistent layouts	Slower, adds model cost

The key truth here: the crawler itself is free. Infrastructure and optional LLM calls are not.

Crawling an Entire Website

A real RAG project rarely crawls a single URL. I usually need /docs/, /docs/install/, /docs/config/, /docs/api/, and so on. Crawl4AI’s deep crawl covers this with a strategy object, for example breadth-first search:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def main():
    strategy = BFSDeepCrawlStrategy(
        max_depth=2,
        include_external=False,
        max_pages=50,
        score_threshold=0.3,
    )
    config = CrawlerRunConfig(deep_crawl_strategy=strategy)
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://docs.example.com/", config=config)
        print(result.markdown)

asyncio.run(main())

Key controls to know:

max_depth: how many link levels to follow.
max_pages: a hard cap on total pages, which prevents runaway crawls.
include_external: whether to stay on the same domain.
URL and domain filters: restrict which pages are allowed.
Resume and recovery: restart a failed crawl instead of starting over.

Production crawls need planning around duplicate URLs, query parameters, calendar pages and other infinite URL spaces, pagination, robots.txt, and rate limiting. BFS alone does not solve those.

Using Crawl4AI with AI Agents

A common agent workflow looks like this:

User
   ↓
AI Agent
   ↓
Crawl target docs
   ↓
Crawl4AI
   ↓
Clean Markdown
   ↓
LLM reasoning
   ↓
Answer

The user asks: “Find the latest installation instructions for project X.” The agent picks the URLs, Crawl4AI fetches and cleans the pages, and the LLM reasons over the Markdown to answer.

It helps to separate two jobs:

Search engines discover URLs.
Crawl4AI fetches and converts page content.

A practical architecture is Search -> Crawl4AI -> Markdown -> Agent. You can wire Crawl4AI into agent frameworks through MCP servers, agent tool integrations, and live web context. One clarification: there is no built-in crawl4ai-mcp command in the project’s entry points. MCP support comes through community servers and integrations. And Crawl4AI does not give an agent “full internet access” by itself; it fetches the pages the agent chooses.

Crawl4AI + Ollama for Local RAG

Local RAG is a natural fit. The pipeline:

Website
   ↓
Crawl4AI
   ↓
Markdown
   ↓
Embeddings
   ↓
Vector DB
   ↓
Ollama

The benefits:

Crawled data stays on your infrastructure.
You can use a local model for extraction and generation.
Privacy control is easier.
There is no mandatory crawling SaaS API.

One caveat: “local” does not mean fully offline. The crawler still makes outbound requests to the target websites. Local refers to where the processing and storage happen.

Crawl4AI vs Firecrawl

Crawl4AI self-hosted architecture versus Firecrawl managed crawling API

This is the decision section. Philosophy first:

Crawl4AI is a crawler framework you operate.
Firecrawl is a scraping service/API you consume.

That difference drives everything else:

Area	Crawl4AI	Firecrawl
Open source	Yes	Yes / self-hosting components available
Default model	Self-hosted crawler	Managed API-oriented
Python integration	Excellent	API / SDK
Browser control	High	Abstracted
Markdown output	Yes	Yes
RAG use	Yes	Yes
Infrastructure	You operate it	Managed option available
Proxy / anti-bot	Mostly BYO	Easier through managed infrastructure
Per-page API fee	No crawler API fee	Managed plans use credits
Local LLM	Easy	Possible depending on architecture
Operational burden	Higher	Lower with hosted service

One wording note: Firecrawl provides a managed API with usage-based pricing while also offering open-source and self-hosted components. It is not simply “paid”.

The architecture difference in two diagrams:

Your App
   ↓
Your Crawl4AI
   ↓
Your Browser
   ↓
Website

Your App
   ↓
Firecrawl API
   ↓
Managed Crawling Infrastructure
   ↓
Website

When Crawl4AI Makes More Sense

Privacy is important and you want data to stay inside your network.
You already run Python infrastructure.
Crawling volume makes per-page fees significant.
You need browser-level control (custom headers, sessions, JS execution).
You want Ollama or local-model workflows.

When Firecrawl Makes More Sense

Time to production matters more than infrastructure control.
You do not want to manage Playwright and browser lifecycles.
You need managed infrastructure and proxy handling.
Your team wants a simple HTTP API without running a crawl service.

The real comparison is often engineering cost vs API cost. Decide which one you would rather spend.

What Crawl4AI Does Not Solve

This is the credibility section. Crawl4AI is good at converting pages, but it does not magically fix:

Anti-bot systems. Playwright is not a guaranteed Cloudflare bypass.
Proxy infrastructure. Large-scale scraping still needs residential proxies, IP rotation, and request throttling.
CAPTCHA. Crawl4AI is not a CAPTCHA solver.
Website instability. DOM changes break CSS selectors and XPath every time a site redesigns.
Infinite crawling. Bad URL policies produce ?page=1, ?page=2, ?page=3 forever, or endless calendar pages.
Operational cost. Free does not mean zero cost: CPU, RAM, bandwidth, browsers, proxies, engineering, and monitoring all cost something.

Running Crawl4AI in Production

The community discussions I have read converge on the same pain points.

Memory consumption

Headless browsers are heavy. Crawling 50 URLs is not 50 lightweight HTTP requests; it can mean many browser pages and contexts alive at once. Watch memory before scaling concurrency.

Concurrency

Do not use unbounded asyncio.gather(...). Use a semaphore, a queue, a worker pool, and rate limits. A bounded version:

import asyncio
from crawl4ai import AsyncWebCrawler

semaphore = asyncio.Semaphore(5)

async def crawl(crawler, url):
    async with semaphore:
        result = await crawler.arun(url)
        return result.markdown

async def main():
    urls = [
        "https://docs.example.com/install",
        "https://docs.example.com/config",
        "https://docs.example.com/api",
    ]
    async with AsyncWebCrawler() as crawler:
        results = await asyncio.gather(*(crawl(crawler, u) for u in urls))
        print("Crawled", len(results), "pages")

asyncio.run(main())

Docker

The official image runs an HTTP server on port 11235. A basic deployment:

docker run -d \
  -p 11235:11235 \
  --name crawl4ai \
  --shm-size=1g \
  unclecode/crawl4ai:latest

curl http://localhost:11235/health

The --shm-size=1g flag matters: Chromium can run out of shared memory in small containers. Production setups should handle the init process, zombie browser processes, browser cleanup, persistent storage, and resource limits.

Monitoring

At minimum, track page success rate, crawl latency, browser crashes, memory usage, failed URLs, and retry counts. A crawler that silently drops 20% of pages will quietly poison your RAG index.

Crawl4AI Security Considerations

The v0.9.3 release (late August 2026) focused on security hardening, including PDF fetching, SSRF, uncontrolled downloads, XSS, and Docker/service security. I will not expand on CVE details here without an official advisory; the point is that security is now an active project concern.

If your crawler accepts user-supplied URLs, for example POST /crawl with a JSON body, you must defend against SSRF:

POST /crawl
{
  "url": "http://127.0.0.1"
}

Malicious targets include http://127.0.0.1, http://169.254.169.254 (cloud metadata), and internal service names.

A production API should consider:

network isolation
URL allow/deny rules
private IP blocking
authentication
request limits
file size limits
timeouts
hook restrictions

A special warning: if you let users pass JavaScript or hooks, that custom execution capability is a severe security risk. Restrict it or disable it.

What’s New in Crawl4AI 0.9.3?

As of September 2026, 0.9.3 is the current release. The user-relevant themes are:

security hardening (the areas listed above)
PDF crawling improvements
bug fixes and production stability

The release notes describe dozens of fixes. I deliberately did not copy the whole changelog or invent an exact fix count. Verify version numbers and dates against the official release notes before you rely on any specific claim.

Should You Use Crawl4AI?

A practical decision matrix:

Scenario	Recommendation
Personal RAG project	Excellent fit
Local Ollama pipeline	Excellent fit
Internal company knowledge base	Strong fit
AI agent crawling public docs	Strong fit
Small team wanting zero ops	Consider Firecrawl
Massive scraping operation	Requires infrastructure work
Heavy anti-bot targets	Crawl4AI alone may not be enough
Simple static HTML page	BeautifulSoup may be sufficient

I want to counter the hype here: if requests.get() plus BeautifulSoup solves your problem, introducing Playwright and a headless browser adds complexity for no benefit. Not every scraping task needs Crawl4AI.

FAQ

Is Crawl4AI free?

The software is Apache 2.0 and self-hosted, so there is no license fee. Infrastructure, browsers, proxies, and engineering are not free.

Does Crawl4AI require an API key?

Basic crawling needs no third-party crawling API key. LLM-based extraction uses models that may require their own API keys (or a local model).

Can Crawl4AI crawl JavaScript websites?

Yes, through Playwright. That does not mean it automatically bypasses anti-bot systems.

Is Crawl4AI better than Firecrawl?

There is no absolute answer. Crawl4AI gives you control and self-hosting; Firecrawl gives you managed convenience. It depends on your engineering budget.

Can Crawl4AI be used with Ollama?

Yes. You can build local extraction and RAG workflows with a local model.

Is Crawl4AI good for RAG?

It is one of its primary use cases. Chunking, deduplication, metadata, and indexing are still your responsibility.

Does Crawl4AI bypass Cloudflare?

Not a guaranteed yes. Browser automation and stealth help on some sites, but advanced anti-bot protection may still require proxies or specialized infrastructure.

Final Thoughts

In this post, I walked through what Crawl4AI is, how to install it and crawl a first page, why its Markdown matters for RAG, how to use fit content filters, deep crawl, structured extraction, and where it fits with AI agents and Ollama. The key takeaway is that Crawl4AI is compelling not because it introduces a completely new form of web scraping, but because it packages browser automation, content cleaning, Markdown generation, and structured extraction around the needs of modern LLM applications.

For RAG, AI agents, internal knowledge ingestion, and local AI pipelines, it is especially attractive. Once crawling becomes a production service, the hard problems shift from “can I fetch this page?” to reliability, browser lifecycle, proxies, anti-bot, security, scheduling, and observability. That is exactly the boundary where the Crawl4AI-vs-Firecrawl decision happens.

My advice: install Crawl4AI, crawl 3-5 websites you actually need, compare the Markdown quality, measure memory and latency, and only then decide whether self-hosting is worth it. Do not pick a tool based on GitHub stars alone.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!