What Results Did 700 Autoresearch Experiments Achieve Overnight? (11% Speedup Revealed)

Mar 30, 2026

I’ve been following Andrej Karpathy’s autoresearch project since he announced it. The idea was simple: run 700 automated experiments overnight on a single H100 GPU and see what happens. The results? An 11% speedup on the “Time to GPT-2” benchmark.

But what actually happened during those 48 hours? Let me break it down.

The Problem: Human Iteration is Slow

Karpathy had been manually tuning his training system for months. Even as an expert, he faced a fundamental bottleneck:

Human Limitation	GPU Reality
Needs 8 hours sleep	Runs 24/7
One hypothesis at a time	Parallel experiments possible
Mental fatigue accumulates	No fatigue, ever
Biased by recent failures	Objectively measures results

The core insight isn’t about AI intelligence—it’s about iteration speed. What took months of human tuning got compressed into 48 hours of automated experimentation.

The Results: By the Numbers

Here’s what the system achieved:

Metric	Value
Total experiments	~700
Duration	~48 hours
Hardware	Single H100 GPU
Successful improvements	20 code changes
Benchmark improvement	2.02h → 1.80h
Improvement percentage	11% faster
Success rate	~2.9% (20/700)

Hour 0:   Benchmark: 2.02h  |████████████████████| Baseline (months of manual tuning)
Hour 12:  Benchmark: 1.98h  |██████████████████▓| First improvements appear
Hour 24:  Benchmark: 1.88h  |█████████████████▓▓▓| QKNorm bug found and fixed
Hour 48:  Benchmark: 1.80h  |████████████████▓▓▓▓| Final: 11% improvement

The Loop: How Autoresearch Works

The system implements a closed feedback loop:

┌──────────────────────────────────────────────────────────────────┐
│                                                                  │
│   ┌─────────────┐    ┌─────────────┐    ┌─────────────┐        │
│   │  Generate   │───▶│   Modify    │───▶│    Run      │        │
│   │ Hypothesis  │    │    Code     │    │ Experiment  │        │
│   └─────────────┘    └─────────────┘    └─────────────┘        │
│          ▲                                     │                │
│          │                                     │                │
│          │                                     ▼                │
│   ┌─────────────┐                       ┌─────────────┐        │
│   │   Next      │◀──────────────────────│  Evaluate   │        │
│   │ Hypothesis  │                       │   Result    │        │
│   └─────────────┘                       └─────────────┘        │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

Each step is automated:

Hypothesis Generation: LLM proposes optimization ideas based on current code and previous results
Code Modification: Changes are applied in isolation
Experiment Run: Training runs on H100 with objective metrics
Result Evaluation: Validation loss, training speed measured
Next Hypothesis: Successful changes accumulate; failures are discarded

The QKNorm Bug: A Concrete Example

One of the most interesting discoveries was a QKNorm bug. The system found that a missing scaler multiplier was making attention too diffuse:

# The issue: Missing scaler in attention normalization
# Effect: Attention scores were becoming too diffuse,
#         spreading focus across too many tokens
# Fix: Properly scale the normalization

# Before (problematic):
attention = query @ key.T / sqrt(d_k)

# After (with proper scaler):
scaler = 1.0 / sqrt(d_k)
attention = query @ key.T * scaler  # Direct multiplication, no division

The system discovered this automatically through:

Proposing changes to normalization layers
Running experiments with different scaling approaches
Observing consistent improvement in one variant
Keeping the fix as part of the 20 successful changes

Why 11% Matters

An 11% improvement might not sound revolutionary. But consider the context:

Initial optimization:     100% → 50%  (easy wins)
Mid-stage optimization:   50%  → 30%  (requires expertise)
Late-stage optimization:  30% → 25%  (very hard)
Current optimization:      25% → 22%  (11% relative improvement)
                                   ↑
                          Karpathy had already spent months here

The significance:

Diminishing returns territory: Karpathy is an expert who had already optimized extensively
Transferability: All 20 improvements transfer to larger models
Compound effect: Each change is additive—they stack
Speed begets speed: Faster experiments = more discoveries = more speed

The Improvement Stack

Here’s what the 20 successful changes look like:

┌─────────────────────────────────────┐
│ 20 Successful Code Changes          │
├─────────────────────────────────────┤
│ • QKNorm scaler fix                 │
│ • Learning rate schedule tweaks     │
│ • Initialization improvements       │
│ • Gradient clipping adjustments     │
│ • Weight decay optimizations        │
│ • [15 other additive improvements]  │
├─────────────────────────────────────┤
│ All transfer to larger models       │
│ All independently measurable         │
│ All discovered without human input   │
└─────────────────────────────────────┘
          ↓
    11% speedup in 48 hours
    vs. months of manual tuning

What This is NOT

The Reddit discussion raised important points about limitations:

What It Is NOT	What It Actually IS
Self-improving AI	Automation of human hypothesis-testing
Novel idea generation	Exploration within known optimization space
Path to hard takeoff	Tool for accelerating research
AGI precursor	LLM-based code search + experiment runner

As Reddit user do-un-to noted: “Not self-improving because it isn’t refining its own ideation engine.” The system doesn’t improve how it generates ideas—it just tests more ideas faster.

The real test, per CryptographerUnable8: “The real test isn’t throughput, it’s surprise.” Can the system generate genuinely novel ideas, or is it bounded by what’s in its training data?

The Math: Why High Failure Rate is Fine

# The economics of automated experimentation
experiments = 700
hours = 48
successful_changes = 20

experiments_per_hour = 700 / 48  # ~14.6 experiments/hour
success_rate = 20 / 700          # ~2.9%

# Key insight: High failure rate is acceptable because:
# 1. GPU time was already budgeted
# 2. Failed experiments cost zero human time
# 3. Only successful changes need human review
# 4. 20 improvements in 48 hours >> 2-3 improvements in months

This is the paradigm shift: failed experiments become essentially free.

What I Learned

Running through the results, three things stood out to me:

Iteration speed is the bottleneck, not intelligence: The system isn’t smarter than Karpathy—it’s just faster at trying things.
Boring improvements compound: No single change was revolutionary. But 20 small improvements added up to 11%.
The surprise factor matters: The system found things Karpathy hadn’t tried, but only because it tried more things, not because it had better ideas.

Should You Use This?

For ML practitioners:

Use Case	Recommended?
Training pipeline optimization	Yes—well-defined objective function
Architecture search	Maybe—requires more compute
Novel research directions	No—bounded by training data
Hyperparameter tuning	Yes—automated grid search alternative

The sweet spot is any problem with:

Clear objective metrics
Many possible configurations
Cheap-to-run experiments
Existing codebase to optimize

Getting Started

If you want to try autoresearch:

# Clone the repository
git clone https://github.com/karpathy/autoresearch.git
cd autoresearch

# You'll need:
# - A GPU (H100 ideal, but smaller works with longer times)
# - A well-defined benchmark to optimize
# - Clear experiment isolation (no state bleeding between runs)

The key is having a reproducible experiment that can run in isolation—each experiment starts fresh.

The 11% speedup isn’t the story. The story is that 700 experiments in 48 hours found 20 genuine improvements that months of human work hadn’t. The bottleneck in ML research isn’t ideas—it’s iteration speed.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 karpathy/autoresearch - GitHub
👨‍💻 Reddit Discussion on Autoresearch Results
👨‍💻 Andrej Karpathy's Website

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!