How Do I Set Up Automated Code Experiments with Claude Code? A Step-by-Step Guide
Problem
I had been manually iterating on code optimizations for weeks. Each attempt required:
- Think about what might improve performance
- Make changes
- Run the benchmark
- Check results
- Decide to keep or revert
- Try something else
This loop consumed hours. Worse, I often got stuck on one approach, missing better solutions elsewhere. Then I discovered that Claude Code could run this entire loop autonomously.
What Automated Code Experiments Look Like
A developer on GitHub shared what happened when they used the ResearcherSkill:
> ### Experiment 5 — Parallelize independent test suites> **Branch:** research/faster-tests · **Parent:** #3 · **Type:** real>> **Hypothesis:** Unit and integration suites don't share state. Running them in parallel should cut total time.> **Changes:** split test config into two parallel jobs in `test.config.ts`> **Result:** 38s (was 94s baseline, 52s best) — **new best**> **Status:** keep>> **Insight:** Most of the remaining time is in integration tests. Unit tests finish in 6s. Focus on integration from here.The agent ran 5 experiments autonomously. It kept improvements, discarded failures, and learned from its own history. The test suite went from 94s to 38s—a 60% improvement.
The Single File Architecture
Here’s what surprised me: the entire experiment automation lives in one Markdown file.
researcher.md ← One skill file, ~200 linesDrop this file into Claude Code, and the agent gains the ability to:
- Design experiments
- Test hypotheses
- Discard failures
- Keep improvements
- Resume interrupted sessions
No configuration files. No scripts. Just one skill that teaches Claude Code how to be a scientist.
How It Works: The Interview-Driven Setup
When I invoke the skill, Claude Code doesn’t start hacking immediately. It interviews me first:
| Question | My Answer ||-----------------------------------|----------------------------------------|| What's the goal? | Reduce p99 API latency || How do we measure it? | `./bench.sh | grep p99` outputs ms || Lower is better? | Yes || What files can I touch? | `src/api/` and `src/db/` || What's off-limits? | Don't change public API contract || How long can one experiment take? | 5 minutes max || When do we stop? | p99 < 50ms, or after 40 experiments |The agent repeats this back. I confirm. Then it builds the lab.
The Lab: Git Branch + .lab Directory
The agent creates two things:
project/├── src/ ← Code (git manages this)└── .lab/ ← Experiment history (gitignored) ├── config.md ← What we agreed on ├── results.tsv ← Every experiment, one row ├── log.md ← Narrative: what, why, what happened ├── branches.md ← Branch registry (if agent forks) └── parking-lot.md ← Ideas for laterGit manages code. .lab/ manages knowledge. They’re independent because the agent will revert code constantly, but the experiment log must survive.
The Commit-Revert Pattern
This is the safety mechanism. Before every experiment run:
git commit -m "experiment: batch order fetches in getUserOrders"This commit is the anchor. If the experiment fails, the agent reverts:
git reset --hard HEAD~1No failed experiments pollute the git history. Only improvements survive.
The Experiment Loop
The skill implements a Think-Test-Reflect loop:
┌───────────────────────────────┐│ │▼ │┌─────────┐ ││ THINK │ Read history, ││ │ analyze, form ││ │ hypothesis │└────┬────┘ │ │ │ ▼ │┌─────────┐ ││ TEST │ Commit, run, ││ │ measure ││ │ Keep or revert │└────┬────┘ │ │ │ ▼ │┌─────────┐ ││ REFLECT │ Log result, ││ │ check convergence ││ │ signals │└────┬────┘ │ │ │ └───────────────────────────┘Think Phase
The agent reads experiment history. What worked? What didn’t? Are there patterns? It forms a hypothesis:
"The N+1 query in getUserOrders is probably the bottleneck.If I batch the order fetches, p99 should drop."Test Phase
The agent makes changes. Before running anything, it commits. Then runs the benchmark.
| Outcome | Action ||----------------------------|--------------------------------------|| Metric improved | Keep. Advance the branch. || Metric equal but simpler | Keep. Simplification win. || Metric equal or worse | Discard. git reset --hard HEAD~1 || Informative but no change | Interesting. Agent decides. |Reflect Phase
The agent logs the result:
## Experiment 1 — Batch order fetches in getUserOrders
**Branch:** research/reduce-p99**Type:** real | **Parent:** #0
**Hypothesis:** N+1 query is the bottleneck, batching should reduce p99**Changes:** replaced loop with batch query in getUserOrders**Result:** p99 = 118ms (was 142ms baseline) — new best**Status:** keep
**Insight:** The N+1 was real. 17% improvement from one change.Check if similar pattern exists in getOrderDetails.Thought Experiments: The Smart Shortcuts
Not every hypothesis needs a benchmark run. If the agent can reason about whether an approach will work, it should:
| Real Experiment | Thought Experiment ||-------------------------|----------------------------|| Changes code | No code changes || Git commit + run | Analysis only || Produces a metric | Produces a conclusion || Logged as keep/discard | Logged as thought |An agent that thinks first, then tests selectively, converges faster. Ten minutes of good analysis can prevent five wasted experiments.
Convergence Detection: When to Pivot
The skill prevents grinding the same idea forever. After every experiment, it checks signals:
| Signal | Meaning | Action ||---------------------------------|--------------------------|---------------------|| 5+ discards in a row | Current approach exhausted| Pivot completely || Metric plateau (<0.5% over 5) | Small tweaks done | Go radical || Same code area modified 3+ times| Over-optimizing one spot | Look elsewhere || Alternating keep/discard | Conflating variables | Isolate them || 2+ timeouts in a row | Approach too expensive | Scale down |These aren’t hard rules. They’re signals. The agent reads them, considers history, and decides.
Session Resume: State Persistence in .lab/
I interrupted an experiment run after 15 iterations. When I restarted:
Agent: "I see .lab/ exists with 15 experiments logged.Last result: p99 = 94ms. Target: 50ms.Continuing from experiment #16."The .lab/ directory persists all state. The agent reads it and continues exactly where it left off.
Installing the ResearcherSkill
Installation is straightforward:
# Method 1: Clone the repogit clone https://github.com/krzysztofdudek/ResearcherSkill.git
# Method 2: Download just the skill filecurl -o ~/.claude/skills/researcher/SKILL.md \ https://raw.githubusercontent.com/krzysztofdudek/ResearcherSkill/main/researcher.mdThe skill activates when I ask Claude Code to optimize something measurable.
Running Your First Experiment
I invoked it on a slow test suite:
Me: "I want to optimize the test suite. npm test takes 94 seconds."
Claude: [Interviews me about goals, metrics, constraints]Claude: [Creates .lab/ directory and research/faster-tests branch]Claude: [Runs baseline: 94s]Claude: [Experiment 1: Remove redundant setup/teardown → 71s → keep]Claude: [Experiment 2: Shared test fixtures → 74s → discard, revert]Claude: [Experiment 3: Mock external HTTP calls → 52s → keep]...I walked away. Came back two hours later. The log showed 30 experiments. Best result: 38s.
What This Works On
Anything where you can say “run this” and “check this number”:
| Domain | Run Command | Measure Command ||-------------------|----------------------------------|-----------------------|| API performance | wrk -t4 -c100 http://localhost | grep p99 || Test suite speed | npm test | time output || Bundle size | npm run build | stat -f%z dist/main.js|| Parser accuracy | ./run-tests.sh | grep "pass rate" |It also works with qualitative metrics. If you can define a rubric, the agent scores against it.
Customizing for Your Use Case
The skill is designed for customization. In the interview phase, I define:
- Goal: What you want to improve- Metric: How you measure success (command + parsing)- Direction: Higher or lower is better- Scope: Which files the agent can touch- Constraints: What's off-limits- Timeout: Max time per experiment- Target: When to stop (metric value, count limit, or never)For API latency, I set strict constraints on scope. For test optimization, I gave broader access.
Best Practices I Learned
After running several experiments:
1. Start with clear, measurable goals2. Define narrow scope to prevent unintended changes3. Set realistic timeout (too short = incomplete experiments)4. Run baseline first (experiment #0 establishes "before")5. Check .lab/log.md periodically to see what's being tried6. Let it run overnight for complex problems7. Review the summary.md before accepting changesThe agent works best when:
- The problem is clear
- The metric is fast to compute
- The search space is wide enough that trying 20 things beats debating which one to try
The Hidden Benefit: Learning from Failures
The .lab/log.md contains failures that git history doesn’t. This is valuable:
Experiment #2: Shared test fixtures → discardReason: Fixture reuse created state pollution between tests.Insight: Tests need isolated state, not shared fixtures.I learned what doesn’t work. The next time I face a similar problem, I skip that approach.
What I Got After 30 Experiments
My project/├── src/ ← Code with only improvements├── .lab/│ ├── config.md ← What we agreed on│ ├── results.tsv ← Spreadsheet-ready data│ ├── log.md ← Reads like a research journal│ ├── branches.md ← The exploration tree│ ├── parking-lot.md ← Ideas for next time│ └── summary.md ← HighlightsThe code changes are real commits. The history is clean because failures got reverted. Only improvements survive in git. The full story lives in .lab/.
Related Knowledge: The Autoresearch Connection
This skill generalizes Andrej Karpathy’s autoresearch concept beyond ML training:
| Aspect | Autoresearch | ResearcherSkill ||------------------|-------------------|--------------------|| Domain | ML training | Any measurable task|| Metric | val_bpb | User-defined || Setup | train.py wired | Interview-driven || Branching | Linear | Non-linear || Thought experiments| No | Yes || Session resume | No | Yes |The loop is universal: try → measure → keep or discard → repeat. This skill makes that loop available for any codebase.
Summary
In this post, I showed how to set up automated code experiments with Claude Code using the ResearcherSkill. The key point is that a single Markdown skill file transforms your AI coding agent into an autonomous scientist that runs 30+ experiments overnight. The interview-driven setup captures your goals and constraints. The commit-revert pattern ensures only improvements survive. The .lab/ directory persists experiment history across failures and interruptions. Setting up automated experiments transforms development from manual iteration to AI-driven continuous improvement: save time, reduce risk, learn faster, scale improvement, document progress.
Final Words + More Resources
My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me
Here are also the most important links from this article along with some further resources that will help you in this scope:
Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!
Comments