Skip to content
BSWEN
Blog

How to Implement Feedback Loops for AI Code Generation: A Complete Guide

Cowrie
Cowrie
Dev @ Bswen

I spent weeks watching my codebase slowly decay. Every time I asked Claude to add a feature, the code worked—but something felt wrong. Functions grew longer. Coupling increased. Complexity metrics I didn’t even know existed were silently accumulating.

The breaking point came when I asked Claude to refactor a module, and it confidently returned code that was more complex than the original. I had no way to prove it was worse, so I merged it anyway.

That’s when I realized: AI code generation without feedback is a one-way transformation.

The Core Problem

Here’s what happens in traditional AI-assisted development:

  1. You prompt the AI
  2. AI generates code
  3. You manually review (or skip review)
  4. Code gets committed

There’s no measurement step. No verification that the code actually improved. No way for the AI to learn from its mistakes.

graph LR
    A[Prompt] --> B[Generate]
    B --> C[Commit]
    C --> D[???]


    style D fill:#ffcccc

The question mark at the end represents the blind spot. You have no idea if your codebase is getting better or worse over time.

My Failed Attempts

Attempt 1: Manual Code Reviews

I tried reviewing every AI-generated change myself. This worked for a while, but I quickly became the bottleneck. And honestly, can you accurately assess cyclomatic complexity by eye?

Attempt 2: Static Analysis Tools

I set up ESLint and Prettier with strict rules. This caught syntax issues but missed architectural problems. A function could pass linting while being a maintainability nightmare.

Attempt 3: Test Coverage

I required 80% test coverage for all PRs. But coverage doesn’t measure code quality—it measures code execution. You can have 100% coverage of spaghetti code.

None of these created a closed loop. The AI never received feedback on whether its changes improved or degraded the codebase.

The Solution: Closed-Loop Architecture

The fix came from an unexpected place: Model Context Protocol (MCP) servers.

MCP allows external tools to expose structured data to AI agents. Combined with tree-sitter for code parsing, I could finally build what I needed:

graph TD
    A[Codebase State] --> B[Measure Metrics]
    B --> C[MCP Server Analysis]
    C --> D[Quality Score]
    D --> E{Target Met?}
    E -->|No| F[Identify Bottleneck]
    F --> G[AI Improvement]
    G --> H[Apply Changes]
    H --> I[Run Tests]
    I --> J{Tests Pass?}
    J -->|Yes| B
    J -->|No| K[Rollback]
    K --> F
    E -->|Yes| L[Commit]


    style B fill:#e1f5ff
    style D fill:#fff4e1
    style G fill:#e8f5e9
    style I fill:#fce4ec

The key insight: The AI sees the score, improves the code, and the score goes up.

How I Built It

Phase 1: Measurement Infrastructure

First, I needed quantifiable metrics. Tree-sitter became my parser of choice because it handles multiple languages and produces concrete syntax trees.

# Simplified metric collection
import tree_sitter_python as tspython
from tree_sitter import Language, Parser


PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)


def measure_function_complexity(source_code: bytes) -> dict:
    tree = parser.parse(source_code)
    # Walk the tree, count decision points
    # Return cyclomatic complexity, nesting depth, etc.

The metrics I track:

  • Cyclomatic complexity: Number of independent paths through code
  • Nesting depth: Maximum level of nested control structures
  • Function length: Lines of code per function
  • Coupling score: How many other modules does this depend on?
  • Cohesion score: Do the functions in this module belong together?

Phase 2: MCP Server Interface

The MCP server exposes these metrics to Claude Code (or Cursor) as callable tools.

# MCP server endpoint (simplified)
@server.tool
def analyze_codebase(path: str) -> dict:
    """Analyze codebase and return quality metrics."""
    metrics = collect_metrics(path)
    score = calculate_quality_score(metrics)
    bottlenecks = identify_issues(metrics)
    return {
        "score": score,
        "metrics": metrics,
        "bottlenecks": bottlenecks,
        "suggestions": prioritize_improvements(bottlenecks)
    }

Now when I ask Claude to improve code, it can call analyze_codebase, see that function processOrder has complexity 47, and target that specific bottleneck.

Phase 3: The Feedback Loop

Here’s where the magic happens. The loop works like this:

  1. Measure: Run analysis, get baseline score (let’s say 72/100)
  2. Target: AI identifies the worst bottleneck (complexity 47 in processOrder)
  3. Improve: AI refactors processOrder into smaller functions
  4. Re-measure: Run analysis again
  5. Verify: Score improved to 78/100? Keep changes. Score dropped? Rollback.
# Pseudo-code for the loop
def feedback_loop(path: str, target_score: int = 85):
    current = analyze_codebase(path)
    iterations = 0


    while current.score < target_score and iterations < 10:
        # AI picks the worst bottleneck
        bottleneck = current.bottlenecks[0]


        # AI generates improvement
        changes = ai_suggest_fix(bottleneck, current)


        # Apply changes
        apply_changes(path, changes)


        # Re-measure
        new_metrics = analyze_codebase(path)


        # Verify improvement
        if new_metrics.score > current.score:
            commit_changes(changes)
            current = new_metrics
        else:
            rollback_changes(changes)


        iterations += 1

Critical Lessons Learned

Lesson 1: Tests Must Come First

I learned this the hard way. Without comprehensive test coverage, the AI will happily refactor code into elegant, non-functional versions.

“I wouldn’t trust it unless there was already complete/comprehensive test coverage, because otherwise claude will just make the code higher quality while eliminating functionality.”

This Reddit comment became my mantra. Now I require 80%+ coverage before enabling autonomous improvement.

Lesson 2: Context Matters

A complexity score of 15 is terrible for a utility function but acceptable for a complex business rule. Raw metrics without context lead to bad decisions.

I added a context layer:

def contextualize_metrics(metrics: dict, file_type: str) -> dict:
    """Adjust thresholds based on file type and purpose."""
    if file_type == "business_logic":
        # Allow higher complexity
        metrics["complexity_threshold"] = 15
    elif file_type == "utility":
        # Stricter requirements
        metrics["complexity_threshold"] = 8
    return metrics

Lesson 3: Incremental Targets Win

Setting “perfect code” as the target caused the AI to over-engineer solutions. Starting with incremental goals—improve by 5 points—produced better results.

Lesson 4: Human Checkpoints for Major Changes

Fully autonomous loops work for minor refactoring. But for major structural changes—extracting modules, changing interfaces—I still want human review.

The Results

After implementing this system:

  • Complexity scores dropped 34% across the codebase over 3 months
  • Test coverage increased from 67% to 89% (coverage became a metric)
  • I can now oversee 3-4 parallel AI refactoring sessions with confidence
  • Regressions caught early: The feedback loop caught 12 potential bugs before merge

Most importantly, I have quantifiable proof that my codebase is improving. No more gut feelings about code quality.

Implementation Checklist

If you want to build this yourself:

Start Here:

  1. Ensure 80%+ test coverage on your codebase
  2. Install tree-sitter and configure parsers for your languages
  3. Define your quality metrics and thresholds

Build the MCP Server: 4. Create endpoints for metric collection 5. Implement scoring algorithms 6. Add differential reporting (before/after comparisons)

Configure the Loop: 7. Set iteration limits (prevent infinite loops) 8. Configure test requirements (must pass before committing) 9. Enable rollback triggers (auto-revert on score decrease)

Monitor and Tune: 10. Track improvement velocity 11. Adjust scoring weights based on results 12. Add new metrics as you learn what matters

The Bigger Picture

This approach transforms AI from a “code generator” into a “code improver.” Instead of one-way transformations, you get continuous refinement.

The feedback loop creates a virtuous cycle:

  • AI makes changes → Metrics capture impact → AI sees results → AI adjusts approach → Better changes

This is how you scale AI-assisted development without sacrificing quality. Not by blindly trusting AI output, but by building systems that verify and improve it automatically.

The future isn’t AI writing code in isolation—it’s AI improving code within guardrails that we define and measure.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags: