Skip to content
BSWEN
Blog

Why Does AI-Generated Code Fail in Production?

Cowrie
Cowrie
Dev @ Bswen

Problem

Last month, I deployed an AI-generated user authentication module. It worked perfectly in testing. Six weeks later, I needed to add password reset functionality. The AI cheerfully added a bcrypt-based reset flow.

Then users started complaining they couldn’t log in after resetting their passwords.

The problem? The original login code used SHA-256 hashing. The new reset code used bcrypt. These two hash formats are incompatible. The AI generated working code for each feature independently, but created a system that would never work as a whole.

This is the real failure mode of AI-generated code: not syntax errors or runtime bugs, but the maintenance cliff. Code works perfectly until requirements change or bugs appear, and then you discover you never understood the codebase deeply enough to fix it effectively.

What happened?

I had built the authentication system using an AI assistant over several sessions:

Session 1: "Create a user login system with SHA-256 hashing"
         -> Generated login(), register(), verify_password()


Session 2 (2 weeks later): "Add session management with JWT tokens"
         -> Generated create_session(), validate_token(), logout()


Session 3 (4 weeks later): "Add password reset functionality"
         -> Generated reset_password() using bcrypt
         -> AI didn't check existing hash format
         -> Code compiled, tests passed, deployed


Session 4 (6 weeks later): Bug reports start rolling in
         -> Users who reset passwords can't login
         -> Debugging reveals hash format mismatch

Each session produced working code. Each feature worked in isolation. But the system was broken.

Here’s what the code looked like:

auth_original.py
import hashlib


def hash_password(password: str) -> str:
    """Original SHA-256 implementation."""
    return hashlib.sha256(password.encode()).hexdigest()


def verify_password(password: str, hashed: str) -> bool:
    """Verify against SHA-256 hash."""
    return hash_password(password) == hashed


def login(username: str, password: str) -> User:
    user = db.get_user(username)
    if user and verify_password(password, user.password_hash):
        return user
    return None

Then the AI added password reset:

auth_reset.py
import bcrypt


def reset_password(email: str, new_password: str) -> bool:
    """Password reset with bcrypt (added later)."""
    user = db.get_user_by_email(email)
    if not user:
        return False


    # Generate bcrypt hash
    salt = bcrypt.gensalt()
    hashed = bcrypt.hashpw(new_password.encode(), salt)


    user.password_hash = hashed.decode()
    db.save(user)
    return True

Both functions work. Both pass tests. But login() uses SHA-256 comparison while reset_password() stores bcrypt hashes. Users who reset passwords get locked out because the login verification expects SHA-256.

Why does this happen?

The core problem isn’t the AI’s fault. It’s a fundamental gap in how we think about AI-assisted development.

The Understanding Spectrum

Understanding code isn’t binary. There’s a spectrum:

"I know this exists"
       |
       v
"I can read this and follow the logic"
       |
       v
"I can modify this without breaking things"
       |
       v
"I can predict behavior under edge cases I haven't seen"
       |
       v
"I can explain why this design was chosen"

AI tools let you skip directly to “working code” without traversing this spectrum. You ship faster, but you never built the understanding foundation that makes maintenance possible.

The Regression Spiral

When you don’t understand the codebase deeply, every fix becomes a gamble:

Bug reported
     |
     v
Ask AI to fix bug
     |
     v
AI generates fix (without full context)
     |
     v
Fix introduces subtle regression
     |
     v
New bug reported
     |
     v
Ask AI to fix new bug
     |
     v
Another regression introduced
     |
     v
[Loop continues until codebase is unmaintainable]

As one developer put it:

“If you never understood it in the first place you’re just prompting blindly hoping the LLM figures it out, and eventually it starts introducing regressions faster than it fixes things.”

The Specification Gap

The deeper problem: you can’t write good specifications if you don’t understand the problem domain.

When I asked for password reset, I should have specified:

- Must use existing SHA-256 hash format
- Must not break backward compatibility
- Must handle users with old hashes
- Must handle users with new hashes

But I didn’t know to specify these things because I didn’t understand the authentication system well enough. The AI can’t ask the right questions if you don’t know what to ask.

The Maintenance Cliff

The economics of AI-generated code are deceptive:

Development Timeline
                    Initial Development (AI-assisted)
                           |
                           v
Feature velocity: ████████████████████ (fast)
Code quality:     ████████████████████ (looks good)
Understanding:    ████░░░░░░░░░░░░░░░░ (low)
                           |
                           v
              Maintenance Phase (80% of lifecycle)
                           |
                           v
Feature velocity: ████░░░░░░░░░░░░░░░░ (slow)
Bug fix velocity: ██░░░░░░░░░░░░░░░░░░ (painful)
Regression rate:  ████████████████████ (high)

Code spends 80% of its lifecycle in maintenance. AI-optimized codebases front-load gains and back-load costs. You ship fast initially, but pay compound interest on technical debt forever.

How to fix it?

After this experience, I changed how I work with AI coding assistants.

1. Write specifications before generating code

Instead of asking AI to “add password reset,” I now write a spec first:

password_reset_spec.md
# Password Reset Specification


## Context
- Current system uses SHA-256 hashing
- All existing passwords are SHA-256 hashes
- New feature must maintain compatibility


## Requirements
1. Use SHA-256 for new password hashing
2. Verify against existing hash format
3. Migrate old hashes to more secure format (bcrypt) over time
4. Support both formats during transition


## Migration Strategy
1. When user logs in, verify with SHA-256
2. If successful, rehash with bcrypt
3. Store both hashes temporarily
4. After 30 days, remove SHA-256 hashes


## Edge Cases
- User resets password multiple times
- User changes password during migration
- Concurrent password changes

Then I ask the AI to implement against this spec. The AI becomes a tool for implementing my design, not a replacement for thinking.

2. Review every single diff

I treat AI-generated code the same way I treat code from any developer: I review every change before accepting it.

AI generates code
       |
       v
I read every line of the diff
       |
       v
I ask: "Do I understand what this does?"
       |
       v
I ask: "Does this match my mental model of the system?"
       |
       v
I ask: "What could go wrong?"
       |
       v
Accept or request changes

If I can’t explain what the code does and why it’s correct, I don’t merge it. This forces me to build understanding even when using AI assistance.

3. Understand before asking

Before asking AI to modify existing code, I spend time reading it:

Terminal window
# Read the files I'm about to modify
cat auth.py
cat user.py
cat session.py


# Check dependencies
grep -r "hash_password" .
grep -r "verify_password" .


# Understand the data flow
# Where does user.password_hash get written?
# Where does it get read?
# What format is expected?

Only after I understand the current state do I ask the AI for changes. This catches issues like “the AI suggested bcrypt but we use SHA-256.”

4. Test the integration, not just the feature

My original tests verified that password reset worked in isolation. They didn’t verify that password reset integrated correctly with login.

test_auth_integration.py
def test_password_reset_integration():
    """Test that reset password allows subsequent login."""
    # Setup
    user = create_user("[email protected]", "oldpassword")


    # Reset password
    reset_password("[email protected]", "newpassword")


    # Critical integration test: can user login with new password?
    logged_in = login("[email protected]", "newpassword")
    assert logged_in is not None, "User should login after password reset"


    # Verify old password doesn't work
    old_login = login("[email protected]", "oldpassword")
    assert old_login is None, "Old password should not work after reset"

This test would have caught the SHA-256 / bcrypt mismatch immediately.

The real solution: Engineering, not prompting

The solution wasn’t better prompts. It was writing detailed specs and reviewing every single diff. Which is basically just… engineering.

AI coding assistants multiply your understanding, they don’t replace it. If you understand the system deeply, AI helps you implement faster. If you don’t understand the system, AI helps you dig a deeper hole faster.

The gap between prototype and production isn’t in the initial working code. It’s in the ability to adapt that code when requirements change. AI helps you ship faster, but you still need to understand what you shipped.

The cost of quick fixes

Every quick fix you accept without understanding adds to your technical debt. The interest rate on that debt compounds during maintenance. What looks like a 10-minute time savings today becomes a 4-hour debugging session next month.

Understanding vs. shipping

There’s a balance between “understand everything before coding” and “ship first, understand later.” AI tools push you toward the latter extreme. The healthy middle ground: understand enough to catch obvious problems, then learn deeply when issues arise.

The Dunning-Kruger effect in AI-assisted development

AI tools create a new variant of the Dunning-Kruger effect: you feel productive because code works, but your actual understanding of the system may be decreasing. The gap between “it works” and “I understand why it works” grows wider with each AI-generated feature.

Summary

In this post, I showed how AI-generated code can pass all tests and deploy successfully, yet fail catastrophically when requirements change. The failure mode isn’t bugs or syntax errors, it’s the maintenance cliff: you never built the understanding foundation needed to maintain what you shipped.

The key points:

  1. AI tools let you skip the understanding spectrum, jumping directly to working code
  2. Without deep understanding, every fix becomes a regression risk
  3. Code spends 80% of its lifecycle in maintenance, where AI-optimized codebases suffer most
  4. The solution is engineering fundamentals: specifications, diff reviews, integration tests
  5. AI multiplies your understanding, it doesn’t replace it

The real skill in AI-assisted development isn’t prompting, it’s knowing when to slow down and understand what you’re building.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags: