Why Do LLMs Get Worse Over Time - Understanding AI Quality Degradation
The Problem: “It Just Feels Dumber”
I’ve been using GPT-4 for months. Same prompts, same workflows, same expectations. But somewhere along the way, things started breaking.
Tasks that once worked flawlessly now fail. Responses feel less nuanced. Refusals happen for legitimate requests. And I’m not imagining it - the Reddit threads confirm I’m not alone.
"Most have deteriorated in quality over the last year""Gemini used to be great. It's unusable ATM""GPT was great now unable for anything not PG rated"Only Kimi and Claude seem to have maintained quality. Everyone else? On a downward slope.
So I dug into why this is happening. Turns out there are real technical and business reasons - and none of them are good news for users.
Factor 1: Aggressive Quantization (The Hidden Cost Cutting)
Here’s what providers don’t tell you: when they run out of GPU capacity, they quietly reduce model precision.
┌─────────────┬──────────────┬───────────────────┐│ Precision │ Memory Usage │ Quality Impact │├─────────────┼──────────────┼─────────────────┤│ FP16/BF16 │ 100% │ Baseline ││ INT8 │ 50% │ 2-5% drop ││ INT4 │ 25% │ 10-15% drop │└─────────────┴──────────────┴───────────────────┘The industry is experiencing what some call the “OpenClaw DDOS” - sustained demand overwhelming cloud providers. Z.ai’s stock crashed 23% due to compute shortage.
What this means for you: You pay premium prices for “smart” models but get quantized versions running in the background. An INT4-quantized model has been described as having “the cognitive weight of a fruit fly.”
I noticed this firsthand when my usual prompts started producing noticeably worse results during peak hours. Same model name, different underlying quality.
Factor 2: Over-Aggressive Safety Filters
I tried asking GPT to help me analyze some network security logs. It refused. “I can’t assist with potentially harmful content.”
The logs were from my own servers. Legitimate security analysis. No amount of explaining would change its mind.
This isn’t a bug - it’s a feature gone wrong. Providers have expanded content policies under legal and PR pressure:
RLHF Training │ ▼┌─────────────────┐│ "Safe" responses │ ← Models learn to refuse│ get rewarded │└─────────────────┘ │ ▼┌─────────────────────────────────────┐│ Result: One-size-fits-all filtering ││ - Medical discussions blocked ││ - Security analysis blocked ││ - Creative writing sanitized ││ - Coding help rejected │└─────────────────────────────────────┘GPT has become notorious for refusing “anything not PG rated.” The problem isn’t safety - it’s context-blind safety. A scalpel in a surgeon’s hand is a medical tool; in a child’s hand, it’s dangerous. Current models can’t tell the difference.
Factor 3: Model Drift from Continuous Updates
I built a production system around GPT-4’s output format. It worked great for three months.
Then it didn’t.
Same prompts, different behavior. My parsing broke. The model had changed without any announcement.
This is model drift - providers constantly fine-tune based on:
- User feedback signals
- New training data
- Safety adjustments
- Performance optimizations
┌──────────────┐ ┌──────────────┐ ┌──────────────┐│ Fine-tune │────▶│ Deploy │────▶│ Monitor ││ (new data) │ │ (no notice) │ │ (feedback) │└──────────────┘ └──────────────┘ └──────────────┘ ▲ │ └───────────────────────────────────────┘
Your production system: "..."The result: You’re building on shifting sand. Features that worked yesterday might not work tomorrow. There’s no version stability, no changelog, no deprecation notice.
Factor 4: Hidden Model Switching
Here’s the kicker: some providers admit they “use a combination of different models regardless of what is set in the software.”
Your Request: "Use GPT-4" │ ▼┌────────────────────────────────┐│ Provider's Routing Logic ││ (opaque, undocumented) │├────────────────────────────────┤│ 30% → Full GPT-4 ││ 40% → GPT-4-turbo (quantized) ││ 30% → GPT-3.5 (fallback) │└────────────────────────────────┘ │ ▼ Quality rouletteI discovered this when I noticed my API costs didn’t match my usage. Same endpoint, same “model,” wildly different outputs across sessions.
What this means: You cannot trust model names. You cannot benchmark against a stable baseline. You cannot debug inconsistent outputs because you don’t know what model you actually got.
Why This Matters for Your Projects
I learned these lessons the hard way:
- Production reliability is a myth when underlying models change without notice
- Cost-value mismatch - premium pricing for degraded quality
- Workflow disruption - prompts that worked suddenly fail
- Vendor lock-in becomes a trap - building on one provider is risky
- Competitive disadvantage - your AI tools degrade while competitors adapt
Common Mistakes I’ve Made
❌ Assumed stable quality → Built without monitoring❌ Single provider dependency → No fallback when GPT degraded❌ Dismissed "feels worse" → User feedback was actually accurate❌ No quality benchmarks → Couldn't prove degradation❌ Trusted model names → "GPT-4" isn't always GPT-4What’s Working: Kimi and Claude
The Reddit thread consensus is clear: only Kimi and Claude have maintained quality while others “turn to enshittification.”
Why? I suspect:
- Better capacity planning (not over-committing)
- More conservative safety tuning
- Transparent model versioning
- Less aggressive quantization
But here’s the thing - I don’t know for certain. None of us do. The opacity is part of the problem.
Related Knowledge
Quantization is the process of reducing the precision of neural network weights. Lower precision means less memory and faster inference, but at the cost of quality. INT4 (4-bit integer) quantization can reduce a model’s effective capacity by over 90%.
RLHF (Reinforcement Learning from Human Feedback) is how providers train models to be “helpful and harmless.” When done poorly, it creates models that refuse reasonable requests because they’ve learned refusal is “safe.”
Model drift in traditional ML refers to the degradation of model performance over time due to changing data distributions. In LLMs, it’s intentional - providers update models continuously, often without disclosure.
Reference Links
- Reddit Discussion on LLM Quality Degradation - Community observations and technical analysis
- Claude AI - One of the few providers maintaining quality
- Kimi AI - Another provider noted for consistent quality
Final Words + More Resources
My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me
Here are also the most important links from this article along with some further resources that will help you in this scope:
Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!
Comments