Skip to content
BSWEN
Blog

Heretic vs Abliterated LLM Models: Key Differences Explained

Cowrie
Cowrie
Dev @ Bswen

When I started exploring uncensored LLM models on HuggingFace, I kept seeing two terms: “Heretic” and “Abliterated.” At first, I thought they were just different names for the same thing. I was wrong. After digging into the technical details, I found they use completely different approaches to remove model censorship.

The Core Difference

Abliterated models use mechanistic interpretability - they surgically remove refusal directions from model weights without retraining. Heretic models use fine-tuning with Bayesian optimization - they actually retrain the model weights.

This fundamental difference affects everything: computational cost, reversibility, and model quality. Let me show you what I learned.

Quick Comparison Table

I created this comparison to help you understand the key differences at a glance:

Heretic vs Abliterated Models Comparison
| Aspect              | Abliterated Models                  | Heretic Models                         |
|---------------------|-------------------------------------|----------------------------------------|
| Core Approach       | Weight projection (no training)     | Fine-tuning with Bayesian optimization |
| Requires Training   | No                                  | Yes                                    |
| Computational Cost  | Low (inference-time only)           | High (needs GPU training)              |
| Reversible          | Yes (with steering vectors)         | No (permanent weight changes)          |
| Model Availability  | 4,967 models on HuggingFace         | 2,164 models on HuggingFace            |
| Tools Required      | OBLITERATUS, TransformerLens        | Custom fine-tuning scripts, Optuna     |
| Technical Knowledge | High (model internals)              | Medium (fine-tuning setup)             |
| Speed to Deploy     | Fast (apply to existing model)      | Slow (requires training)               |

What Are Abliterated Models?

Abliterated models represent a surgical approach to removing censorship. The technique identifies and removes specific neural pathways responsible for refusal behavior.

How Abliteration Works

I found the process fascinating. Here’s what happens:

  1. Refusal Direction Extraction: The system uses mathematical techniques like SVD decomposition, PCA, and mean-difference analysis to find the exact “refusal directions” in the model’s weights
  2. Surgical Removal: It projects out these refusal directions while preserving the model’s other capabilities
  3. No Retraining: This happens directly on pre-trained weights - no GPU training required

Key Characteristics

From my research, I identified these important features:

  • Reversible: You can use steering vectors for temporary modification
  • Computationally Efficient: No training required - just weight manipulation
  • Precision-Focused: Targets specific refusal mechanisms only
  • Architecture-Agnostic: Works with any HuggingFace transformer model

I found these frequently downloaded abliterated models:

  • Huihui-Qwen3.5-35B-A3B-abliterated (24.5k downloads)
  • lukey03/Qwen3.5-9B-abliterated
  • Over 4,967 abliterated models available on HuggingFace

What Are Heretic Models?

Heretic models take a retraining approach. They use fine-tuning with Bayesian optimization to create uncensored models through actual weight modification.

How Heretic Works

The process is more involved than abliteration:

  1. Bayesian-Optimized Kernel Methods: Uses Optuna TPE search with 7 global parameters
  2. Parametric Kernel Optimization: Applies Bell-curve layer weighting
  3. Activation Winsorization: Prevents outlier-dominated directions
  4. Fine-Tuning Process: Requires actual GPU compute for training
  5. Heretic Scale: Ranges from “tainted heresy” to “total heresy”

Key Characteristics

These are the features I identified:

  • Permanent Changes: Weights are permanently modified through training
  • Resource-Intensive: Requires significant GPU compute
  • Comprehensive: Modifies model behavior more deeply
  • Often Combined: Frequently paired with NEO-Imatrix, distillation, or other techniques

Notable heretic models I found:

  • DavidAU/Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED
  • llmfan46/Qwen3.5-35B-A3B-heretic-v2
  • Over 2,164 heretic models on HuggingFace

When to Choose Abliterated Models

Based on my analysis, I recommend abliterated models when:

  1. You want reversibility: Steering vectors let you temporarily modify behavior
  2. Computational resources are limited: No training required
  3. You need precise control: Target specific refusal behaviors
  4. You’re doing research: Easy to test different removal strategies
  5. You need quick deployment: Apply to existing models immediately

When to Choose Heretic Models

I found heretic models work better when:

  1. You need comprehensive uncensoring: Deeper behavioral modification
  2. You have GPU resources: Training infrastructure available
  3. You want community-tested models: Many pre-trained options exist
  4. You’re combining enhancements: Adding distillation, reasoning capabilities
  5. Permanent modification is acceptable: No need to revert

Technical Implementation Comparison

Let me show you the implementation difference:

Abliteration Example

Abliteration Implementation Process
Step 1: Load base model (e.g., Llama-3.1-8B-Instruct)
Step 2: Extract refusal directions using SVD/PCA
Step 3: Project out refusal directions from weights
Step 4: Save modified model (no training required)


Time: Minutes to hours
Hardware: CPU sufficient for small models
Tools: OBLITERATUS, TransformerLens

Heretic Example

Heretic Implementation Process
Step 1: Prepare uncensored training dataset
Step 2: Configure Optuna optimization parameters
Step 3: Set up GPU training infrastructure
Step 4: Run fine-tuning with layer-weighted kernels
Step 5: Evaluate refusal rate and capabilities


Time: Hours to days
Hardware: GPU required (significant VRAM)
Tools: Custom fine-tuning scripts, Optuna

Performance Trade-offs

I analyzed the pros and cons of each approach:

Abliterated Models

Abliterated Models Performance Analysis
Pros:
- Fast to apply
- Reversible with steering vectors
- Works with any architecture
- Preserves base model capabilities
- Low computational cost


Cons:
- May leave residual refusal
- Requires understanding model internals
- Quality depends on extraction method
- Less comprehensive modification

Heretic Models

Heretic Models Performance Analysis
Pros:
- Comprehensive behavioral modification
- Can add capabilities during training
- Community-tested models available
- Deep integration with model weights


Cons:
- Resource-intensive (GPU required)
- Permanent changes (no reversal)
- Training quality dependent
- Longer implementation time

Common Misconceptions

I encountered several myths while researching:

Myth 1: “Both methods are the same”

Reality: They use fundamentally different approaches. Abliteration modifies inference behavior through weight projection; Heretic retrains the model through fine-tuning.

Myth 2: “Abliteration always preserves model quality”

Reality: While abliteration aims to preserve capabilities, aggressive removal can impact model coherence. Quality depends on the specific method used.

Myth 3: “Heretic models are always better”

Reality: Quality varies significantly. Some abliterated models outperform poorly-trained heretic variants. The base model quality matters more than the uncensoring method.

Practical Decision Framework

I created this decision matrix to help you choose:

Decision Matrix for Choosing Between Methods
Your Situation                          | Recommended Approach
----------------------------------------|----------------------
Limited GPU resources                   | Abliterated
Need to test multiple models quickly    | Abliterated
Want reversible modifications          | Abliterated
Researching refusal mechanisms         | Abliterated
----------------------------------------|----------------------
Have GPU training infrastructure       | Heretic
Need comprehensive uncensoring         | Heretic
Want to add capabilities during process| Heretic
Permanent model modification acceptable| Heretic

Getting Started

For Abliterated Models

  1. Visit HuggingFace and search for “abliterated”
  2. Download a model like Huihui-Qwen3.5-35B-A3B-abliterated
  3. Use with your preferred inference engine (Ollama, llama.cpp, etc.)
  4. No additional processing needed

For Heretic Models

  1. Visit HuggingFace and search for “heretic”
  2. Download a pre-trained heretic model
  3. Use immediately with inference engines
  4. Or create your own with fine-tuning tools

Both approaches are evolving rapidly:

  • Abliteration: The OBLITERATUS project adds 15 analysis modules, analysis-informed pipelines, and community telemetry
  • Heretic: New variations like “ultra-heretic” and combinations with reasoning models emerge regularly

The field is moving toward hybrid approaches that combine abliteration’s precision with fine-tuning’s comprehensiveness.

My Recommendation

After comparing both approaches, here’s what I recommend:

  • Start with abliterated models if you’re new to uncensored LLMs - they’re faster to test and require less infrastructure
  • Move to heretic models if you need deeper modification and have GPU resources available
  • Test both on your specific use case - performance varies by base model and application

Both methods produce high-quality uncensored models. The best choice depends on your resources, timeline, and specific requirements.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags: