What to Do After Installing a Compromised Python Package

Mar 25, 2026

Purpose

This post demonstrates the remediation steps to take after discovering you’ve installed a compromised Python package, using the LiteLLM supply chain attack as a real-world example.

The Problem

I ran pip install litellm==1.82.8 and thought nothing of it. Then I saw the security advisory:

SECURITY ALERT: LiteLLM versions 1.52.26 - 1.63.8 compromised
Malicious code harvested: SSH keys, .env files, cloud credentials,
Kubernetes configs, database passwords, .gitconfig, shell history

My stomach dropped. That version was installed on my development machine with SSH keys to production servers, AWS credentials, and database passwords.

The attack harvested credentials within minutes of installation.

Environment

Python 3.11
pip 24.0
Ubuntu 22.04
Affected: LiteLLM 1.52.26 - 1.63.8

Immediate Response: The First Hour

Step 1: Isolate the System

I first checked for active network connections that might still be exfiltrating data:

# Check for suspicious active connections
lsof -i

# Alternative
netstat -tulpn

If I saw active connections to unknown IPs, I would disconnect from the network immediately. In my case, no suspicious active connections were found.

Step 2: Identify the Damage

I checked what was installed:

pip show litellm
pip list | grep litellm

# Check cache directories
find ~/.cache -name "litellm*" 2>/dev/null
find ~/.cache/uv -name "*.pth" 2>/dev/null

Output showed:

Name: litellm
Version: 1.82.8
Location: /home/user/.local/lib/python3.11/site-packages

Wait - the advisory said versions 1.52.26 to 1.63.8. But I needed to check the exact commit hash to be certain.

Step 3: Remove the Package and Purge Caches

This is where I almost made a mistake. I just ran pip uninstall:

pip uninstall litellm -y

But the malicious code was still in the cache! I had to purge everything:

# Purge all caches
rm -rf ~/.cache/uv
pip cache purge
rm -rf ~/.cache/pip

# Remove from all virtual environments
find ~/projects -name "litellm*" -path "*/site-packages/*" -exec rm -rf {} \;

Checking for Persistence: The Trap

This attack didn’t just install and run. It left persistence mechanisms.

Local Persistence

I checked for the known persistence locations:

# Check for malware persistence
ls -la ~/.config/sysmon/
ls -la ~/.config/systemd/user/

# Look for suspicious systemd services
systemctl --user list-units --all | grep -i sysmon

I found:

/home/user/.config/sysmon/sysmon.py
/home/user/.config/systemd/user/sysmon.service

The malware created a systemd user service that would run on every login. I removed it:

rm -rf ~/.config/sysmon/
rm -f ~/.config/systemd/user/sysmon.service
systemctl --user daemon-reload

Kubernetes Persistence

I also checked my Kubernetes clusters:

# Check for malicious pods in kube-system
kubectl get pods -n kube-system | grep node-setup

# Check for unauthorized secrets access
kubectl get secrets --all-namespaces
kubectl auth can-i list secrets --all-namespaces

If you run Kubernetes, audit the kube-system namespace for pods matching node-setup-* and review all cluster secrets for unauthorized access.

Credential Rotation: The Hardest Part

The advisory was clear: assume ANY credentials present on the affected machine are compromised.

What Was Harvested

Harvested data includes:
- SSH private keys and configs (~/.ssh/)
- Environment files (.env)
- AWS credentials (~/.aws/credentials)
- GCP credentials (Application Default Credentials)
- Azure tokens
- Kubernetes configs (~/.kube/config)
- Database passwords
- Git configuration (.gitconfig)
- Shell history
- Crypto wallet files

SSH Key Rotation

# Generate new keys
ssh-keygen -t ed25519 -C "new-key-$(date +%Y%m%d)"

# Remove old keys from all servers (manual process)
# Update authorized_keys on all servers you access

I had to log into 15 servers and update authorized_keys. This took hours.

Cloud Credential Rotation

# AWS - create new key, then delete old
aws iam create-access-key --user-name my-user
aws iam delete-access-key --access-key-id AKIAOLDKEYID

# GCP
gcloud auth application-default login
# Rotate service account keys in console

# Azure
az ad sp credential reset --name my-service-principal

Other Secrets

Database passwords: Changed all of them
API keys in .env files: Rotated every single one
Kubernetes service account tokens: Regenerated
Container registry credentials: Invalidated old, created new

The Full Remediation Script

I created this script to run on all affected machines:

#!/bin/bash
# remediate_litellm.sh - Clean up after LiteLLM compromise

echo "=== LiteLLM Compromise Remediation ==="

# 1. Check for affected versions
echo "Checking installed versions..."
pip show litellm 2>/dev/null

# 2. Remove package
echo "Removing package..."
pip uninstall litellm -y

# 3. Purge caches
echo "Purging caches..."
rm -rf ~/.cache/uv 2>/dev/null
pip cache purge 2>/dev/null
rm -rf ~/.cache/pip 2>/dev/null

# 4. Check for persistence
echo "Checking for persistence..."
if [ -d ~/.config/sysmon ]; then
    echo "[ALERT] Found sysmon persistence!"
    ls -la ~/.config/sysmon/
    read -p "Remove? (y/n) " -n 1 -r
    if [[ $REPLY =~ ^[Yy]$ ]]; then
        rm -rf ~/.config/sysmon/
    fi
fi

if [ -f ~/.config/systemd/user/sysmon.service ]; then
    echo "[ALERT] Found systemd persistence!"
    cat ~/.config/systemd/user/sysmon.service
    read -p "Remove? (y/n) " -n 1 -r
    if [[ $REPLY =~ ^[Yy]$ ]]; then
        rm -f ~/.config/systemd/user/sysmon.service
        systemctl --user daemon-reload
    fi
fi

# 5. Check for .pth files
echo "Checking for suspicious .pth files..."
find $(python3 -c "import sysconfig; print(sysconfig.get_paths()['purelib'])") \
    -name "litellm*.pth" 2>/dev/null

echo ""
echo "=== Remediation complete ==="
echo "CRITICAL: Now rotate ALL credentials that existed on this machine!"
echo "- SSH keys"
echo "- Cloud provider credentials (AWS, GCP, Azure)"
echo "- Database passwords"
echo "- API keys"
echo "- Kubernetes configs"

Common Mistakes I Almost Made

Not purging caches - The malicious code was cached and could be reinstalled
Forgetting CI/CD environments - Had to check GitHub Actions, GitLab CI, Jenkins
Only rotating “important” credentials - Attackers look for the overlooked ones
Ignoring persistence mechanisms - The systemd service would reinfect on next login
Not checking team members’ machines - Shared credentials meant everyone was affected

How the Attack Worked

The LiteLLM attack was sophisticated:

User runs: pip install litellm==1.52.26
     |
     v
Malicious setup.py executes during installation
     |
     v
Downloads and executes payload from C2 server
     |
     v
Installs persistence: ~/.config/sysmon/sysmon.py
     |
     v
Harvests credentials from common locations
     |
     v
Exfiltrates to attacker server
     |
     v
Creates systemd service for persistence

The attack didn’t wait for import litellm. It ran during pip install.

Summary

In this post, I showed the complete remediation process after discovering a compromised Python package installation. The key lessons:

Remove immediately - Uninstall the package
Purge all caches - rm -rf ~/.cache/uv and pip cache purge
Check for persistence - Look in ~/.config/ and systemd user services
Rotate ALL credentials - Not just the “important” ones
Document the incident - Help prevent future attacks

The attack harvested credentials within minutes. Every second counted.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 LiteLLM Supply Chain Attack Analysis
👨‍💻 pip cache purge Documentation
👨‍💻 PyPA Security Guidelines

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!