What Happened in the LiteLLM Supply Chain Attack
The Problem
I ran pip install litellm and got a compromised package:
$ pip install litellmCollecting litellm Downloading litellm-1.82.8-py3-none-any.whl (2.1 MB)Successfully installed litellm-1.82.8The installation looked normal. But versions 1.82.7 and 1.82.8 on PyPI contained credential-stealing malware. The attacker exploited weak GitHub Actions security from Trivy, a security vendor.
What the Malware Did
The malicious code harvested credentials from my system:
- SSH keys and configurations
- AWS/GCP/Azure credentials
- Kubernetes secrets
- Database passwords
- Environment variables
- Cryptocurrency wallet files
The malware triggered automatically through .pth files during package installation. No import needed.
The Attack Timeline
I traced back what happened:
Timeline:1. Attacker gains access to build pipeline (via Trivy GitHub Actions)2. Uploads malicious versions 1.82.7 and 1.82.8 to PyPI3. No corresponding GitHub releases (red flag ignored)4. Developer runs: pip install litellm5. pip downloads compromised version6. .pth file executes automatically7. Credentials exfiltrated to attackerThe malware was uploaded at 10:52 UTC. Thousands of developers installed it within hours.
How Trivy’s GitHub Actions Were Exploited
The attack exposed a critical weakness in GitHub Actions security.
The Problem with Tag-Based References
GitHub Actions tags and releases are mutable by default:
- Anyone with write access can move a tag
- Anyone with write access can delete and recreate a release
- No built-in immutability guarantee
I checked our workflows:
# VULNERABLE: Using tag-based referenceThe problem: tags are mutable. An attacker with access can move 0.20.0 to point at a malicious commit.
What the Attacker Did
The attacker:
- Exploited Trivy’s GitHub Actions workflow
- Created compromised releases/tags
- Made them immutable (ironically, what Trivy should have done)
This is the irony: Trivy is a security vendor (Aqua Security). Their own GitHub Actions workflow was the attack vector.
What Should Have Been Done
I fixed our workflows to use commit SHAs:
name: Security Audit
on: [push, pull_request]
jobs: audit: runs-on: ubuntu-latest steps: # VULNERABLE: tags are mutable # - uses: actions/checkout@v4
# SECURE: pin to commit SHA - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
# SECURE: pin trivy to specific commit - uses: aquasecurity/trivy-action@d710683aac9c950989e77bc0988f228f9402d472 # 0.20.0 with: scan-type: 'fs' ignore-unfixed: trueWhy the Attack Spread So Fast
The speed was alarming:
Speed factors:- Malware uploaded at 10:52 UTC- Developers installing within minutes- No hash verification in most workflows- uvx and pip pull latest by defaultLiteLLM is used for LLM routing. It often has:
- Network access to LLM APIs
- Access to prompts (potentially sensitive)
- Runs in trusted environments
This made it a high-value target.
Detecting the Compromised Package
I wrote a script to check if my LiteLLM installation was compromised:
import subprocess
def check_litellm_integrity(): """Check if installed litellm is compromised.""" try: import litellm version = litellm.__version__
# Check against GitHub releases result = subprocess.run( ["gh", "release", "view", "--repo", "BerriAI/litellm", f"v{version}"], capture_output=True, text=True )
if result.returncode != 0: return { "status": "COMPROMISED", "message": f"Version {version} exists on PyPI but not GitHub!", "version": version }
# Check for known bad versions if version in ["1.82.7", "1.82.8"]: return { "status": "COMPROMISED", "message": f"Version {version} is known malware!", "version": version }
return {"status": "OK", "version": version}
except ImportError: return {"status": "NOT_INSTALLED"}
# Run checkresult = check_litellm_integrity()print(result)The key insight: PyPI versions 1.82.7 and 1.82.8 had no corresponding GitHub releases. That was the red flag.
I also created a verification script for CI/CD:
#!/bin/bash# verify_actions.sh - Verify all actions are pinned to SHAs
echo "Checking GitHub Actions pinning..."
# Find all workflow filesworkflows=$(find .github/workflows -name "*.yml" -o -name "*.yaml")
for workflow in $workflows; do # Check for tag-based references (vulnerable) if grep -E 'uses:.*@v[0-9]+\.[0-9]+' "$workflow"; then echo "[WARNING] $workflow uses tag-based action reference" echo " Tag references are mutable and can be compromised" fi
# Check for SHA-based references (secure) if grep -E 'uses:.*@[a-f0-9]{40}' "$workflow"; then echo "[OK] $workflow uses SHA-pinned action" fidoneThe GitHub Actions Security Gap
I dug deeper into GitHub Actions immutability issues:
- Tags can be moved - Anyone with write access can move a tag to a different commit
- Releases can be recreated - Delete and recreate with different content
- No cryptographic verification - No way to verify action contents match expected
- Default behavior encourages tags - Documentation shows tag-based references
To verify PyPI against GitHub:
# Check if PyPI version matches GitHubpypi_version=$(curl -s https://pypi.org/pypi/litellm/json | jq -r '.info.version')github_release=$(gh release list --repo BerriAI/litellm --limit 10 | grep "v$pypi_version")
if [ -z "$github_release" ]; then echo "WARNING: PyPI version $pypi_version not found on GitHub!"fiProxy Layer vs SDK Attack Surface
The original reporter noted that attack surface differs based on usage.
SDK Usage (Higher Risk):
- Direct code execution in your process
- Access to all environment variables
- Access to all files your process can read
- Runs with your permissions
Proxy Layer Usage (Lower Risk):
- Runs in isolated container/process
- Network access only
- Limited environment exposure
- Can be sandboxed
If you use LiteLLM as a proxy layer instead of importing it directly as an SDK, your exposure is limited.
The Solution
I took these steps to protect our projects:
- Pin all actions to commit SHAs - No tag-based references
- Verify PyPI against GitHub releases - Cross-reference before trusting
- Use hash verification in requirements.txt:
litellm==1.59.8 \ --hash=sha256:abc123... \ --hash=sha256:def456...- Consider proxy architecture - Run sensitive dependencies in isolated containers
Summary
The LiteLLM supply chain attack exposed critical weaknesses:
- PyPI packages can be uploaded independently of GitHub releases - Always cross-reference
- GitHub Actions tags are mutable - Always pin to commit SHAs
- Even security vendors can have supply chain vulnerabilities - Verify everything
- Speed of attack propagation - Hash pinning is essential
The key lesson: The attacker used the same technique that should have protected Trivy - making releases immutable. Supply chain security must be proactive, not reactive.
Final Words + More Resources
My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me
Here are also the most important links from this article along with some further resources that will help you in this scope:
- 👨💻 LiteLLM GitHub Repository
- 👨💻 Trivy Security Scanner
- 👨💻 PyPI Security Best Practices
- 👨💻 GitHub Actions Security Hardening
Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!
Comments