Skip to content

What Happened in the LiteLLM Supply Chain Attack

The Problem

I ran pip install litellm and got a compromised package:

Terminal window
$ pip install litellm
Collecting litellm
Downloading litellm-1.82.8-py3-none-any.whl (2.1 MB)
Successfully installed litellm-1.82.8

The installation looked normal. But versions 1.82.7 and 1.82.8 on PyPI contained credential-stealing malware. The attacker exploited weak GitHub Actions security from Trivy, a security vendor.

What the Malware Did

The malicious code harvested credentials from my system:

  • SSH keys and configurations
  • AWS/GCP/Azure credentials
  • Kubernetes secrets
  • Database passwords
  • Environment variables
  • Cryptocurrency wallet files

The malware triggered automatically through .pth files during package installation. No import needed.

The Attack Timeline

I traced back what happened:

Timeline:
1. Attacker gains access to build pipeline (via Trivy GitHub Actions)
2. Uploads malicious versions 1.82.7 and 1.82.8 to PyPI
3. No corresponding GitHub releases (red flag ignored)
4. Developer runs: pip install litellm
5. pip downloads compromised version
6. .pth file executes automatically
7. Credentials exfiltrated to attacker

The malware was uploaded at 10:52 UTC. Thousands of developers installed it within hours.

How Trivy’s GitHub Actions Were Exploited

The attack exposed a critical weakness in GitHub Actions security.

The Problem with Tag-Based References

GitHub Actions tags and releases are mutable by default:

  • Anyone with write access can move a tag
  • Anyone with write access can delete and recreate a release
  • No built-in immutability guarantee

I checked our workflows:

# VULNERABLE: Using tag-based reference
- uses: aquasecurity/[email protected]

The problem: tags are mutable. An attacker with access can move 0.20.0 to point at a malicious commit.

What the Attacker Did

The attacker:

  1. Exploited Trivy’s GitHub Actions workflow
  2. Created compromised releases/tags
  3. Made them immutable (ironically, what Trivy should have done)

This is the irony: Trivy is a security vendor (Aqua Security). Their own GitHub Actions workflow was the attack vector.

What Should Have Been Done

I fixed our workflows to use commit SHAs:

.github/workflows/security.yml
name: Security Audit
on: [push, pull_request]
jobs:
audit:
runs-on: ubuntu-latest
steps:
# VULNERABLE: tags are mutable
# - uses: actions/checkout@v4
# SECURE: pin to commit SHA
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
# SECURE: pin trivy to specific commit
- uses: aquasecurity/trivy-action@d710683aac9c950989e77bc0988f228f9402d472 # 0.20.0
with:
scan-type: 'fs'
ignore-unfixed: true

Why the Attack Spread So Fast

The speed was alarming:

Speed factors:
- Malware uploaded at 10:52 UTC
- Developers installing within minutes
- No hash verification in most workflows
- uvx and pip pull latest by default

LiteLLM is used for LLM routing. It often has:

  • Network access to LLM APIs
  • Access to prompts (potentially sensitive)
  • Runs in trusted environments

This made it a high-value target.

Detecting the Compromised Package

I wrote a script to check if my LiteLLM installation was compromised:

import subprocess
def check_litellm_integrity():
"""Check if installed litellm is compromised."""
try:
import litellm
version = litellm.__version__
# Check against GitHub releases
result = subprocess.run(
["gh", "release", "view", "--repo", "BerriAI/litellm", f"v{version}"],
capture_output=True, text=True
)
if result.returncode != 0:
return {
"status": "COMPROMISED",
"message": f"Version {version} exists on PyPI but not GitHub!",
"version": version
}
# Check for known bad versions
if version in ["1.82.7", "1.82.8"]:
return {
"status": "COMPROMISED",
"message": f"Version {version} is known malware!",
"version": version
}
return {"status": "OK", "version": version}
except ImportError:
return {"status": "NOT_INSTALLED"}
# Run check
result = check_litellm_integrity()
print(result)

The key insight: PyPI versions 1.82.7 and 1.82.8 had no corresponding GitHub releases. That was the red flag.

I also created a verification script for CI/CD:

#!/bin/bash
# verify_actions.sh - Verify all actions are pinned to SHAs
echo "Checking GitHub Actions pinning..."
# Find all workflow files
workflows=$(find .github/workflows -name "*.yml" -o -name "*.yaml")
for workflow in $workflows; do
# Check for tag-based references (vulnerable)
if grep -E 'uses:.*@v[0-9]+\.[0-9]+' "$workflow"; then
echo "[WARNING] $workflow uses tag-based action reference"
echo " Tag references are mutable and can be compromised"
fi
# Check for SHA-based references (secure)
if grep -E 'uses:.*@[a-f0-9]{40}' "$workflow"; then
echo "[OK] $workflow uses SHA-pinned action"
fi
done

The GitHub Actions Security Gap

I dug deeper into GitHub Actions immutability issues:

  1. Tags can be moved - Anyone with write access can move a tag to a different commit
  2. Releases can be recreated - Delete and recreate with different content
  3. No cryptographic verification - No way to verify action contents match expected
  4. Default behavior encourages tags - Documentation shows tag-based references

To verify PyPI against GitHub:

Terminal window
# Check if PyPI version matches GitHub
pypi_version=$(curl -s https://pypi.org/pypi/litellm/json | jq -r '.info.version')
github_release=$(gh release list --repo BerriAI/litellm --limit 10 | grep "v$pypi_version")
if [ -z "$github_release" ]; then
echo "WARNING: PyPI version $pypi_version not found on GitHub!"
fi

Proxy Layer vs SDK Attack Surface

The original reporter noted that attack surface differs based on usage.

SDK Usage (Higher Risk):

  • Direct code execution in your process
  • Access to all environment variables
  • Access to all files your process can read
  • Runs with your permissions

Proxy Layer Usage (Lower Risk):

  • Runs in isolated container/process
  • Network access only
  • Limited environment exposure
  • Can be sandboxed

If you use LiteLLM as a proxy layer instead of importing it directly as an SDK, your exposure is limited.

The Solution

I took these steps to protect our projects:

  1. Pin all actions to commit SHAs - No tag-based references
  2. Verify PyPI against GitHub releases - Cross-reference before trusting
  3. Use hash verification in requirements.txt:
litellm==1.59.8 \
--hash=sha256:abc123... \
--hash=sha256:def456...
  1. Consider proxy architecture - Run sensitive dependencies in isolated containers

Summary

The LiteLLM supply chain attack exposed critical weaknesses:

  1. PyPI packages can be uploaded independently of GitHub releases - Always cross-reference
  2. GitHub Actions tags are mutable - Always pin to commit SHAs
  3. Even security vendors can have supply chain vulnerabilities - Verify everything
  4. Speed of attack propagation - Hash pinning is essential

The key lesson: The attacker used the same technique that should have protected Trivy - making releases immutable. Supply chain security must be proactive, not reactive.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments