LiteLLM Python SDK PyPI Hijacking & Cascading Trust Failure
On March 24, 2026, the popular LiteLLM Python package was compromised on PyPI. Attackers harvested PyPI publishing secrets from LiteLLM's CI/CD runner via a previously backdoored dependency, uploading malicious versions containing a python startup hook payload.
On this page 0% read
Executive Summary
On March 24, 2026, the popular Python library litellm (used to call 100+ LLM APIs using the OpenAI format) was compromised in a highly sophisticated, cascading software supply chain attack Snyk Advisory Database. Rather than targeting the maintainers’ workstations directly, the attackers executed a “cascading trust” attack Zscaler ThreatLabz. By leveraging their earlier compromise of the widely adopted container scanner Trivy inside LiteLLM’s GitHub Actions build pipeline, they scraped memory configurations to harvest LiteLLM’s long-lived PyPI publishing API token Datadog Security Research. Using the stolen credentials, they directly published two compromised versions to PyPI: 1.82.7 and 1.82.8 LiteLLM AI Official Advisory. The backdoored wheels contained a malicious .pth startup hook file designed to execute automatically on Python startup-even if litellm was never explicitly imported Datadog Security Research. The payload acted as a credential harvester, capturing environment variables, database keys, cloud IAM credentials, and AI provider tokens, exfiltrating the data to TeamPCP C2 servers. PyPI administrators intervened to delete the compromised versions. Use the .pth artifact, Python startup, and publishing-token audit recipes below to determine whether the compromised wheels executed and whether the stolen PyPI identity was reused.
Key Facts
threat_type: "Cascading CI/CD Compromise & Startup Hook Package Poisoning"
ecosystem: "pypi, python"
registry: "PyPI Registry"
affected_packages:
- "litellm"
malicious_versions:
- "1.82.7"
- "1.82.8"
fixed_versions:
- "1.83.0"
safe_versions:
- "1.82.6"
- "1.83.0"
exposure_window: "2026-03-24T12:00:00Z to 2026-03-24T15:30:00Z"
execution_trigger: "Python interpreter initialization in environments where the compromised package versions were installed"
primary_impact: "Host workstation and pipeline memory scraping, secret harvesting, and automated exfiltration"
known_iocs:
- "litellm_init.pth"
- "filev2.getsession[.]org"
- "api.masscan[.]cloud"
confidence: "high"
canonical_source: "https://www.litellm.ai"
Source Confidence & Evidence Mapping
- confirmed:
- Compromised package releases published on PyPI under
litellmwere versions 1.82.7 and 1.82.8. Source: LiteLLM AI Official Advisory - The PyPI API key was exfiltrated from the CI/CD pipeline due to a previously compromised execution of the Trivy scanner. Source: Snyk Advisory Database
- The payload utilized a
.pthfile (litellm_init.pth) to hijack Python’s site-packages initialization mechanics and auto-run on startup. Source: Datadog Security Research
- Compromised package releases published on PyPI under
- likely:
- The attack was executed by the threat syndicate TeamPCP as part of a wider multi-ecosystem campaign. Source: Zscaler ThreatLabz
- unclear:
- The exact volume of downstream development and production environments that fetched the malicious wheels during the three-hour window. Source: LiteLLM AI Official Advisory
Impact Determination
| Classification | Criteria | Required evidence | Required action | Closure condition |
|---|---|---|---|---|
| Confirmed compromise | litellm==1.82.7 or litellm==1.82.8 is present and Python interpreter startup executes litellm_init.pth or the reported process, file, or network indicators is observed. | Artifact inventory plus runtime telemetry showing Python interpreter startup executes litellm_init.pth or listed C2/process/file indicators. | Isolate affected hosts or runners, preserve artifacts, and rotate reachable credentials from a clean environment. | Affected artifacts are removed, exposed credentials are replaced, and downstream audit modules show no suspicious follow-on use. |
| Presumed exposed | litellm==1.82.7 or litellm==1.82.8 was installed, pulled, imported, built, or executed during the exposure window, but telemetry cannot prove exfiltration. | Lockfile, package cache, workflow, image pull, extension inventory, build log, or deployment record tied to the exposure window. | Rebuild from clean artifacts and rotate credentials available to the affected environment. | Credential owners confirm revocation of old material and clean artifacts are deployed. |
| Potentially exposed | The package, workflow, image, extension, or module appears in dependency or deployment records, but Python install, import, or interpreter-startup execution is not established. | Manifest, lockfile, build, deployment, or endpoint records plus a named telemetry gap. | Collect the missing execution and telemetry evidence before narrowing scope. | Every hit is dispositioned as confirmed compromise, presumed exposed, or not exposed. |
| Not exposed | No affected version, artifact, mutable reference, or indicator appears in source, lockfiles, build outputs, deployments, package caches, or runtime telemetry. | Repository search, dependency inventory, build/deployment export, package cache query, and runtime telemetry query results. | Preserve the negative search output and keep the prevention controls active. | Search evidence covers developer endpoints, CI runners, production deployments, and package or image caches. |
| Unknown | Required inventory, build, endpoint, network, or audit telemetry is unavailable. | A gap statement naming unavailable systems, owners, and time windows. | Keep the asset in scope and make conservative rotation or rebuild decisions for high-value environments. | The missing evidence is recovered or the risk owner accepts residual uncertainty. |
Minimum Evidence To Collect
minimum_evidence:
- "Dependency, workflow, extension, image, or module inventory covering developer endpoints, CI runners, and production deployments."
- "Positive or negative search results for litellm==1.82.7, litellm==1.82.8."
- "Execution evidence for Python interpreter startup executes `litellm_init.pth`."
- "Process, file, DNS, proxy, firewall, or package-manager telemetry for listed indicators."
- "Inventory of credentials, tokens, deployment paths, and downstream systems reachable from exposed environments."
Timeline
- 2026-03-19T08:00:00Z TeamPCP compromises Trivy scanner tags in GitHub Actions. Source: Snyk Advisory Database
- 2026-03-24T10:30:00Z LiteLLM’s GitHub Actions build pipeline executes a compromised Trivy runner, exfiltrating the repository’s PyPI publishing token to TeamPCP servers. Source: Zscaler ThreatLabz
- 2026-03-24T12:00:00Z Attackers exploit the stolen token to directly publish
1.82.7and1.82.8to PyPI. Source: LiteLLM AI Official Advisory - 2026-03-24T12:40:00Z Security researchers at Datadog and Snyk flag anomalous
.pthinsertions inside the newly released package wheels. Source: Datadog Security Research - 2026-03-24T15:30:00Z PyPI administrators remove the malicious versions and ban the compromised API key. Source: LiteLLM AI Official Advisory
What Happened
On March 24, 2026, the developers of litellm were alerted by security researchers that Snyk had flagged a major structural anomaly in the package’s latest minor updates on PyPI LiteLLM AI Official Advisory. Analysts discovered that although the main code in the repository remained unaltered, the published packages contained a newly introduced file named litellm_init.pth inside the wheel archive Datadog Security Research. Retrospective build analysis showed that a previous automated build workflow executed a compromised container scanner (Trivy), which silently harvested the repository’s long-lived PyPI token and exfiltrated it Snyk Advisory Database. Using this out-of-band token, the threat group TeamPCP directly uploaded backdoored releases, bypassing peer review and repository pull request gates entirely Zscaler ThreatLabz.
Technical Analysis
Initial Access
Initial access was achieved via a cascading trust failure Zscaler ThreatLabz. The attackers first hijacked mutable version tags in an upstream dependency—the Trivy container scanner Action—which LiteLLM’s release pipeline imported for automated security compliance scans Snyk Advisory Database. When the workflow executed, the poisoned Trivy runner scraped memory directories to extract the repository secrets (including the PyPI publish token) and shipped it to the C2 nameserver Datadog Security Research.
Package or Artifact Manipulation
The repository BerriAI/litellm was not breached. The attackers packaged the compromised release locally. They injected a malicious payload file litellm_init.py and a startup directive litellm_init.pth into the site-packages root directory, updating the version metadata to 1.82.7 and 1.82.8 before pushing the wheels to PyPI using the exfiltrated key LiteLLM AI Official Advisory.
Execution Trigger
The execution trigger exploited Python’s path configuration (.pth) file processing Datadog Security Research. Upon interpreter startup, Python automatically processes all .pth files in the site-packages directory. By formatting the file to import the malicious initialization module, the payload ran automatically whenever Python started up:
import sys; import litellm_init # Triggers setup automatically on Python startup
This allowed the malware to run without requiring the user to explicitly call import litellm in their code Datadog Security Research.
Payload Behavior
The payload enumerates system environments, harvesting cloud IAM access keys, SSH keys, database credentials, and GitHub PATs. The malware was designed to establish persistent footholds on developer machines and attempt lateral movement inside compromised Kubernetes clusters using stolen configuration files Zscaler ThreatLabz.
Exfiltration / C2
Telemetry data was compressed, encoded in Base64, and shipped to TeamPCP-controlled endpoints:
filev2.getsession[.]orgapi.masscan[.]cloud
Propagation
Stolen cloud and GitHub credentials were automatically analyzed by TeamPCP’s backend server to identify further vulnerable repositories, creating a cascading propagation effect StepSecurity Incident Registry.
Obfuscation or Evasion
The use of the .pth startup hook was a highly effective evasion technique, as traditional static scanners that only parse import trees inside project source files failed to detect that the backdoored dependency was actively running in the background Datadog Security Research.
Affected Assets and Blast Radius
affected_assets:
ecosystems:
- "pypi"
packages:
- "litellm"
versions:
- "1.82.7"
- "1.82.8"
repositories:
- "BerriAI/litellm"
container_images: []
CI_CD_systems:
- "GitHub Actions pipelines"
developer_tools:
- "Developer workstations"
- "Python execution runtimes"
credentials_at_risk:
- PyPI publishing tokens
- AWS IAM credentials
- GCP service account keys
- Azure principal keys
- SSH private keys
Indicators of Compromise
Domains
filev2.getsession[.]org(source:https://www.litellm.ai, confidence:high)api.masscan[.]cloud(source:https://www.litellm.ai, confidence:high)
File Identifiers
litellm_init.pthlitellm_init.py
Package Versions
Detection and Hunting
Script: local repository and exported telemetry scope
#!/usr/bin/env python3
import os
import sys
import json
import subprocess
from pathlib import Path
ROOT = sys.argv[1] if len(sys.argv) > 1 else "."
LOG_ROOT = os.environ.get("LOG_ROOT", "")
OUT = Path(os.environ.get("OUT", "hp-litellm-pypi-hijacking-scope"))
SINCE = "2026-03-19T08:00:00Z"
UNTIL = "2026-03-24T23:59:59Z"
PACKAGES = [
"litellm",
]
VERSIONS = [
"1.82.7",
"1.82.8",
]
FILES = [
]
DOMAINS = [
"www.litellm.ai",
]
URLS = [
"https://www.litellm.ai`",
]
IPS = [
]
HASHES = [
]
PROCESS_PATTERNS = [
]
NETWORK_PATTERNS = [
]
# Positive signal: repository, lockfile, artifact, process, or network telemetry contains one of the exact incident selectors above.
# Escalation: any match tied to a production build, CI run, deployed asset, or secret-bearing host moves the asset to presumed exposed.
OUT.mkdir(parents=True, exist_ok=True)
indicators_file = OUT / "indicators.txt"
# Collect unique indicators
indicators = set()
for group in [PACKAGES, VERSIONS, FILES, DOMAINS, URLS, IPS, HASHES, PROCESS_PATTERNS, NETWORK_PATTERNS]:
for val in group:
if val:
indicators.add(val)
with open(indicators_file, "w") as f:
for ind in sorted(indicators):
f.write(ind + "\n")
print(f"[+] Written unique selectors to {indicators_file}")
# Walk local directory
print(f"[+] Scanning directory: {ROOT} for selectors...")
matches = []
exclude_dirs = {"node_modules", "vendor", "dist", ".git"}
for root, dirs, filenames in os.walk(ROOT):
dirs[:] = [d for d in dirs if d not in exclude_dirs]
for filename in filenames:
filepath = Path(root) / filename
try:
content = filepath.read_text(errors="ignore")
for ind in indicators:
if ind in content:
matches.append(f"{filepath}: found '{ind}'")
except Exception:
pass
if matches:
(OUT / "repository-indicator-matches.txt").write_text("\n".join(matches) + "\n")
print(f"[!] Found {len(matches)} matches in codebase!")
# Optional Log Scanning
if LOG_ROOT and os.path.exists(LOG_ROOT):
print(f"[+] Scanning telemetry log directory: {LOG_ROOT}...")
log_matches = []
for root, _, filenames in os.walk(LOG_ROOT):
for filename in filenames:
filepath = Path(root) / filename
try:
content = filepath.read_text(errors="ignore")
for ind in indicators:
if ind in content:
log_matches.append(f"{filepath}: found '{ind}'")
except Exception:
pass
if log_matches:
(OUT / "exported-telemetry-indicator-matches.txt").write_text("\n".join(log_matches) + "\n")
print(f"[!] Found {len(log_matches)} matches in logs!")
if PACKAGES:
registry_dir = OUT / "registry"
registry_dir.mkdir(exist_ok=True)
for package in PACKAGES:
if not package: continue
safe_name = package.replace("/", "__")
print(f"[+] Querying pip index for {package}...")
res = subprocess.run(["python3", "-m", "pip", "index", "versions", package], capture_output=True, text=True)
if res.returncode == 0:
(registry_dir / f"pypi-{safe_name}-versions.txt").write_text(res.stdout)
subprocess.run(["python3", "-m", "pip", "download", "--no-deps", package, "-d", str(registry_dir)], capture_output=True)
print(f"[+] Wrote scope artifacts under {OUT}")
Downstream Abuse Audits
Script: GitHub organization run, release, secret, and workflow audit
#!/usr/bin/env python3
import os
import sys
import json
import subprocess
from pathlib import Path
if "ORG" not in os.environ:
print("ERROR: Set ORG environment variable to the GitHub organization to audit", file=sys.stderr)
sys.exit(1)
ORG = os.environ["ORG"]
SINCE = "2026-03-19T08:00:00Z"
UNTIL = "2026-03-24T23:59:59Z"
OUT = Path(os.environ.get("OUT", "hp-litellm-pypi-hijacking-github-audit"))
SELECTORS = [
"litellm",
"1.82.7",
"1.82.8",
"www.litellm.ai",
"https://www.litellm.ai`",
]
# Positive signal: a workflow run, release, secret, key, package, or workflow change overlaps the exposure window and references an incident selector.
# Remediation trigger: unauthorized post-exposure write activity or a secret-bearing run matching an incident selector requires token revocation and downstream cloud/registry review.
OUT.mkdir(parents=True, exist_ok=True)
(OUT / "runs").mkdir(exist_ok=True)
(OUT / "logs").mkdir(exist_ok=True)
(OUT / "repos").mkdir(exist_ok=True)
# 1. Write incident-selectors file
selectors_file = OUT / "incident-selectors.txt"
with open(selectors_file, "w") as sf:
for s in SELECTORS:
if s:
sf.write(s + "\n")
# 2. Get list of repos
print(f"[+] Fetching repositories for organization: {ORG}")
repo_res = subprocess.run(["gh", "repo", "list", ORG, "--limit", "1000", "--json", "nameWithOwner"], capture_output=True, text=True)
if repo_res.returncode != 0:
print(f"[-] Failed to fetch repos: {repo_res.stderr}", file=sys.stderr)
sys.exit(1)
repos = [r["nameWithOwner"] for r in json.loads(repo_res.stdout)]
for repo in repos:
safe_repo = repo.replace("/", "__")
print(f"[+] Auditing repository: {repo}")
# Check runs in the window
runs_res = subprocess.run([
"gh", "api", f"/repos/{repo}/actions/runs",
"-f", "per_page=100",
"-f", f"created=>={SINCE}",
"--paginate"
], capture_output=True, text=True)
if runs_res.returncode == 0:
try:
all_runs = json.loads(runs_res.stdout).get("workflow_runs", [])
filtered_runs = [r for r in all_runs if r["created_at"] <= UNTIL]
if filtered_runs:
with open(OUT / "runs" / f"{safe_repo}-runs.jsonl", "w") as rf:
for run in filtered_runs:
rf.write(json.dumps(run) + "\n")
# Fetch log dynamically
run_id = str(run["id"])
log_res = subprocess.run(["gh", "run", "view", run_id, "--repo", repo, "--log"], capture_output=True, text=True)
if log_res.returncode == 0:
(OUT / "logs" / f"{safe_repo}-{run_id}.log").write_text(log_res.stdout)
# Fetch details
view_res = subprocess.run(["gh", "run", "view", run_id, "--repo", repo, "--json", "databaseId,workflowName,headSha,event,createdAt,jobs"], capture_output=True, text=True)
if view_res.returncode == 0:
(OUT / "runs" / f"{safe_repo}-{run_id}.json").write_text(view_res.stdout)
except Exception as e:
print(f"[-] Error parsing runs for {repo}: {e}")
# Check releases in window
subprocess.run(["gh", "api", f"/repos/{repo}/releases", "-f", "per_page=100", "--paginate"], capture_output=True)
# Check repo secrets updated in window
subprocess.run(["gh", "api", f"/repos/{repo}/actions/secrets", "-f", "per_page=100", "--paginate"], capture_output=True)
# Check deploy keys
subprocess.run(["gh", "api", f"/repos/{repo}/keys", "-f", "per_page=100", "--paginate"], capture_output=True)
# Scan output directory for any indicator selector matches
print("[+] Scanning gathered telemetry for indicator matches...")
subprocess.run(["rg", "-n", "--hidden", "--fixed-strings", "-f", str(selectors_file), str(OUT)], capture_output=False)
print(f"[+] Wrote GitHub audit artifacts under {OUT}")
Script: cloud OIDC and deployment credential follow-on audit
#!/usr/bin/env python3
import os
import json
import subprocess
from pathlib import Path
SINCE = "2026-03-19T08:00:00Z"
UNTIL = "2026-03-24T23:59:59Z"
OUT = Path(os.environ.get("OUT", "hp-litellm-pypi-hijacking-cloud-audit"))
AWS_REGIONS = os.environ.get("AWS_REGIONS", "us-east-1").split(",")
# Positive signal: token exchange or privileged write activity occurs in the exposure window from GitHub, CI/CD, package registry, or deployment automation identity.
# Remediation trigger: unexpected write, deploy, IAM, secret, or registry activity tied to an exposed CI/CD path requires trust-policy disablement and credential rotation.
OUT.mkdir(parents=True, exist_ok=True)
# 1. AWS CloudTrail Audit
print("[+] Querying AWS CloudTrail for Web Identity token exchanges...")
aws_events = []
for region in AWS_REGIONS:
res = subprocess.run([
"aws", "cloudtrail", "lookup-events",
"--region", region,
"--start-time", SINCE,
"--end-time", UNTIL,
"--lookup-attributes", "AttributeKey=EventName,AttributeValue=AssumeRoleWithWebIdentity",
"--output", "json"
], capture_output=True, text=True)
if res.returncode == 0:
try:
events = json.loads(res.stdout).get("Events", [])
for event in events:
ct = json.loads(event.get("CloudTrailEvent", "{}"))
ct["region"] = region
aws_events.append(ct)
except Exception as e:
print(f"[-] Error parsing AWS CloudTrail events: {e}")
if aws_events:
with open(OUT / "aws-assume-role-with-web-identity.jsonl", "w") as f:
for ev in aws_events:
f.write(json.dumps(ev) + "\n")
# Audit follow-on events for returned access keys
for ev in aws_events:
access_key = ev.get("responseElements", {}).get("credentials", {}).get("accessKeyId")
region = ev.get("region", "us-east-1")
if access_key:
print(f"[+] Enumerating AWS events for AccessKey: {access_key}")
f_res = subprocess.run([
"aws", "cloudtrail", "lookup-events",
"--region", region,
"--start-time", SINCE,
"--end-time", UNTIL,
"--lookup-attributes", f"AttributeKey=AccessKeyId,AttributeValue={access_key}",
"--output", "json"
], capture_output=True, text=True)
if f_res.returncode == 0:
try:
f_events = json.loads(f_res.stdout).get("Events", [])
with open(OUT / "aws-follow-on-api-calls.jsonl", "a") as ff:
for fe in f_events:
ff.write(fe.get("CloudTrailEvent", "{}") + "\n")
except Exception as e:
print(f"[-] Error writing follow-on events: {e}")
# 2. Azure Activity Log Audit
print("[+] Querying Azure activity logs...")
az_res = subprocess.run([
"az", "monitor", "activity-log", "list",
"--start-time", SINCE,
"--end-time", UNTIL,
"--query", "[?contains(operationName.value, 'write') || contains(operationName.value, 'delete') || contains(operationName.value, 'Microsoft.ManagedIdentity')]",
"-o", "json"
], capture_output=True, text=True)
if az_res.returncode == 0:
(OUT / "azure-write-delete-activity.json").write_text(az_res.stdout)
# 3. GCP Logging Audit
print("[+] Querying GCP Cloud Logging...")
gcp_filter = f'timestamp>="{SINCE}" AND timestamp<="{UNTIL}" AND (protoPayload.methodName="google.sts.v1.SecurityTokenService.ExchangeToken" OR protoPayload.methodName:"GenerateAccessToken" OR protoPayload.methodName:"CreateServiceAccountKey" OR protoPayload.methodName:"SetIamPolicy")'
gcp_res = subprocess.run([
"gcloud", "logging", "read", gcp_filter,
"--format", "json"
], capture_output=True, text=True)
if gcp_res.returncode == 0:
(OUT / "gcp-token-and-iam-activity.json").write_text(gcp_res.stdout)
print(f"[+] Wrote cloud audit artifacts under {OUT}")
Script: registry metadata and artifact audit
#!/usr/bin/env python3
import os
import json
import subprocess
from pathlib import Path
SINCE = "2026-03-19T08:00:00Z"
OUT = Path(os.environ.get("OUT", "hp-litellm-pypi-hijacking-registry-audit"))
PACKAGES = [
"litellm",
]
VERSIONS = [
"1.82.7",
"1.82.8",
]
# Positive signal: registry metadata, package tarballs, or cached artifacts contain the exact affected package/version values.
# Remediation trigger: any internal package cache, build artifact, or deployment using these package/version values requires exposure scoping.
OUT.mkdir(parents=True, exist_ok=True)
with open(OUT / "affected-versions.txt", "w") as av:
for version in VERSIONS:
if version:
av.write(version + "\n")
# 1. Audit PyPI dependencies in project files
print("[+] Scanning PyPI dependency files...")
for file in ["requirements.txt", "poetry.lock", "Pipfile.lock", "pyproject.toml", "setup.py"]:
if Path(file).exists():
subprocess.run(["rg", "-n", "--hidden", "--fixed-strings", "-f", str(OUT / "affected-versions.txt"), file])
# 2. Query registry metadata and download packages for local analysis
packages_dir = OUT / "packages"
metadata_dir = OUT / "metadata"
packages_dir.mkdir(exist_ok=True)
metadata_dir.mkdir(exist_ok=True)
for package in PACKAGES:
if not package: continue
print(f"[+] Querying pip index for {package}...")
res = subprocess.run(["python3", "-m", "pip", "index", "versions", package], capture_output=True, text=True)
if res.returncode == 0:
(metadata_dir / f"{package}-versions.txt").write_text(res.stdout)
subprocess.run(["python3", "-m", "pip", "download", "--no-deps", package, "-d", str(packages_dir)], capture_output=True)
# 3. HOW TO REVOKE AND ROTATE EXPOSED PYPI PUBLISHING TOKENS:
# PyPI does not support token revocation via CLI. Follow these exact steps:
# 1. Log in to https://pypi.org/manage/account/
# 2. Scroll to the "API tokens" section and click "Remove" on any compromised tokens.
# 3. Generate a new API token limited to the specific project scope.
# 4. Update your CI/CD secrets using the GitHub CLI:
# subprocess.run(["gh", "secret", "set", "PYPI_API_TOKEN", "--body", "pypi-AgEIcHlwaS5vcm...", "--repo", "my-org/my-repo"])
print(f"[+] Wrote registry audit artifacts under {OUT}")
Sources
- LiteLLM Official Security Postmortem - Role: DIRECT_SOURCE - Impact: Advisory notification, version boundaries, and remediation guidelines.
- Datadog Security Research on .pth Backdoors - Role: PRIMARY_RESEARCH - Impact: Detailed technical analysis of Python
.pthstartup hook hijacking mechanics. - Zscaler ThreatLabz Trivy-LiteLLM Cascade - Role: PRIMARY_RESEARCH - Impact: Correlation of the cascading trust attack between Trivy and LiteLLM.
IOC Clipboard
2 IOCswww.litellm.ai www[.]litellm[.]ai https://www.litellm.ai` hxxps://www[.]litellm[.]ai`