What's Happening?
The LiteLLM 1.107.0 software version, available on PyPI and official Docker images, was found to contain malicious code. This compromise occurred on October 21, when attackers used stolen GitHub CLI OAuth tokens from three engineers to trigger and approve
a release workflow. The malicious payload, which included `cook.pth`, `cook.py`, and `sitecustomize.py` files, was designed to steal credentials. It targeted cloud credentials (AWS, GCP, Azure), LLM provider keys, SSH and GPG keys, browser cookies, shell history, and the LiteLLM credentials table. The malware also incorporated a backdoor, which on Linux and macOS systems, would periodically check a GitHub dead drop for a domain to download and execute a second-stage Python script. The last safe version of the software is 1.106.2. The company behind LiteLLM, BerriAI, has since removed the poisoned Docker tags.
Why It's Important?
This incident highlights a significant supply chain security risk, particularly for organizations relying on open-source libraries and containerized deployments. The compromise of LiteLLM 1.107.0 means that any system that installed or updated to this version during the exposure window could have had sensitive credentials stolen and potentially be subject to remote code execution. The fact that LiteLLM is often pulled in as a transitive dependency by other agent and LLM frameworks, such as DSPy and CrewAI, means many users might be unknowingly affected. The malware's ability to target a wide array of credentials, including those for major cloud providers and LLM services, poses a severe threat to data security and operational integrity for businesses and developers across the U.S. and globally. The incident underscores the critical need for robust security practices, including strict version pinning and continuous monitoring of software supply chains.
What's Next?
Organizations that may have been affected are advised to immediately contain the threat by pinning LiteLLM to version 1.106.2 and blocking 1.107.0 in their package proxies or Workstation Protection. Workloads running affected images should be stopped, and any hosts, runners, or laptops that installed 1.107.0 should be isolated. All traffic to `litellm.dev` should be blocked. A critical next step is to rotate all potentially compromised credentials, including every key in the LiteLLM credentials table, cloud roles, CI tokens, SSH keys, and developer tokens on affected machines. Investigations should be conducted to scope the exposure by querying for `litellm==1.107.0` in SBOMs, lockfiles, and images, including those built from 'latest' tags during the compromise period. Additionally, systems should be checked for the presence of the backdoor files (`cook.pth`, `cook.py`, `sitecustomize.py`) and any outbound calls that fetched additional code. Rebuilding affected environments is recommended over cleaning them due to the potential for second-stage malware. Long-term, implementing cooldown policies for new releases, pinning versions with hashes, replacing floating image tags with digests, and reviewing release workflow approvals are crucial to prevent similar incidents.
Beyond the Headlines
This security breach extends beyond immediate credential theft, revealing deeper implications for software development and deployment practices. The incident underscores the vulnerability inherent in relying on 'latest' tags for container images and loose version ranges in dependency management, as these practices can inadvertently introduce malicious code into production environments. The use of stolen OAuth tokens to compromise the release pipeline highlights the importance of multi-factor authentication and stringent access controls for developer accounts and CI/CD systems. Furthermore, the attack's sophistication, targeting not just individual machines but also the centralized credential management of an LLM gateway, points to an evolving threat landscape where attackers are increasingly focusing on high-value targets within the software supply chain. This event serves as a stark reminder that even widely used and trusted libraries can become vectors for attack, necessitating a shift towards more proactive and comprehensive security strategies that encompass every stage of the software lifecycle, from development to deployment and ongoing operations.
