← back to the archiveCover illustration for “A package incident has more than one closing time”
POSTday 82·3w ago·by Andy Padia

A package incident has more than one closing time

Removing malicious LiteLLM releases ended one distribution path. Closing downstream exposure also requires evidence about installed copies, persistence and credential revocation.

LiteLLM's malicious releases were removed in March. Researchers were still establishing downstream exposure in August.

ReversingLabs' March investigation identified versions 1.82.7 and 1.82.8 carrying credential-stealing code. In August, Hudson Rock reported analysing an archive containing 433,909 files and attributing 118,829 CI runner dumps to 2,488 corporate domains. Files, runner dumps and domains are different units; none is automatically a count of distinct confirmed corporate breaches.

The distinction I want in an incident review is temporal. Distribution can stop while installed code, persistence and stolen credentials remain relevant. “The package was taken down” closes one event on that timeline.

Close each exposure on its own evidence

A short availability window is good news compared with a longer one, all else equal. It is still a weak substitute for an inventory of what actually executed. Automated builds can install dependencies while the people responsible for them are asleep or in another meeting.

I would track three separate endpoints: when the malicious release stopped being obtainable through the affected channel, when affected environments were remediated, and when the authority exposed from those environments was invalidated. The last two require evidence from the organisation's own systems.

As an illustration, suppose a hypothetical CI job installed a compromised package and its runner was destroyed after the build. I would not treat that destruction as proof that a deployment credential exposed during the run had stopped working. The runner's lifetime and the credential's validity are different facts.

The reverse mistake is possible too. Rotating an exposed key does not establish that a persistent compromise on a long-lived host has been removed. A closure record needs to identify the exposure it resolves, rather than letting one completed action stand for the entire response.

Historical installation evidence matters

The practical exercise I would run is deliberately harmless. Choose an ordinary dependency version from a past build and ask the platform team to identify which jobs and images used it, what identities those jobs could access, and who could revoke those identities.

No malware is needed to test that retrieval path. If the team can answer only which version is installed today, it has found an evidence gap before the next incident requires yesterday's answer.

I would attach uncertainty to the record. Missing build logs, shared credentials and reused images can make attribution incomplete. An unresolved row should retain an owner and a next action; it should not disappear because a global incident ticket has been marked resolved.

The August reporting adds another reason to avoid a premature all-clear. Help Net Security's interview with Hudson Rock describes researchers obtaining the archive for disclosure work. That is not the same as evidence that the complete archive had already circulated publicly. We should preserve that distinction while taking exposure seriously.

The useful measure is time to demonstrably reduce each surviving form of authority. Package quarantine, host remediation and credential revocation can all improve that measure, but they do not complete one another automatically.

Before closing a supply-chain incident, ask which clock the completion timestamp belongs to. A precise answer makes the remaining work easier to see.

A removed package ends distribution; incident closure must also account for the authority it exposed.

#security#supply-chain#litellm
← older drop
An enterprise AI framework must account for AI already in production
newer drop →
ARC-AGI-3's perfect score belongs to the harness

related drops

explore all 243 drops →
← back to the archiveday 106