3days since the last Git and AI fuck-up

Incidents 23 Oct committed a secreta human

The key that was deleted three hours later

Status: deleted in May. Still there in October.

What happened

In May a config file carrying an admin key was committed to one of LiteLLM’s public repositories and deleted three hours later, which in Git means it stayed.

In October someone found it, used it to get further into the project’s GitHub organisation and shipped a poisoned 1.107.0 to PyPI.

Why it happened

Deleting a file in Git adds one more commit. The commit that carried the key stayed in the repository’s public history until 22 October. The key had been noted as needing revocation, and it was still valid more than four months later.

How it ended

Eighteen minutes after a researcher opened an issue, the intruder force-pushed main back to the previous commit over the force-push block, which left both commits retrievable by SHA and an audit log entry for every override.

The PyPI releases were frozen by 13:23 UTC, about two hours after they went out. LiteLLM says it is revoking every token tied to its internal systems and vendor platforms.

From the cut

From the cut: The key that was deleted three hours laterFrom the cut: Deleted.From the cut: Still worked.From the cut: Logged anyway.From the cut: The key that was deleted three hours later
The day before 22 OctThe watcher took the site downStatus: hired to watch. Took the site down.Watch it

The day after: The notes that force-pushed main →