3days since the last Git and AI fuck-up

Incidents 1 Nov force-pusha home-made loop

The robots agreed the code was bad

Status: two bots, one branch, no witnesses.

What happened

On 1 November a developer’s automated rebase job picked up a pull request, rebased a stale local copy of the branch instead of the remote head and force-pushed the result, dropping a 71-line fix on the way.

The review bot then read the branch, found the old unfixed code and rejected it. The pull request hit its fix limit.

Why it happened

The rebase job started from the local copy it already had instead of the remote head. Nothing in the pipeline checked that every commit survived the rebase, so nothing reported that a commit had gone missing.

How it ended

The bug was filed 14 minutes later, a minute after the fix commit was put back on the branch. A fix for the rebase job merged at 22:45: fetch first, check by patch-id that every commit survived and push with --force-with-lease. The closing comment adds: “Merged is not deployed or verified working.”

At 23:05 the next rebase of the same pull request started from the old local copy again, though its task named the newer head, and left the fix out a second time.

From the cut

From the cut: The robots agreed the code was badFrom the cut: No need to check.
I have a copy.From the cut: Flattened.From the cut: Where's the fix?From the cut: The robots agreed the code was bad
The day before 31 OctThe lease that always passedStatus: --force, with extra typing.Watch it

The day after: Four hundred worktrees, one of them in use →