What happened
GitHub keeps an internal service that tracks which database servers should receive writes. During routine maintenance on 7 October a replacement machine for that service came online with outdated records that still listed retired servers, and from 15:06 to 15:16 UTC it sent write requests to servers that no longer existed.
GitHub found the machine and paused automated maintenance. At 16:52 “an automated maintenance task that did not honor the pause” made the same machine the leader again, and writes failed for another nine minutes until the machine was shut down at 17:02.
Why it happened
The first failure came from the replacement machine’s outdated records. The second came from the pause itself, which had been broken by an update to GitHub’s Puppet setup.
How it ended
Across the two windows up to 26% of Git authentication requests failed and more than 12,000 organisations were affected. No data was lost. The automation now honours the pause.
