3days since the last Git and AI fuck-up

Incidents 6 Nov GitHub itselfGitHub

Resolved, three times

Status: resolved. Thank you for your patience.

What happened

On Tuesday 3 November Git operations over HTTPS on GitHub were slow, and some failed, from 15:00 to 22:37 UTC. The status page showed it from 20:57, for an hour and 44 minutes.

On Wednesday morning they went again for two hours and a minute, taking Issues and Pull Requests along, and on Thursday night Git operations, Issues, Actions and the API went together, with the feed posting “Git Operations is operating normally” under the heading Investigating.

Why it happened

The first has since had its write-up: too many connections had piled up on the hosts that serve Git over HTTPS. The other two have no cause on record yet.

How it ended

The second and third closed with the same sentence, promising a detailed root cause analysis as soon as it is available. For the first, GitHub says it is working to alert on sustained Git HTTPS latency sooner.

On Hacker News someone who had seen about 30% of pushes and pulls fail for a couple of weeks, even when everything was green, shared their fix, while ! git push; do sleep 5; done.

From the cut

From the cut: Resolved, three timesFrom the cut: Resolved, three timesFrom the cut: Resolved, three timesFrom the cut: Resolved, three timesFrom the cut: Resolved, three timesFrom the cut: Resolved, three times
The day before 5 NovConflict resolved, commit goneStatus: conflict resolved. Saved by a timeout.Watch it