Incident with Git Operations, API Requests, Actions, Pages, Issues and Copilot

Wed, 10 Apr 2024 19:03:05 +0000

Apr 10, 19:03 UTC
Update - On April 10, 2024, between 2024-04-10 18:33 UTC and 2024-04-10 19:03 UTC, several services were degraded due to the release of a compute-intensive database query that prevented a key database cluster from serving other queries.

GitHub Actions saw delays and failures across the entire run life cycle and had a significant increase in the number of timeouts in API requests. All Pages deployments failed for the duration of the incident. Git Systems saw approximately 12% of raw file download requests and 16% of repository archive download requests return HTTP 50X error codes for the duration of the incident. Issues experienced increased latency for issue creation and updates. Codespaces saw roughly 500 requests to create and resume a Codespace timeout during the incident.

We mitigated the incident by rolling back the offending query. We are working to introduce measures to automatically detect compute-intensive queries in test runs during CI to prevent an issue like this one from recurring.

Apr 10, 19:03 UTC
Investigating - Git Operations, API Requests, Actions, Pages, Issues and Copilot are operating normally.

Apr 10, 19:03 UTC
Resolved - This incident has been resolved.

Apr 10, 19:03 UTC
Update - Git Operations, API Requests, Actions, Pages, Issues and Copilot are operating normally.

Apr 10, 19:01 UTC
Update - Copilot is experiencing degraded performance. We are continuing to investigate.

Apr 10, 18:55 UTC
Update - We're aware of issues impacting multiple services and have rolled back the deployment. Systems appear to be recovering and we will continue to monitor.

Apr 10, 18:53 UTC
Update - API Requests is experiencing degraded performance. We are continuing to investigate.

Apr 10, 18:45 UTC
Update - Copilot is experiencing degraded availability. We are continuing to investigate.

Apr 10, 18:42 UTC
Update - Issues is experiencing degraded performance. We are continuing to investigate.

Apr 10, 18:42 UTC
Update - API Requests is experiencing degraded availability. We are continuing to investigate.

Apr 10, 18:41 UTC
Investigating - We are investigating reports of degraded performance for Git Operations, API Requests, Actions and Pages