Incident with Issues and Pull Requests

Wed, 10 Apr 2024 09:38:36 +0000

Apr 10, 09:38 UTC
Resolved - Between 8:18 and 9:38 UTC on Wednesday, April 10th, customers experienced increased error rates across several services due to an overloaded primary database instance, ultimately caused by an unbounded query. We mitigated the impact by failing the instance over to more capable hardware and shipping an improved version of the query that runs against read replicas. In response to this incident, we are also working to make performance improvements to the class of queries that most frequently resulted in failed requests during this timeframe.

Web-based repository file editing saw a 17% failure rate during the incident with other repository management operations (e.g. rule updates, web-based branch creation, repository renames) seeing failure rates between 1.5% and 8%. API failure rates for these operations were higher.

Issue and Pull Request authoring was heavily impacted during this incident due to reliance on the impacted database primary. We are continuing work to remove our dependence on this particular primary instance from our authoring workflows for these services.

GitHub search saw a 5% failure rate throughout this incident due to reliance on the impacted primary database when authorizing repository access. The majority of failing requests were for search bar autocomplete with a limited number of search result failures as well.

Apr 10, 09:38 UTC
Update - Issues and Pull Requests are operating normally.

Apr 10, 09:38 UTC
Update - The mitigation rolled out has successfully resolved the issue. We have seen failure rates reduce and normal service return across all affected features.

Apr 10, 09:30 UTC
Update - We are aware of impact across a number of GitHub features. This is primarily seen to be impacting write actions for Issues, Repositories and Pull Requests. Additionally we are seeing increased failure rates for search queries.

Our team has rolled out a mitigation and is monitoring for recovery.

Apr 10, 09:22 UTC
Investigating - We are investigating reports of degraded availability for Issues and Pull Requests