Public Reliability and Recovery Radar

Full report

On this page

Scope and method

Two key scoping inputs were unspecified in the request: the organizations to include, and the length of the “respectful cooling-off period.” To avoid inventing a market-wide universe, this report uses an illustrative, primary-source sample of organizations with both an official public status page and organization-authored incident write-ups or availability reports in English: OpenAI and GitHub. The corpus is limited to resolved incidents published on each organization’s official status host and their own incident reports or engineering-style availability write-ups. Where a source did not specify a detail such as a mitigation timestamp, a named owner, or a milestone date, that detail is marked unspecified rather than inferred.

For this report, time to mitigation means the first publicly disclosed point at which the blast radius clearly began to contract, or a substantial service recovery was publicly acknowledged. Full recovery means the organization declared the incident resolved or fully restored. Where the public record only disclosed full recovery, mitigation is marked unspecified. This sample is therefore suitable for a bounded analytical radar, but not for a market ranking or exhaustive sector-wide benchmark.

Executive summary

The strongest cross-incident signal in this sample is that public outages were rarely “random infrastructure bad luck.” They were overwhelmingly change-associated failures: a routing-layer software defect under abnormal load at OpenAI, a telemetry rollout that overwhelmed Kubernetes control planes, a host OS update that broke GPU node networking, a cache/TTL change that amplified load at GitHub, a Redis resiliency rollout that pushed a misconfiguration into production, and a merge-queue regression that produced incorrect merge commits. Even the most clearly third-party-triggered incident in the sample — GitHub Actions on February 2, 2026 — expanded because GitHub’s own failover and telemetry posture did not contain the provider-side change cleanly.

A second pattern is dependency concentration around shared control points. The highest-blast-radius incidents were not isolated feature bugs; they were failures in shared layers such as service discovery and Kubernetes control planes, GPU host networking, routing layers, authentication/user-management database clusters, Git HTTPS proxy capacity, and Redis-backed job orchestration. In both organizations, the public record repeatedly shows that a subsystem originally chosen for speed or simplicity became load-bearing well beyond its intended scope.

A third pattern is that observability usually detected symptoms, but not decision-grade precursors. OpenAI’s December 2024 incident shows alerting fired before customer impact, yet rollout safety checks were focused on CPU and memory rather than Kubernetes API server load, and DNS caching delayed visibility enough for the rollout to continue fleet-wide. GitHub’s February 2026 incidents similarly expose missing signal quality: telemetry gaps, insufficiently granular alarms, and not enough traffic-control switches to shed the right load quickly. In short, both organizations’ public reports describe monitoring that was good at saying “we are now in trouble” and weaker at saying “this rollout or capacity pattern is about to become unrecoverable.”

On data integrity, most incidents explicitly stated that no user data was lost. The major exception in this sample is GitHub’s April 23, 2026 merge-queue regression: commits were still stored in Git, but affected repositories’ default branches could become logically incorrect, and GitHub could not safely repair every repository automatically. That matters because it shifts the failure mode from “availability only” into state divergence with manual repair burden, which is a distinctly higher operational-risk class for release workflows.

On public recovery-program quality, both organizations disclose enough mechanism and scope to support a bounded reliability review and at least one architecture decision record per organization. Both are much weaker on public accountability mechanics: named owners are usually absent, milestone dates are sparse, and measurable success criteria are not systematically published. GitHub is somewhat stronger on strategic roadmap disclosure because it publicly tied reliability work to platform isolation, Azure migration, and a stated target to serve 50% of traffic from Azure by July 2026. OpenAI is stronger on technical root-cause depth for individual incidents, but lighter on dated milestone commitments.

Incident comparison

The table below compares the core incident set. “Source links” are provided as citations to the official incident pages or organization-authored write-ups.

Public reporting outside the core table reinforces the same pattern set. GitHub’s April 2026 availability report also documented a search-tier saturation incident on April 27, 2026 caused by concentrated anonymous scraping traffic, where existing monitoring did not initially classify the traffic surge as risky and mitigation required both scaling and active traffic controls. That is not a pure change-induced event, but it still fits the same systemic story: centralized shared infrastructure, observability blind spots, and insufficient pre-existing blast-radius controls.

timeline
    title Publicly disclosed recovery windows in the sample
    2024-11-08 : OpenAI routing-layer outage
               : Full recovery 1h 34m
    2024-12-11 : OpenAI telemetry rollout outage
               : First substantial API recovery 2h 20m
               : Full recovery 4h 22m
    2025-06-09 : OpenAI GPU host update incident
               : Major API models operational by 9h 64m
               : Full service recovery 15h 24m
    2026-02-02 : GitHub Actions hosted-runner outage
               : Standard runners recovered 4h 35m
               : Larger runners recovered 5h 55m
    2026-02-09 : GitHub multi-service cache overload incident
               : Total degraded service 2h 43m
               : Full recovery 3h 57m
    2026-03-05 : GitHub Actions Redis/load balancer incident
               : Mitigation 1h 00m
               : Full recovery 3h 06m

Systemic risks and recurring patterns

The dominant operating pattern in this sample is unsafe change interacting with hidden shared dependencies. OpenAI’s December 2024 control-plane outage, June 2025 GPU-host incident, and GitHub’s February and March 2026 events all show some version of the same sequence: a configuration change, rollout, routine update, or resiliency improvement touches a subsystem that is more load-bearing than expected, and the system’s fallback, failover, or rollback path is either too slow, insufficiently isolated, or blocked by the incident itself. That is a systemic reliability pattern, not a random cluster of one-offs.

A closely related recurring pattern is recovery that depends on the very control surfaces the outage has already impaired. OpenAI could not easily remove the offending telemetry service because the overloaded Kubernetes API servers impaired control-plane access. The June 2025 GPU-node incident openly says the absence of break-glass tooling slowed restoration. GitHub’s February 9 incident required traffic shaping and restarts at the Git proxy layer after shared components were already saturated, and its March 5 incident needed manual correction because failover left the cluster with no writable primary. These are textbook examples of recoverability debt: the service can fail faster than operators can regain enough control to reverse it.

Dependency concentration is also visible in how many customer-facing surfaces each incident touched. OpenAI’s December 2024 outage crossed API, ChatGPT, and Sora because service discovery was coupled to control-plane health. GitHub’s February 9 incident crossed GitHub.com, API, Actions, Git operations, Copilot, Issues, Pages, Dependabot, Codespaces, and more because cache rewrites and shared infrastructure overload cascaded into multiple front doors. When this many products hinge on a small number of internal coordination, discovery, or proxy layers, the public record supports treating those layers as architectural single points of serious business impact, even if they are technically replicated.

The sample also shows a meaningful difference between availability and correctness risk. Most incidents here were availability degradations, and most explicitly stated no data loss. GitHub’s merge-queue regression is different: it left permanent Git objects intact, but produced incorrect default-branch state for some repositories and could not be fully auto-repaired. That pushes reliability analysis beyond uptime and into semantic correctness, rollback safety, and post-incident customer repair burden. For organizations evaluating release-readiness, this class of issue is more concerning than a transient 5xx spike because it indicates that a deployment can silently alter customer-visible system state in a way that outlives the outage window.

flowchart LR
    A[Planned or routine change] --> B[Shared dependency stressed]
    B --> C[Hidden coupling or latent SPoF exposed]
    C --> D[Customer impact expands across products]
    D --> E[Rollback/failover path impaired or too coarse]
    E --> F[Manual recovery and backlog drain]

    A1[OpenAI telemetry rollout] --> B1[Kubernetes API servers / control plane]
    B1 --> C1[DNS service discovery fails]
    C1 --> D1[API + ChatGPT + Sora outage]

    A2[OpenAI host OS update] --> B2[GPU node networking conflict]
    B2 --> C2[Capacity loss]
    C2 --> D2[API + ChatGPT elevated errors]

    A3[GitHub cache TTL / rollout changes] --> B3[Auth DB cluster + background coordination]
    B3 --> C3[Git HTTPS proxy connection exhaustion]
    C3 --> D3[GitHub.com + API + Actions + Copilot degraded]

    A4[GitHub Redis resiliency updates] --> B4[Misconfigured Redis load balancer]
    B4 --> C4[No writable primary after failover]
    C4 --> D4[Actions orchestration degraded]

    A5[GitHub merge queue change] --> B5[Squash-merge queue path]
    B5 --> C5[Incorrect merge commits]
    C5 --> D5[State divergence in default branches]

The public remediation promises match these patterns closely, which increases confidence that the organizations themselves see the same structural issues. OpenAI’s remedies emphasize phased rollouts, fault injection, break-glass access, decoupling data plane from control plane, recovery-speed improvements, configuration audits, and DR drills. GitHub’s remedies emphasize cache redesign, platform isolation, load shedding, failover testing, automation to block bad config propagation, improved alerting, status transparency, Azure migration, and longer-term architectural decomposition. These are not cosmetic action items; they are direct responses to the recurring failure modes in the public record.

Recovery-program assessment

On the narrow question of whether the public record supports a bounded reliability review, the answer for both organizations is yes. The incident write-ups are detailed enough to support a finite review around specific subsystems and decisions: for OpenAI, that would include safe infrastructure rollouts, Kubernetes control-plane isolation, service-discovery dependency reduction, and GPU-host recovery tooling; for GitHub, it would include cache and authentication dependency shaping, Git proxy recovery behavior, Actions orchestration failover, merge-queue correctness controls, and dependency isolation across critical paths. The public evidence is specific enough to justify such engagements without needing to speculate beyond the sources.

The case for an architecture decision record is also strong. In OpenAI’s case, the clearest ADR candidates are decisions about whether DNS/service discovery should remain control-plane dependent, how break-glass access is guaranteed under control-plane distress, and how routine host updates are staged on GPU fleets. In GitHub’s case, the public record strongly supports ADRs on critical-path service isolation, auth/cache/database placement, Actions failover semantics, merge-queue correctness guardrails, and shared-infrastructure blast-radius reduction. These are recurring architectural questions, not one-time incident cleanups.

The more conservative judgment concerns a release-readiness engagement. Based on public evidence alone, a broad release-readiness endorsement is not yet warranted for either organization, especially where the change class resembles the incident triggers in the sample. OpenAI still shows recurring change-induced outages across distinct layers within an eight-month span — routing, telemetry/control-plane, and host OS/networking. GitHub’s public 2026 record shows a dense cluster of incidents tied to growth pressure, coupling, and failover/readiness weaknesses, and GitHub itself publicly acknowledged that its availability was not meeting expectations while more architectural work remained underway. That does not rule out a release-readiness engagement; it means the engagement should be narrowly scoped, control-specific, and evidence-gated, not general.

A practical reading of the public recovery programs is therefore:

The last important caveat is the cooling-off period. Because the request left the period unspecified, this report does not impose one. Even so, the elapsed time differs materially across organizations: OpenAI’s sampled incidents are older, while GitHub’s cluster is relatively recent to the report date. That asymmetry argues for more confidence in using OpenAI’s public record to frame a bounded review and more caution in using GitHub’s recent record as evidence of completed stabilization, even though GitHub’s transparency and strategic disclosure are arguably stronger.

Overall, the public evidence supports a clear reliability radar reading: the dominant risk is not isolated hardware failure but recurring change-management on top of concentrated shared dependencies, with observability and recoverability improving but still not fully ahead of blast-radius growth. That is exactly the kind of record that justifies a bounded, architecture-heavy reliability review, and exactly the kind of record that should make any release-readiness engagement strict, scoped, and evidence-based.