# The Restart Loop That Kept the Outage Alive

Microsoft and GitLab experienced restart loops just three weeks apart. Different causes, but the same underlying problem: automated recovery mechanisms that made recovery harder.

Author: Sverre Senneset — CEO & Co-founder, Queuey.ai
Published: 2026-10-29T21:23:14Z

On September 29, Microsoft had an unusual problem during an Azure outage. Its infrastructure kept restarting backend instances because they couldn't establish connections to a database. The database itself was healthy, but the infrastructure nodes had exhausted their capacity to establish new outbound connections.

Restarting the instances didn't solve the problem. Quite the opposite, as fewer instances were now available to handle requests, putting more pressure on those that remained.

Three weeks later, GitLab ran into something similar. This time it was OpenBao running in Kubernetes, where incorrectly configured health probes caused the active node to restart before it had completed startup.

Two different incidents, both involving what we call a *restart loop*. The infrastructure detects a problem, restarts the service, encounters the same problem again, and repeats the process.

## Microsoft: Nearly six hours to recover

The Microsoft incident affected Azure OpenAI, Foundry Agent Service, Foundry Models and other Azure AI services in Sweden Central. Customers experienced intermittent failures, increased latency and HTTP 500 and 503 errors between 10:03 and 15:58 UTC.

According to Microsoft's [Post Incident Review](https://azure.status.microsoft/en-us/status/history/?trackingId=7QL5-Z50), the initial trigger was a planned deployment that cleared internal caches during peak traffic. Two existing defects in the cache lookup logic caused most lookups to bypass the cache and go directly to a regional backend.

Within minutes, traffic to that backend had increased to approximately **20 times its normal level**.

The caching layer became saturated, response times increased, and connections remained open longer. Client retries added even more traffic. Eventually, the infrastructure nodes exhausted their capacity for new outbound network connections, leaving backend instances unable to establish new connections to the cache or database.

The health checks then introduced another problem.

Microsoft's automated health checks required successful database connectivity. When an instance couldn't connect, it was considered unhealthy and restarted. Since the underlying connection exhaustion remained, the restarted instances encountered the same problem.

As more instances entered the restart loop, fewer were available to handle incoming requests. The remaining instances received additional traffic, including retries from dependent services.

Engineers confirmed that the database itself was healthy and attempted several mitigations, including scaling the number of instances and increasing compute resources. None resolved the incident.

At 15:40 UTC, Microsoft reconfigured the health check so it no longer depended on remote connectivity. The restart loop stopped, and service availability recovered by 15:58.

The health check wasn't necessarily reporting something false. The instances genuinely couldn't establish database connections. The problem was the decision made from that information.

Microsoft also noted that existing network and diagnostic metrics hadn't made the connection exhaustion particularly obvious, contributing to the time it took to identify the problem.

## GitLab: Restarting before startup completed

On October 20, GitLab experienced a related issue affecting SaaS groups and projects with GitLab Secrets Manager enabled. Users received HTTP 500 errors when opening merge requests, work items and other pages containing the sidebar.

The incident lasted 3 hours and 23 minutes.

According to [GitLab's incident report](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23288), the problem originated with incorrectly configured Kubernetes probes for OpenBao, its secrets management service.

The active OpenBao node was repeatedly restarted before it could finish its startup sequence and become available.

Each restart sent the service back to the beginning. It started initializing, failed a probe, was restarted, and had to start over. Unless something changed, there was little reason to expect a different outcome on the next attempt.

GitLab also had another issue that made the incident more visible to users. The UI called OpenBao while rendering the sidebar but didn't safely handle unexpected non-JSON responses when the service was unavailable. This allowed an OpenBao failure to affect pages that otherwise might have remained accessible.

The immediate fix was to correct the Kubernetes probe configuration. GitLab also identified the need to improve how the sidebar handles OpenBao being unavailable.

Compared with Microsoft, the technical cause was different. GitLab wasn't restarting OpenBao because of exhausted network connections. It was restarting a service that hadn't been given enough time to become ready.

## This isn't just a Kubernetes problem

Restart loops are often associated with Kubernetes, particularly through the familiar `CrashLoopBackOff` status. But there's nothing Kubernetes-specific about repeatedly restarting a service when the conditions causing the failure haven't changed.

A Linux application, for example, might exit when it loses its database connection. With `Restart=on-failure`, systemd can automatically restart it, only for the application to encounter the same unavailable database and exit again. Systemd has safeguards such as restart rate limiting, but that doesn't make the underlying dependency problem disappear.

Kubernetes has separate liveness, readiness and startup probes for exactly this reason. A service can be running without being ready to accept traffic, and slow initialization doesn't automatically mean the process needs to be restarted.

The [Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/pods/probes/) explicitly warns that incorrectly configured liveness probes can cause cascading failures.

The important distinction isn't whether a health check detects a real problem. It's whether restarting the process can do anything about it.

External dependencies can certainly be relevant to readiness. If a service can't reach the database it needs, it might not be able to serve requests. But using that same dependency to determine liveness is a different decision, particularly when restarting the process won't restore connectivity.

There's also a practical complication. Removing an instance from service through a readiness check can be useful, but if every instance becomes unready because they all depend on the same unavailable database, you've still got a problem. The difference is that you're not necessarily making the situation worse by repeatedly terminating processes that could recover.

**Being unable to serve traffic and needing to be restarted are two different conditions.**

## We've seen a similar pattern with retries

GitHub experienced a related failure pattern during its [August 17 outage](https://queuey.ai/articles/github-retry-storm/), when a latent retry bug in VS Code amplified traffic to an already struggling service by approximately 10 times.

A retry storm isn't the same as a restart loop. One increases demand, while the other can repeatedly remove available capacity. But both can turn a temporary problem into something considerably harder to recover from.

Microsoft experienced both effects during the same incident. Retries increased the load on the backend while the restart loop reduced the capacity available to handle it.

Recovery mechanisms aren't free. Retries add traffic, restarts remove capacity, and both can make an overloaded system worse.

## Would your infrastructure do the same?

Reading these two reports, there's a fairly simple test that could uncover similar problems in other systems.

Take a service that depends on a database or another external component and make that dependency temporarily unavailable. Watch what happens to the health checks and the running instances.

Does the service stop accepting traffic until the dependency recovers, or does the infrastructure start restarting it? If it restarts, what condition is that supposed to fix?

I'd also try deliberately slowing down startup, particularly for services that restore state after a crash. Does the infrastructure allow enough time to initialize, or can the service end up being restarted indefinitely?

Finally, test it under load. Restarting one instance might be harmless under normal conditions, but during an incident, every instance taken out of service shifts more traffic onto the remaining capacity. New instances may also add load while establishing connections and warming caches.

These aren't particularly complicated tests, but I suspect they'd reveal some interesting behavior in systems that otherwise appear healthy.

Both incidents were eventually resolved by changing how the health checks worked. Microsoft had already tried adding capacity, while GitLab had a service that simply wasn't given enough time to start.

I suspect quite a few production systems would behave the same way. I'd be curious to know how many teams have actually tested their health checks with the database down and the service under load.

---

## References

1. [Microsoft Azure: Post Incident Review, September 29, 2026](https://azure.status.microsoft/en-us/status/history/?trackingId=7QL5-Z50)
2. [GitLab: October 20, 2026 incident report](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23288)
3. [Kubernetes: Liveness, Readiness, and Startup Probes](https://kubernetes.io/docs/concepts/workloads/pods/probes/)
4. [systemd: Service restart behavior](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html)
5. [Queuey: GitHub's Retry Storm Shows What Retry Logic Is Missing](https://queuey.ai/articles/github-retry-storm/)

Source: https://queuey.ai/articles/the-restart-loop-that-kept-the-outage-alive/