Surviving Redis Failover Without Logging Everyone Out

A single Redis restart signs out every user at once, and the login surge that follows can take down the authentication path that was supposed to recover from it. This page is part of the managing session state in Redis guide and covers the persistence, replication and client-side behaviour that turn a failover into a few retried writes instead of a full outage.

Why a Restart Is Worse Than It Looks

Losing sessions is only the first-order effect. The second-order effect is that every one of those users tries to log in within the same few minutes, and the login path is the most expensive request your application serves — password hashing, an identity provider round trip, a new session write, often a multi-factor challenge.

How a session store restart becomes a wider outage Losing sessions logs everyone out, which produces a synchronised login surge, which saturates the authentication path and can exhaust identity provider rate limits, extending the outage well past the restart itself. Restart seconds All sessions gone every user, at once Login surge the expensive path Provider limits minutes to hours The restart lasts seconds; the consequences last as long as the slowest rate limit in the chain.
The cost is not the outage window — it is the recovery window, which is set by the least elastic component in your login path.

Persistence: Make a Restart Survivable

Enable the append-only file with everysec fsync. It writes each mutation to disk and flushes once a second, which costs a small amount of throughput and means a restart loses at most a second of session writes rather than everything. Snapshot-only persistence is a poor fit for sessions: an hourly snapshot means an hour of new logins vanish on restart, which is the same problem with extra steps.

Size the disk and watch rewrite behaviour. The append-only file grows and is periodically rewritten in the background, and a rewrite on an undersized volume fails in a way that quietly disables persistence until someone notices. Alert on the last successful rewrite timestamp as well as on disk usage.

Persistence is not a substitute for replication — it protects against a restart, not against losing the machine. It also does not protect against a corrupted file, so confirm that your instance actually loads the file at start-up in a staging drill rather than assuming it.

Replication and Promotion

Run at least one replica and a mechanism that promotes it: Sentinel for self-hosted deployments, or the managed failover your provider offers. What matters more than the mechanism is knowing what your clients do while it happens.

Client behaviour during a failover window During promotion a client sees connection errors and read-only replies, so it must retry with backoff, cap the retry window, and fail closed rather than treating a failed lookup as an absent session. During promotion connection refused READONLY replies a few seconds Client should retry with backoff cap the total wait then give up cleanly Client must not treat an error as "no session" that is a silent logout
The dangerous line of code is the one that catches a store error and continues as an anonymous request — it converts a brief infrastructure blip into a mass logout.
export async function loadSession(sid: string) {
  try {
    return await withRetry(() => redis.hGetAll(`sess:${sid}`), {
      attempts: 3,
      backoffMs: [50, 150, 400],       // covers a typical promotion window
      retryOn: (e) => isConnectionError(e) || isReadOnlyError(e),
    });
  } catch (err) {
    // The store is unavailable — this is NOT "no session".
    throw new SessionStoreUnavailable(err);   // handled as 503, never as anonymous
  }
}

Distinguish “no such session” from “cannot reach the store” everywhere in the code path. The first is a normal 401 that sends the user to log in; the second is a 503 that keeps their session intact for when the store returns. Collapsing the two is the single most common way a failover turns into a stampede.

Controlling the Stampede When It Does Happen

Some failures do lose sessions, so plan the recovery. Rate-limit the login endpoint per address and globally, with a queue rather than an outright rejection where possible, so the surge is spread rather than amplified by clients retrying. Add jitter to any client-side automatic re-login so a thousand tabs do not retry in lockstep.

If you use an external identity provider, know its rate limits before you need them and make sure your login path backs off rather than hammering — many providers respond to a flood by throttling for a period that outlasts your outage, which is how a thirty-second Redis restart becomes a twenty-minute login failure.

Finally, make the failure visible to users rather than mysterious. A clear “we are having a problem, please try again in a moment” page, served without hitting the store, is both kinder and cheaper than a login form that fails silently and invites immediate retries.

Frequently Asked Questions

Should the application fail open when the session store is down?

No. Failing open means treating every request as unauthenticated, which signs out every active user and is indistinguishable from the outage you were trying to avoid — or, worse, means skipping the authorization check entirely if the code is written carelessly. Return a 503 with a retry hint, keep the sessions intact, and let the store recover.

Is append-only persistence enough on its own?

It protects against a restart of the same instance, not against losing the machine or the volume. Pair it with at least one replica and an automatic promotion mechanism. Persistence and replication solve different failures, and a deployment with only one of them will eventually meet the failure the other one covers.

How long should a client retry during a failover?

Long enough to cover a typical promotion — a few hundred milliseconds across three attempts with backoff — and no longer. Retrying for seconds ties up request handlers and turns a store problem into thread exhaustion, which is a worse outage than the one you are riding out. Cap the total wait, then fail cleanly with a 503.

Can I fall back to an in-memory session store?

It is tempting and it is a trap. Sessions created in a local fallback exist on one instance, disappear on the next deploy, cannot be revoked centrally, and are invisible to “sign out everywhere” — so you have traded a visible outage for an invisible security gap. If continuity genuinely matters more than consistency for some read-only surface, scope the fallback to that surface explicitly rather than to authentication as a whole.

How do I test failover safely?

In a staging environment with production-like traffic, trigger a promotion deliberately and measure three things: how long clients saw errors, whether any request was served as anonymous, and how the login rate behaved afterwards. Repeat it after any client library upgrade, because retry and reconnection behaviour is exactly the sort of thing that changes quietly between versions.

Connection Pooling and Timeouts

Most of the damage during a failover comes from client configuration rather than from the store itself. A pool sized without thought, or a command with no timeout, turns a few seconds of unavailability into exhausted request handlers and a queue that never drains.

Size the pool per process against your concurrency rather than per route, and set both a connect timeout and a command timeout — a second or less for each is generous for a store whose normal response is sub-millisecond. An unbounded command timeout is the specific setting that converts a slow store into an application outage, because every request handler waits indefinitely on a call that will never return.

Enable automatic reconnection with a bounded backoff, and make sure your client library re-resolves the endpoint on reconnect rather than caching an address. Managed services move the primary to a different address during failover, and a client that reconnects to the old one keeps failing long after the promotion has finished — a failure that looks like a much longer outage than actually occurred.

Finally, keep the health check honest. A liveness probe that only checks the process, rather than issuing a real command against the store, will report a healthy application that cannot serve a single authenticated request.

Rehearsing the Failure

Configuration you have never exercised is a hypothesis. Book a recurring exercise — quarterly is enough — in which you promote a replica in a staging environment under realistic traffic and measure three numbers: how long clients saw errors, how many requests were served as anonymous rather than as 503, and how the login rate behaved for the following ten minutes.

The second number should be zero, and if it is not you have found the fail-open path before your users did. The third tells you whether your rate limiting and jitter are adequate, and it is the one most likely to surprise: a small number of automatic client retries can produce a surprisingly sharp spike when they all fire on the same timer.

Record the results somewhere durable, with the date and the versions in play. A failover drill whose outcome nobody wrote down is one you will run again from scratch next year, and the comparison between runs is what tells you whether a change improved things or merely moved the problem.

Repeat the exercise after every client library upgrade and every change to pool configuration. Reconnection and retry behaviour is precisely the kind of thing that changes between minor versions, and the change is invisible until the day it matters.

Persistence Options Compared

Persistence choices and what a restart costs with each With no persistence a restart loses every session, with periodic snapshots it loses everything since the last snapshot, and with an append-only file flushed each second it loses at most a second of writes. None every session lost full login stampede Snapshots loses the last interval an hour of logins gone Append-only loses about a second the right choice
Snapshot-only persistence is the trap: it looks configured, and it discards exactly the newest sessions — the ones belonging to users who just logged in.