Client stuck in isOpen && !isReady forever after server restart — reconnect loop dies silently

[vizyonok]

Environment

redis (node-redis)	6.1.0
Node.js	22 LTS (also seen on 24.x)
Redis server	7.x (redis:7-alpine, --appendonly yes)
OS	reproduced on both Linux (CI containers) and Windows 11
Topology	single client, single standalone server, no cluster, no sentinel

Summary

After the server goes away abruptly and comes back (SHUTDOWN NOSAVE followed by a
restart), the client can land in a state where:

• client.isOpen === true
• client.isReady === false
• no further lifecycle events are ever emitted — no error, no reconnecting,
  no end, indefinitely
• reconnectStrategy is never called again
• await client.connect() rejects with Socket already opened, so the application
  cannot recover the client either

The reconnect loop appears to have terminated without emitting a terminal event, while
the socket-state flag still claims the client is open. From the application's point of
view the client is neither usable nor recoverable, and — critically — nothing tells
it so. A process that reports readiness from isOpen will report healthy forever.

It reproduces roughly 50% of the time in a bare loop with no application load.

Expected

One of:

• the reconnect loop keeps retrying per reconnectStrategy until it succeeds (the
  documented behaviour), or
• the client gives up loudly — emits error/end and flips isOpen to false, so
  connect() can be called again.

Either is actionable. The current outcome — open, not ready, silent, and refusing
connect() — is not.

Actual

The client sits at isOpen: true, isReady: false with a dead retry chain and emits
nothing further. Observed in production-shaped code as "the pod never reconnects
after a Redis failover".

Minimal reproduction

Requires Docker (any Redis 7 works; the AOF flag only keeps the dataset around and is
not needed to trigger the bug).

// repro.js — node repro.js
import { execFileSync } from 'node:child_process';
import { createClient } from 'redis';

const NAME = 'node-redis-zombie-repro';
const PORT = 6399;
const sh = (...args) => execFileSync('docker', args, { stdio: 'ignore' });
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function attempt(n) {
  try {
    sh('rm', '-f', NAME);
  } catch {
    /* not running */
  }
  sh('run', '-d', '--name', NAME, '-p', `${PORT}:6379`, 'redis:7-alpine');
  await sleep(1000);

  const client = createClient({
    socket: {
      host: '127.0.0.1',
      port: PORT,
      connectTimeout: 2000,
      reconnectStrategy: (retries) => Math.min(50 * 2 ** retries, 2000),
    },
    disableOfflineQueue: true,
  });

  const events = [];
  for (const event of ['connect', 'ready', 'reconnecting', 'error', 'end']) {
    client.on(event, () => events.push([event, Date.now()]));
  }

  await client.connect();

  // Abrupt server death, then bring it back.
  try {
    sh('exec', NAME, 'redis-cli', 'SHUTDOWN', 'NOSAVE');
  } catch {
    /* SHUTDOWN kills the connection; a non-zero exit here is expected */
  }
  await sleep(500);
  sh('start', NAME);

  // A healthy reconnect loop emits at least every ~2s (the backoff cap).
  // Watch for 15s of total silence while open-but-not-ready.
  for (let i = 0; i < 30; i += 1) {
    await sleep(500);
    if (client.isReady) {
      console.log(`attempt ${n}: recovered`);
      await client.destroy();
      sh('rm', '-f', NAME);
      return false;
    }
  }

  const quietMs = Date.now() - (events.at(-1)?.[1] ?? 0);
  const zombie = client.isOpen && !client.isReady && quietMs > 10_000;
  console.log(
    `attempt ${n}: isOpen=${client.isOpen} isReady=${client.isReady} ` +
      `quietMs=${quietMs} lastEvent=${events.at(-1)?.[0]} ZOMBIE=${zombie}`,
  );
  if (zombie) {
    // The application has no way out from here:
    await client.connect().catch((err) => console.log(`  connect() -> ${err.message}`));
  }
  await client.destroy().catch(() => {});
  sh('rm', '-f', NAME);
  return zombie;
}

let zombies = 0;
for (let n = 1; n <= 10; n += 1) if (await attempt(n)) zombies += 1;
console.log(`\n${zombies}/10 attempts ended in a zombie client`);

Typical output on an affected run:

attempt 1: recovered
attempt 2: isOpen=true isReady=false quietMs=14512 lastEvent=error ZOMBIE=true
  connect() -> Socket already opened
attempt 3: isOpen=true isReady=false quietMs=14498 lastEvent=error ZOMBIE=true
  connect() -> Socket already opened
attempt 4: recovered
...
5/10 attempts ended in a zombie client

Why this matters

isOpen is the natural signal for a Kubernetes readiness probe. In the zombie state it
stays true, so the pod keeps reporting healthy and keeps receiving traffic while every
command fails. There is no event to react to and no supported way to rebuild the client
in place.

Workaround we shipped

A watchdog that treats lifecycle-event silence as the symptom, since the flags lie:
a healthy retry loop emits at least every backoff-cap interval (~2 s), so
open-but-not-ready with no event for ≥ 10 s is structurally impossible for a live loop.
On detection it hard-recycles the connection with destroy() + connect(), which
revives the client immediately (6/6 in the repro loop above).

Happy to turn the watchdog into a PR if a reconnectStrategy-side fix is not
straightforward.

[PavelPashov]

We couldn’t reproduce this: the supplied script recovered successfully in all 10 runs. The issue body also contains unremoved drafting instructions, internal references marked “do not paste,” and empty template fields; please clean it up and provide a reproducible case before we investigate further.

[vizyonok]

Apologies for the state of the original post — it went up with our internal drafting scaffolding pasted in by mistake. The body is now cleaned to just the report, and the title is fixed.

On reproduction: the race is timing-sensitive, so the hit rate varies a lot by machine. We see roughly 50% per attempt on two quite different setups (Windows 11 bare metal, and Linux CI-class runners); a fast machine with little contention may well recover every time. Two suggestions that should widen the window if your 10/10 recovered:

• run more attempts (20-30 rather than 10);
• constrain the server container's CPU (e.g. docker run --cpus=0.5 ...) or run the loop under parallel CPU load - the failure needs the restart to land at a particular point in the client's retry cycle, and contention makes that alignment more likely.

One diagnostic detail from our zombie runs: the last lifecycle event emitted is an error, then nothing - reconnectStrategy is never consulted again, and connect() rejects with Socket already opened. If it would help, we're happy to run an instrumented build with extra socket-state logging on one of the machines where this reproduces and attach the trace from a zombie run - tell us what you'd like logged.