Add custom reconnect delay strategy option - #857
Conversation
Allow opt-in recovery to use a caller-supplied `calculateDelay(attempt)` function instead of the built-in exponential-backoff-with-jitter, for strategies like full jitter, decorrelated jitter, or a fixed schedule. If the function throws or returns something other than a finite, non-negative number, amqplib falls back to the built-in strategy for that attempt and reports the problem via a `handler-error` event, rather than failing to reconnect. Closes amqp-node#855
|
Thanks for picking this up, @GiHoon1123 — this is exactly the shape I proposed in #855. I maintain an AMQP adapter (Connectum) built on the v2 recovery, which was the original motivation for the request, so a review from the requester's side: The API matches the proposal. Two things: 1. CI has not run. The PR currently reports zero check runs, which looks like the pending-approval gate for a first-time contributor rather than anything wrong with the branch. @cressie176 — could you approve the workflow run so the suite reports? Nothing else here is blocked on the author. 2. Scope of "Closes #855". #855 raised two points, and this PR addresses one of them. The hook lets a caller avoid the cap overshoot by owning the number, but Happy to help validate: the PR notes the recovery tests use an in-memory model, and we have a testcontainers-based RabbitMQ recovery harness. I can run this branch against a live broker — measuring the actual inter-attempt intervals across real connection drops, plus the fallback-on-throw and fallback-on-invalid paths — and report back, or contribute a broker-level integration test in the style of the existing suite, whichever is more useful to you. |
|
Good catch. I pushed a follow-up that clamps the built-in delay after jitter and added a deterministic test for the maximum positive jitter case. The recovery tests, lint, and typecheck pass locally. |
|
The clamp looks right, and thanks for turning it around so fast — that closes the second half of #855. One observation about the shape it produces, worth considering before this lands. Clamping after symmetric jitter puts a point mass exactly on the cap: at steady state The usual ways to keep the cap without the atom are to shrink the base ( Either way |
|
Thanks for the PR @GiHoon1123 and the thoughtful comments @intech. I plan to go through this tonight. |
| } | ||
| throw new Error(`calculateDelay must return a finite, non-negative number of milliseconds (got ${String(custom)})`); | ||
| } catch (err) { | ||
| setImmediate(() => core.emit('handler-error', toError(err, 'calculateDelay failed'), 'calculateDelay')); |
There was a problem hiding this comment.
handler-error is for when a custom error handler throws an exception. I don't think we should use it in this instance. I think I would prefer throwing the error synchronously
| // as an uncaught exception or unhandled rejection and crash the process. | ||
| // Fall back to the built-in strategy and report the problem via | ||
| // `handler-error` unconditionally - a harmless no-op if nobody's listening. | ||
| function resolveDelay(core, recovery, attempt) { |
There was a problem hiding this comment.
"resolve" in "resolveDelay" implies asynchronous calculation. I might just call it "calculateDelay" and have it delegate to the supplied function or built in.
| // Fall back to the built-in strategy and report the problem via | ||
| // `handler-error` unconditionally - a harmless no-op if nobody's listening. | ||
| function resolveDelay(core, recovery, attempt) { | ||
| if (recovery.calculateDelay) { |
There was a problem hiding this comment.
Prefer guard conditions (immediate return) rather than long lived if and nesting
|
|
||
| Without `recovery` options, behavior is unchanged. | ||
|
|
||
| ### Custom delay strategy |
There was a problem hiding this comment.
Needs to be clear that the user must implement maxDelay otherwise they might assume the library will enforce it
| const jitterPart = base * recovery.jitter; | ||
| const offset = jitterPart > 0 ? Math.random() * jitterPart * 2 - jitterPart : 0; | ||
| return Math.max(0, Math.round(base + offset)); | ||
| return Math.min(recovery.maxDelay, Math.max(0, Math.round(base + offset))); |
There was a problem hiding this comment.
As @intech pointed out, enforcing maxDelay means the jitter will not be applied once maxDelay has been exceeded. I would prefer to keep the jitter, or capping the base to maxDelay - maxJitter so there is always room for the jitter?
"resolve" implies asynchronous work; this just delegates to the supplied function or the built-in strategy.
No behavior change - inverts the condition to return early for the no-custom-strategy case instead of nesting the rest of the function inside the if branch.
A broken custom calculateDelay (throws, or returns something other than a finite, non-negative number) is a bug in caller-supplied code. Silently falling back to the built-in strategy and reporting it via handler-error hid the problem instead of surfacing it - switch to letting it propagate synchronously so it's caught immediately. Drops the now-unused handler-error fallback path and the core parameter it needed. Updates tests accordingly; removes the test for the old no-listener fallback behavior, which no longer applies.
Callers supplying calculateDelay must enforce their own cap - amqplib doesn't apply maxDelay on top of a custom function's return value.
Capping base at maxDelay and then clamping base+offset put a point mass exactly on maxDelay once base saturated: every positive jitter draw (half of them) collapsed onto the same value, defeating the purpose of jitter - a fleet of clients would reconnect in lockstep at that instant. Capping base at maxDelay / (1 + jitter) instead means the maximum possible offset lands exactly on maxDelay, so the result never exceeds it without a final clamp, and the distribution stays smooth right up to the cap. Adds a regression test with a fixed Math.random() proving the result is no longer collapsed onto maxDelay for a mid-range jitter draw once the base has saturated.
Missed updating these when calculateDelay switched from catching and falling back (via handler-error) to propagating synchronously - the JSDoc still described the old behavior.
|
Thanks for the review - pushed changes addressing all of this:
One consequence of the synchronous throw: it happens before |
Summary
Closes #855.
Adds an optional
calculateDelay(attempt)recovery setting for custom reconnect delays such as full jitter, decorrelated jitter, or a fixed schedule.The option works with both the promise and callback APIs.
Also caps the built-in jittered delay at
maxDelay. The base is capped atmaxDelay / (1 + jitter)rather thanmaxDelayitself, so the max jitter offset lands exactly on the cap instead of every delay above it collapsing onto the same value.Behavior
calculateDelayerrors (throw, or an invalid return value) now propagate synchronously instead of falling back silently.maxDelayonly bounds the built-in strategy -calculateDelaycallers cap their own delay.'1000'isn't coerced.calculateDelay, the built-in strategy is used.Tests
node --test test/recovery.test.js(12 passing),npm run lint,npm run typecheckpass. Also rannpm test(326 passing) against a local RabbitMQ broker.