Skip to content

fix(hyperliquid): bump HL_DEPOSIT_LOOKBACK default to 7 days#3438

Open
droplet-rl wants to merge 1 commit into
masterfrom
droplet/T90K0AL22-C0AQEHGS56Z-1780148503-175839
Open

fix(hyperliquid): bump HL_DEPOSIT_LOOKBACK default to 7 days#3438
droplet-rl wants to merge 1 commit into
masterfrom
droplet/T90K0AL22-C0AQEHGS56Z-1780148503-175839

Conversation

@droplet-rl
Copy link
Copy Markdown
Contributor

Summary

Bumps the HL_DEPOSIT_LOOKBACK default in HyperliquidExecutorConfig from 1 hour → 7 days, so the HyperliquidExecutor / HyperliquidFinalizer pod can survive a normal-cadence restart without losing visibility of in-flight SwapFlowInitialized events.

Failure mode this addresses

HyperliquidExecutor.initialize() (src/hyperliquid/HyperliquidExecutor.ts:187-194) anchors:

const fromBlock = await getBlockForTimestamp(this.logger, this.chainId, toBlock.timestamp - this.config.lookback);
this._dstSearchConfig = { to: toBlock.number, from: fromBlock, ... };

dstSearchConfig.from is fixed at startup and never advances backward. finalizeSwapFlows queries outstanding orders via that anchored window. So any SwapFlowInitialized older than (last_startup_ts − lookback) is permanently invisible to the running instance — no retry, no catch-up scan.

With the previous 1h default, any pod restart longer than ~1h after a SwapFlowInitialized event causes that flow to be dropped silently: the L1 limit order still fills, the SwapHandler retains the resulting finalToken, but finalizeSwapFlows is never called and the recipient never receives funds.

Production hit

Sponsored-OFT ETH → HyperCore deposit 0xc582a7440f1b…f380 (15,940 USDT0 → USDC, recipient 0xD103CC…7fb2).

  • SwapFlowInitialized fired 2026-05-29 12:14:50 UTC, quoteNonce = 0xa5133bb1…b4a3, finalToken = 0xb88339cb…630f (USDC HyperEVM), via DstOFTHandler 0xeb8feE79…2284.
  • L1 spot swap filled — SwapHandler 0x6aa49d33…602e now holds 15,937.39965114 USDC on HL spot.
  • No SwapFlowFinalized ever emitted. User has 0 USDT0 on HL spot and never received the USDC.
  • Same SwapHandler did finalise a smaller deposit (932 USDT0 → 931.07 USDC, recipient 0x85cfea68…) initiated 2026-05-30 00:21 UTC, processed 2.5 h later — consistent with the bot having restarted between the two events and the older one falling outside the 1h lookback.

Full investigation thread: https://app.slack.com/client/T90K0AL22/C0AQEHGS56Z/thread/C0AQEHGS56Z-1780148503.175839

Why 7 days

  • Covers any realistic deploy / OOM / SIGHUP / cluster-maintenance window.
  • HyperEVM blocks are ~1 s, so a 7-day startup scan is ~600k blocks paginated via CHAIN_MAX_BLOCK_LOOKBACK — one-time RPC cost on startup, no steady-state overhead.
  • Still env-overridable (HL_DEPOSIT_LOOKBACK) for operators that want a different value.

Recovery for the stuck deposit

Once this lands and the HyperliquidFinalizer pod is restarted with the new default (or with HL_DEPOSIT_LOOKBACK ≥ ~96000), the bot should:

  1. Pick up SwapFlowInitialized at quoteNonce 0xa5133bb1…b4a3.
  2. Recompute limitOrderOut (≈ 15,932 USDC at current stable rates).
  3. Bundle DstOFTHandler.finalizeSwapFlows(USDC, [0xa5133bb1…b4a3], [limitOrderOut]).
  4. Emit SwapFlowFinalized; SwapHandler L1-spotSends to the user.

Pre-conditions to verify before relying on this path:

  • Bot signer still holds PERMISSIONED_BOT_ROLE on DstOFTHandler 0xeb8feE79…2284.
  • SwapHandler USDC inventory (15,937.40) ≥ recomputed limitOrderOut. Edge case if USDT0/USDC drifts notably above 1 before restart.
  • No other outstanding orders ahead in the FIFO that would drain inventory first. Currently none.

If those don't hold, recovery falls back to an admin FUNDS_SWEEPER_ROLE sweep + manual L1 spotSend.

Follow-ups (not in this PR)

  • Replace the startup-anchored window with a Redis-backed cursor so restarts can't lose history at all.
  • Add a Datadog/relayer monitor: SwapFlowInitialized > 30 min ago AND no matching SwapFlowFinalized.
  • Across UI (across-protocol/dapp) coherence patches for OFT/HyperCore deposits — separate work referenced in the Slack thread.

Test plan

  • yarn typecheck — passes locally.
  • yarn prettier --check on the edited file — clean.
  • yarn eslint on the edited file — no warnings.
  • Deploy to a non-prod / dev cluster, restart with default lookback, observe startup dstSearchConfig.from extends ≥ 7 days back.
  • Restart prod HyperliquidFinalizer and watch for SwapFlowFinalized(0xa5133bb1…b4a3, 0xD103CC…7fb2, …) on DstOFTHandler 0xeb8feE79…2284.
  • Confirm user 0xD103CC…7fb2 HL spot USDC balance increases by ≈ 15,937 USDC.

🤖 Generated with Claude Code

HyperliquidExecutor.initialize() anchors dstSearchConfig.from at
(startup_timestamp - lookback) and never widens it backward. Any
SwapFlowInitialized event older than that boundary is permanently
invisible to the running bot instance.

With the previous 1h default, a routine pod restart can silently drop
in-flight SwapFlows: the SwapHandler still holds the swapped-out
finalToken, but no SwapFlowFinalized is ever submitted and the user's
funds sit at the SwapHandler indefinitely. We hit this in production
with a 15,940 USDT0 -> USDC HyperCore deposit that was skipped while a
later 932 USDT0 deposit on the same SwapHandler finalised normally.

Bump the default to 7 days so a pod restart inside a normal operational
window cannot lose visibility of an outstanding swap. HL_DEPOSIT_LOOKBACK
is still env-overridable for operators that want a shorter or longer
window. A cleaner fix (Redis-backed cursor that survives restarts) is
tracked separately.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant