feat(mempool): add timeout for reorg

[swift1337]

Description

• Unify mempool.Select and mempool.SelectBy calls
• Share re-check function across the whole reorg loop
• Implement "max timeout" for total recheck tx for the reog loop in evm mempool
• Add basic test case

Timeout logic

LegacyPool.WaitForReorgHeight now also has a timeout parameter (hard-coded to 300ms in mempool.Select)

func (pool *LegacyPool) WaitForReorgHeight(ctx context.Context, height int64, timeout time.Duration)

Inside this function, we call reorgContext that either sets ctx.timeout = timeout or lowers existing timeout by 100ms ~ ctx.timeout -= 100ms.

After ctx is timed out, is calls an atomic bool reorgCancelRequested that is checked for every account during promoteExecutables or demoteUnexecutables

Note: due to how EVM mempool works, it's challenging to properly propagate ctx across all goroutines as the mempool follows the actor model with lots of channels. This leads to a semi-forced context timeout, where we might still have some operations running after the timeout (ie, hard timeout if not guaranteed)

Closes: STACK-1927

---

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

• tackled an existing issue or discussed with a team member
• left instructions on how to review the changes
• targeted the main branch

[linear]

STACK-1927 EVM Mempool Select timeout or cursor tracking validations

[swift1337]

No mutex required, as this is an internal structure and its access is already under a lock

[swift1337]

@mattac21 @Eric-Warehime , so this is the place where we'd set the last checked height.

The drawback of this approach: now, to be included in pool.Pending() → mempool.SelectBy() results, tx should go through demoteUnexecutables, which is not the case for same-block-inclusion (imagine an addition to an empty mempool) eg:

[time_0: block(1), t1: add_tx(), t2: propose_block(2)]

Do you see any easy ways to handle this case?

[swift1337]

Closing, might open a re-implementation later if needed.