fix(mempool): duplicate tx check and gossipping systemtests

[vladjdk]

closes: #895, #896, #898

In #890, we fixed transaction rebroadcasting by ignoring the "already found" error in the mempool. Completely ignoring the error will still fail out transaction duplicates when sent from the client, but the RPC returns an erroneous Comet mempool cache error instead: "failed to send eth legacy tx: failed to broadcast transaction: : tx already in mempool [cosmossdk.io/[email protected]/errors.go:77]". This PR fixes that and emits the correct "already" found error by checking for txpool existence on the JSON-RPC side. Since rebroadcasting is done via a Cosmos transaction, this means that we can have the best of both worlds—correct erroring on the JSON-RPC side, as well as proper transaction gossipping of promoted queued transactions.

This PR also fixes the systemtests that check for broadcasting of queued transactions. For each broadcasted transaction, we ensure that every node in the systemtest receives each pending transaction through gossipping. Previous tests were passing because transactions were being propagated via committed blocks. This is incorrect, and hid our rebroadcasting errors found in #889. This is fixed by enforcing that each test case completes within the same block.

We also test the rebroadcasting function by first submitting a queued transaction that gets promoted only after a nonce gap is filled. The test ensures that the previously queued transactions becomes available to every other node via gossip after being promoted.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.73%. Comparing base (ad11ddc) to head (5b86d5a).

Files with missing lines	Patch %	Lines
mempool/mempool.go	0.00%	4 Missing ⚠️
rpc/backend/call_tx.go	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #897      +/-   ##
==========================================
- Coverage   64.97%   64.73%   -0.25%     
==========================================
  Files         317      317              
  Lines       21634    21651      +17     
==========================================
- Hits        14056    14015      -41     
- Misses       6375     6390      +15     
- Partials     1203     1246      +43     

Files with missing lines	Coverage Δ
rpc/backend/call_tx.go	42.08% <0.00%> (-18.56%)	⬇️
mempool/mempool.go	56.40% <0.00%> (-1.63%)	⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[vladjdk]

Removed this test. It doesn't actually test for rebroadcasting, and instead it waits for transactions to be committed.

[vladjdk]

This fn allows us to get tx existence on the rpc