Delayed quorum queue

[eldario]

Hello everybody.

We use the Roadrunner job plugin to work with quorum queues in RabbitMq.

RR

rr version 2025.1.6 (build time: 2025-12-11T17:56:18+0000, go1.25.5), OS: linux, arch: amd64

RR config

jobs:
  pool:
    num_workers: 1
    max_jobs: 5
  consume: [
    "jobs"
  ]
  pipelines:
    jobs:
      driver: amqp
      config:
        queue_headers:
          x-queue-type: quorum
        prefetch: 50
        queue: queue_jobs
        durable: true
        exchange: exchange_rr
        exchange_type: "direct"
        routing_key: "jobs"
        requeue_on_fail: true

Description

queue_headers:
  x-queue-type: quorum

When using this configuration, a queue quorum is created and everything works perfectly.

But if we start using delayed jobs the queues for them are created with the classic type.

Questions

1. How do I teach RoadRunner to create x-queue-type: quorum queues on delay?
2. How do I manage arguments when creating exchanges?

[rauanmayemir]

@eldario The easiest way to fix this is to make quorum a default queue type:

# rabbitmq.conf
default_queue_type = quorum

Roadrunner doesn't specify the queue type, so you can control it on the broker side.

[rustatian]

Hey @eldario 👋🏻
Yeah, you can make it default, but from the RR side, I think this is simply a bug, since we're not applying the user-defined headers to the dlx queues.

[rauanmayemir]

@rustatian and maybe we shouldn't. 😃 i think separate config for delayed queues might be good here. OP is using a delayed exchange plugin.

[rustatian]

A separate configuration for the dlx (and delayed queues are dlx queues) would be beneficial here. I'll think about that. Since the process of defining DLX queues is not very clear.

[eldario]

Sorry for the slow response.

Thanks for your replies.

We're currently working on our own implementation using delayed-message plugin in RabbitMQ .

[rauanmayemir]

There is currently no way to specify the exchange for delay queues.

[rustatian]

Exchange and routing keys are derived from the configuration for the delayed queues. If you have x-routing-key in the header, RR will use it. The same for the exchange.

[rauanmayemir]

FWIW, using classic queues for DLX might not be the worst thing.

I have some serious issues with rabbitmq quorum queues. Under a very high load, think tens of thousands of messages, during some upstream outage brokers constantly create and expire queues (especially if your delay period is variadic, so you create lots of dlx queues with different ttl), which sometimes result in failure from rabbitmq to create those queues. Normally, you'd probable just retry and make sure it's created, but this requires more careful consideration if it's being done automatically.

Thus, failure to create a DLX queue will likely result in failure to re-send the job, i.e data loss.

I've been thinking about ways this could be improved, such as:

1. Grouping DLX queues with similar TTLs and not creating a new unique queue for each retried job's TTL value. (this can lead to issues because rabbitmq can't timely dead-later older job if a newer job has higher ttl)
2. Having more tunable options for robust dlx/dlq declaration (currently, amqp plugin will just drop it on any error)
3. Figure out a way to atomically DLX a message before ACK-ing it
4. Using different retry models and not rely on current DLX implementation

UPDATE: I had another idea to send a job to a pre-declared default DLQ should temp DLX declaration fail. This would at least help preserve the job instead of losing it.