Support for Kafka or Additional Message Brokers Beyond RabbitMQ/Postgres

[TheCaffinatedDeveloper]

Hatchet has a great competitive advantage in that its “batteries included” setup is easy, thoughtful and just works. Without wanting to completely open the can of worms around complexity, I did want to inquire about the potential of supporting message bus technologies beyond just RabbitMQ or Postgres-only mode.

The Why:
While RabbitMQ is easy to set up and mostly trivial to maintain, it is yet another dependency in the stack. I imagine several organizations (like mine) are already committed to another technology, like Kafka. There’s real value in having traditional consumers, microservice-to-microservice communication, streaming SQL, and Hatchet, all use the same message brokering solution. This allows us to monitor lag, scale, and issues in a unified way.

As I’m unfamiliar with which RabbitMQ-specific features Hatchet leverages, I’m curious how feasible it is to expand to other brokers like Kafka without major changes to the codebase. I’ve looked through prior issues and discussions around Kafka support, but didn’t find any hits.
It’s not a deal breaker for us, but supporting additional brokers would be a nice-to-have and could, in my opinion, speed up adoption for teams with existing message bus preferences.

Questions:
• Are there any architectural blockers to supporting something like Kafka as an alternative to RabbitMQ/Postgres?
• Which RabbitMQ-specific features are currently required by Hatchet?
• Would a plugin or adapter system for message brokers be considered on the roadmap?

Thank you for your time and the thoughtful defaults you’ve provided!

[abelanger5]

Hey @TheCaffinatedDeveloper good question and thanks for the kind words! It's something we've chatted about internally quite a bit.

We're not opposed to exposing a plugin mechanism where users can provide their own message bus implementation, but it won't be a plugin officially supported by the Hatchet team (at least for a while). For the rest of the year, the focus is most likely going to be on supporting and scaling Postgres as much as possible. We'd actually prefer to reduce dependency on RabbitMQ and make the Postgres-only mode more scalable and make Hatchet more lightweight since this would benefit all workloads.

Once we feel that Postgres-only mode is appropriately scalable we'll start thinking about adding support for some additional infrastructure components. There are lots of plugins we want to publish, like Timescale for aggregates (which we currently use on Hatchet Cloud), logging systems, monitoring, and message buses.

If you'd like to chat more specifically about your use-case or are open to collaborating on a beta version of a plugin system I'd be happy to: https://cal.com/team/hatchet/founders - or we can DM privately somewhere (feel free to DM on Discord).