Skip to content

Bug: Ack management with Kafka broker does not work as described in documentation #2618

@antoinehumbert

Description

@antoinehumbert

Describe the bug

In the documentation for Acknowledgment in Kafka consumers, the following use cases are described (note that the 0.6 documentation still mention the auto_commit parameter despite being deprecated):

  • auto_commit=True : topic offset is commited on message consumption.
  • auto_commit=False : topic offset is commited after message is successfully processed. In the case of an exception being raised, the message is not acknowledged
  • manual ack:
    • topic offset is commited when calling msg.ack()
    • topic offset is rewind to the offset of the message when calling msg.nack()

This does not look straightforward reading this documentation, but not acknowledged in the case of auto_commit=False is not the same as explicitely not acknowledged when calling msg.nack(). In the first case, the message is not really nacked, it's acknowledgement status is unset (leading to no new commited offset in Kafka) : that means that if the consumer fetches a new message and the new message succeed, then the failed message will never be consumed (because topic offset is now commited after the successfull message) ; however, if faststream is stopped and restarted before any new message is consumed, the failed message will be consumed again upon new startup (because offset of failed message was not commited). On the other hand, in the case of an explicit msg.nack(), the offset is commited to the offset of the failed message, leading to an immediate retry of the failed message : in this case, any new message in the topic's partition won't be consumed until this message succeed.

I think that:

  • either the difference between not setting acknowledgement status vs explicitely not acknowledged deserves an explaination in the documentation
  • or the behaviour in the case of auto_commit=False may be changed to explicitely not acknowledge message

What do you think ?

Also, in the new documentation for AckPolicy, the statements made for Kafka for the different AckPolicy values are not correct:

  • ACK_FIRST: "Kafka commits offset" (True)
  • ACK: "Kafka: offset commit". That's NOT True. Even after correction from PR 2612 where ACK will lead to the same behaviour as auto_commit=False, there is no offset commited (see abvove). So, the error behaviour described (Ack sent anyway; message not redelivered) is not fully correct (the message may be redelivered if no other successful message is processes in the same topic before restarting application) ; message should be explicitely nacked to ensure no redelivery.
  • REJECT_ON_ERROR: "Kafka commits offset". That's NOT True. Right now, there is no reject() definition in KafkaAckableMessage. That means in case of reject, there is nothing done on Kafka side: the behaviour is the same as for ACK
  • NACK_ON_ERROR: "Kafka commits as fallback". That's True assuming "as fallback" means that offset is moved to the failed message offset, meaning that it is immediately re-consumed (as explained above in explicit nack).

As for the Kafka specific documentation on acknowledgements, we should:

  • either update this documentation with correct/more precise statements for Kafka broker
  • and/or change implementation for ACK / REJECT_ON_ERROR to explicitely nack messages, ensuring they are effectively not redelivered / retried

Please, let me know what you think the correct behaviour would be ? Upon decision, I can contribute a PR if necessary.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions