-
Notifications
You must be signed in to change notification settings - Fork 300
Description
Describe the bug
In the documentation for Acknowledgment in Kafka consumers, the following use cases are described (note that the 0.6 documentation still mention the auto_commit parameter despite being deprecated):
auto_commit=True: topic offset is commited on message consumption.auto_commit=False: topic offset is commited after message is successfully processed. In the case of an exception being raised, the message is not acknowledged- manual ack:
- topic offset is commited when calling
msg.ack() - topic offset is rewind to the offset of the message when calling
msg.nack()
- topic offset is commited when calling
This does not look straightforward reading this documentation, but not acknowledged in the case of auto_commit=False is not the same as explicitely not acknowledged when calling msg.nack(). In the first case, the message is not really nacked, it's acknowledgement status is unset (leading to no new commited offset in Kafka) : that means that if the consumer fetches a new message and the new message succeed, then the failed message will never be consumed (because topic offset is now commited after the successfull message) ; however, if faststream is stopped and restarted before any new message is consumed, the failed message will be consumed again upon new startup (because offset of failed message was not commited). On the other hand, in the case of an explicit msg.nack(), the offset is commited to the offset of the failed message, leading to an immediate retry of the failed message : in this case, any new message in the topic's partition won't be consumed until this message succeed.
I think that:
- either the difference between not setting acknowledgement status vs explicitely not acknowledged deserves an explaination in the documentation
- or the behaviour in the case of
auto_commit=Falsemay be changed to explicitely not acknowledge message
What do you think ?
Also, in the new documentation for AckPolicy, the statements made for Kafka for the different AckPolicy values are not correct:
ACK_FIRST: "Kafka commits offset" (True)ACK: "Kafka: offset commit". That's NOT True. Even after correction from PR 2612 whereACKwill lead to the same behaviour asauto_commit=False, there is no offset commited (see abvove). So, the error behaviour described (Ack sent anyway; message not redelivered) is not fully correct (the message may be redelivered if no other successful message is processes in the same topic before restarting application) ; message should be explicitely nacked to ensure no redelivery.REJECT_ON_ERROR: "Kafka commits offset". That's NOT True. Right now, there is noreject()definition inKafkaAckableMessage. That means in case of reject, there is nothing done on Kafka side: the behaviour is the same as forACKNACK_ON_ERROR: "Kafka commits as fallback". That's True assuming "as fallback" means that offset is moved to the failed message offset, meaning that it is immediately re-consumed (as explained above in explicit nack).
As for the Kafka specific documentation on acknowledgements, we should:
- either update this documentation with correct/more precise statements for Kafka broker
- and/or change implementation for
ACK/REJECT_ON_ERRORto explicitely nack messages, ensuring they are effectively not redelivered / retried
Please, let me know what you think the correct behaviour would be ? Upon decision, I can contribute a PR if necessary.