Resolve incorrect event routing in heterogeneous Sequencing Policy setup among Event Handling Components - #4769
Conversation
A PooledStreamingEventProcessor hosts several EventHandlingComponents behind a single WorkPackage per Segment. Each component may declare its own SequencingPolicy, so for one and the same event two components can resolve different sequence identifiers and therefore legitimately belong to different segments. This is a supported topology, not a misconfiguration. Routing was decided in two disconnected places that disagreed: * Admission (scheduling): DefaultWorkPackageEventFilter.canHandle(...) admits an event into a segment's queue when *any* component's sequence identifier hashes into that segment (sequenceIdentifiersFor(...) + anyMatch). That is a per-(segment, event) boolean; it does not remember *which* component caused the match. * Handling: ProcessorEventHandlingComponents.handle(...) then invoked *every* component that supports the event and never consulted the segment, even though the Segment is present in the ProcessingContext under Segment.RESOURCE_KEY. The consequence: when components use different sequencing policies, the same event is admitted to multiple segments (each because a *different* component matched), and every admitting segment then runs *all* supporting components. A component whose identifier belongs to segment 1 was still executed in segment 0 (and vice versa), so each component handled the event once per segment instead of exactly once. That duplicates side effects (projections, notifications, mailings, ...) and violates the single-writer-per-sequence guarantee the sequencing policy is meant to provide. With the common setup, where every component yields the same identifier for an event, the identifier set collapses to a single segment and the defect stays hidden, which is why it went unnoticed. Handling now applies the same per-component routing decision the filter already relies on: a component handles an event only when its own sequence identifier hashes into the segment currently attached to the context (canSegmentHandle). Admission and handling therefore agree. When no segment is attached (for example a SubscribingEventProcessor), handling is not segmented and every supporting component runs, preserving existing behaviour. sequenceIdentifiersFor(...) additionally ignores components that do not support the event, so a non-supporting component can no longer make a segment claim an event it will never handle, keeping the admission and handling views of "which components route here" consistent. Covered by unit tests in ProcessorEventHandlingComponentsTest and end-to-end tests in PooledStreamingEventProcessorTest that fail on the previous behaviour.
hatzlj
left a comment
There was a problem hiding this comment.
nice catch 👍, no blocking issues for me
| return segment == null | ||
| || segment.matches(component.sequenceIdentifierFor(event, context)); | ||
| } |
There was a problem hiding this comment.
Does it probably make sense to reuse the SegmentMatcher logic here or create a shared helper, to make sure the same logic as for scheduling (as you state in the Javadoc) is invoked in both places.
I mean, the logic is trivial, but as you point out in the javadoc for this method, it should "mirror the routing decision made while scheduling the event".
| * @param context the processing context in which the event is handled | ||
| * @return {@code true} when the component should handle the event within the given segment | ||
| */ | ||
| private static boolean canSegmentHandle(@Nullable Segment segment, |
There was a problem hiding this comment.
| private static boolean canSegmentHandle(@Nullable Segment segment, | |
| private static boolean canHandleInSegment(@Nullable Segment segment, |
There was a problem hiding this comment.
I prefer this rename suggestion
smcvb
left a comment
There was a problem hiding this comment.
Once the two nits from @hatzlj have been tackled, you're free to merge this in, @MateuszNaKodach!
Ow, and I think we should port this fix to axon-5.2.x as well, so that it'll be a part of 5.2.1. Thus, would you mind following this up with a port PR once done?
| * @param context the processing context in which the event is handled | ||
| * @return {@code true} when the component should handle the event within the given segment | ||
| */ | ||
| private static boolean canSegmentHandle(@Nullable Segment segment, |
There was a problem hiding this comment.
I prefer this rename suggestion
| return segment == null | ||
| || segment.matches(component.sequenceIdentifierFor(event, context)); | ||
| } |
The per-component segment gate applied while handling a batch computed the match inline as segment.matches(component.sequenceIdentifierFor(...)). That was a second, independent copy of the routing logic already used when an event is assigned to a segment, so the two could silently drift apart. Perform the check through the same SegmentMatcher instead. Assigning an event to a segment and handling that event now share one routing implementation: a component handles an event only when its sequence identifier hashes into the segment currently processing the batch. Behaviour is unchanged; the intent is a single source of truth for the segment routing decision.
The helper reads more naturally at its call site as "can this component handle the event in this segment", matching how it is used inside handle(). No behavioural change.
d503b21 to
91535dd
Compare
[Port] fix(messaging): resolve incorrect event routing in heterogeneous Sequencing Policy setup among Event Handling Components (#4769)
Route events per component when handling across segments
Summary
A
PooledStreamingEventProcessorcould handle the same event more than once and in thewrong segment whenever a single processor hosted multiple
EventHandlingComponents that usedifferent
SequencingPolicyinstances. This PR makes event handling apply the sameper-component segment routing that the admission filter already relies on, so each component
handles an event exactly once, in the single segment it belongs to.
Background: how a pooled processor routes an event
A
PooledStreamingEventProcessorruns oneWorkPackageperSegment, and every work packageshares the same set of
EventHandlingComponents (wrapped inProcessorEventHandlingComponents).Which segment an event belongs to is derived from a component's sequence identifier
(
SequencingPolicy.sequenceIdentifierFor(...)), hashed and masked against the segment.Crucially, routing is decided in two separate phases:
sequenceDiagram participant Coord as Coordinator participant WP as WorkPackage for Segment S participant Filter as DefaultWorkPackageEventFilter participant PEHC as ProcessorEventHandlingComponents Note over Coord,Filter: Phase 1 - Scheduling / admission Coord->>WP: scheduleEvent(E) WP->>Filter: canHandle(E, ctx, Segment S) Filter-->>WP: boolean - does ANY component identifier hash to S Note over WP: result stored per entry, E enqueued only if true Note over WP,PEHC: Phase 2 - Handling, later, per batch WP->>PEHC: handle(batch, ctx) with Segment S in context Note over PEHC: BEFORE - runs EVERY supporting component, segment ignored Note over PEHC: AFTER - runs only components whose identifier hashes to SThe important detail: admission produces a single
booleanper(segment, event). It answers"does this event belong in this segment for anybody?" using
anyMatchover all componentidentifiers. It does not record which component caused the match.
The problem
Each
EventHandlingComponentmay declare its ownSequencingPolicy. So for one and the sameevent, component A and component B can resolve different sequence identifiers and therefore
legitimately belong to different segments. This is a supported topology, not a
misconfiguration.
Before this change:
DefaultWorkPackageEventFilter.canHandle) admitted the event into a segment'squeue when any component's identifier hashed into that segment.
ProcessorEventHandlingComponents.handle) then invoked every component thatsupports(...)the event and never consulted the segment - even though theSegmentis sittingin the
ProcessingContextunderSegment.RESOURCE_KEY.Because the per-component reason for admission was discarded, every admitting segment re-ran all
components:
flowchart TD E["Event E<br/>A -> hashes to segment 0<br/>B -> hashes to segment 1"] E --> F0 E --> F1 subgraph S0["WorkPackage - Segment 0"] F0{"Filter: any identifier hashes to seg 0?<br/>YES (via A)"} H0["handle(): run ALL supporting components"] A0["A handles E (correct)"] B0["B handles E (WRONG - B belongs to seg 1)"] F0 -->|admitted| H0 H0 --> A0 H0 --> B0 end subgraph S1["WorkPackage - Segment 1"] F1{"Filter: any identifier hashes to seg 1?<br/>YES (via B)"} H1["handle(): run ALL supporting components"] A1["A handles E (WRONG - A belongs to seg 0)"] B1["B handles E (correct)"] F1 -->|admitted| H1 H1 --> A1 H1 --> B1 endNet result: A is handled twice and B is handled twice (once per segment) instead of exactly
once each. This duplicates side effects (projections, notifications, mailings, ...) and breaks the
single-writer-per-sequence guarantee the sequencing policy is meant to provide.
Why it stayed hidden
In the common configuration every component yields the same identifier for a given event
(e.g. all sequence by aggregate id, or all use the default policy). Then
sequenceIdentifiersForcollapses to a single-element set that hashes to exactly one segment: only that segment admits the
event and all components run once. No duplication. The defect only appears once components use
heterogeneous sequencing policies for the same event - which is exactly the case that was
untested.
The fix
Handling now applies the same per-component routing decision the filter already relies on: a
component handles an event only when its own sequence identifier hashes into the segment currently
attached to the context.
flowchart TD E["Event E<br/>A -> hashes to segment 0<br/>B -> hashes to segment 1"] E --> F0 E --> F1 subgraph S0["WorkPackage - Segment 0"] F0{"Filter: any identifier hashes to seg 0?<br/>YES (via A)"} H0["handle(): canSegmentHandle(seg0, component) per component"] A0["A: id hashes to 0 -> HANDLES E"] B0["B: id hashes to 1 -> skipped"] F0 -->|admitted| H0 H0 --> A0 H0 --> B0 end subgraph S1["WorkPackage - Segment 1"] F1{"Filter: any identifier hashes to seg 1?<br/>YES (via B)"} H1["handle(): canSegmentHandle(seg1, component) per component"] A1["A: id hashes to 0 -> skipped"] B1["B: id hashes to 1 -> HANDLES E"] F1 -->|admitted| H1 H1 --> A1 H1 --> B1 endNet result: A once, B once - each in the single segment it belongs to. Admission and handling
now agree.
Concretely, in
ProcessorEventHandlingComponents:Per-component segment gate while handling.
handle(...)reads theSegmentfrom thecontext (
Segment.fromContext) and invokes a component only when it supports the event andcanSegmentHandle(segment, component, event, context)is true. The check issegment.matches(component.sequenceIdentifierFor(event, context))- the identical computationthe filter performs, so the two phases can no longer diverge.
Non-segmented fallback preserved. When no
Segmentis attached to the context (for examplea
SubscribingEventProcessor),canSegmentHandlereturnstrueand every supporting componenthandles the event, exactly as before.
sequenceIdentifiersFor(...)now ignores non-supporting components. A component that doesnot support the event no longer contributes an identifier, so it can no longer cause a segment
to claim an event it will never handle. This keeps the admission and handling views of "which
components route here" symmetric.
DefaultWorkPackageEventFilter,WorkPackage,SegmentandSegmentMatcherare unchanged.Testing
Added, following TDD (written to fail against the previous behaviour first):
Unit tests (
ProcessorEventHandlingComponentsTest->@Nested SegmentAwareRouting)segment (one test per segment);
sequenceIdentifiersFor(...)excludes components that do not support the event.End-to-end tests (
PooledStreamingEventProcessorTest->@Nested SegmentRouting,initialSegmentCount(2))segment;
(guards against over-restriction).
Verification:
tests red (duplicate deliveries), confirming they are genuine regression guards.
messagingmodule: 4139 tests, 0 failures, 0 errors (505 skipped).Compatibility
mask == 0) are unaffected:Segment.matchesalwaysreturns
true, so every supporting component still handles every supported event.SubscribingEventProcessorand any non-segmented handling path are unaffected (noSegmentincontext -> all supporting components run).
sequencing policies - the case that was previously duplicating deliveries.