Skip to content

Conversation

@XiaoyangCai360
Copy link

Purpose

This PR fixes the flaky tests:

  • org.jbpm.process.workitem.bpmn2.JaxWSServiceTaskTest.testServiceInvocationWithMultipleIntParams
  • org.jbpm.process.workitem.bpmn2.JaxWSServiceTaskTest.testServiceInvocationWithMultipleIntIntParams

These tests may pass or fail nondeterministically due to the non-deterministic iteration order of multi-value parameters within the WebService handler, which affects the concatenation order in the returned greeting string.


Why the tests fail (root cause)

Both tests validate the output of a web service invocation that concatenates multiple input parameters (e.g., ["john", "doe"] or [2, 3]) into a greeting message such as "Hello doe, john".

However, the internal parameter iteration (e.g., through a HashMap or array conversion in the WebServiceWorkItemHandler) does not guarantee a consistent order of iteration across JVM executions.
For example:

  • Expected:
Hello doe, john
  • but was:
Hello john, doe

These are semantically equivalent, but because the test compares the raw string output, the equality check is order-sensitive, causing intermittent failures, especially under randomized iteration (e.g., NonDex).


How to reproduce the test failure

Using NonDex, which randomizes collection iteration order:

For example, for the test testServiceInvocationWithMultipleParams,

# Run the test directly (may pass)
mvn -pl jbpm-workitems/jbpm-workitems-bpmn2 -am \
  -Dtest=org.jbpm.process.workitem.bpmn2.JaxWSServiceTaskTest#testServiceInvocationWithMultipleParams\
  -DfailIfNoTests=false test
# Run with NonDex (flakiness appears after a few runs)
mvn -pl jbpm-workitems/jbpm-workitems-bpmn2 -am \
  edu.illinois:nondex-maven-plugin:2.1.7:nondex \
  -Dtest=org.jbpm.process.workitem.bpmn2.JaxWSServiceTaskTest#testServiceInvocationWithMultipleParams \
  -DnondexRuns=10 \
  -DfailIfNoTests=false

Actual Results

Example flaky output:

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   JaxWSServiceTaskTest.testServiceInvocationWithMultipleParams:221 expected:<Hello [doe, john]> but was:<Hello [john, doe]>

These greetings are equivalent but differ in key order, resulting in a false test failure.


Description of the fix

The fix makes the assertions order-insensitive by parsing and normalizing the returned greeting before comparison.

Instead of asserting raw string equality, the test now:

  • Extracts the substring following "Hello ".
  • Splits it into individual tokens by commas.
  • Sorts the tokens alphabetically.
  • Compares the sorted arrays against the expected parameter values (also sorted).

This approach ensures:

  • Both expected parameters are present,
  • Order differences do not cause failures,
  • The greeting prefix remains validated.

Verification of the fix

Using the same test commands:

  • All 10 NonDex runs passed
  • Regular test passed consistently
  • No production code changes made

Key points

  • Only modifies the test code
  • Preserves original behavior and intent.
  • Removes nondeterministic ordering dependence.
  • Stabilizes the tests under randomized conditions.
  • No effect on runtime functionality or handler logic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant