Passing the `--load-balance` flag changes partitions

[wagenet]

For example:

ember exam --path='test-dist' --split=8 --partition=4 --parallel=2 --load-balance

and

ember exam --path='test-dist' --split=8 --partition=4

do not run the same set of tests.

[tniezurawski]

I know it's almost a year since you raised the issue 🙈 but I'll try to answer with my best knowledge since I'm just trying to learn how everything works in ember-exam, ember-qunit, testem and qunit ecosystem (there are too many little pieces here! 😬).

I think this issue can be fixed only by being clearer in docs about how --load-balance works. Currently, the behavior you described is expected.
Balancing (sorting by weight) is done before splitting. This way partitions receive a "balanced" load. That will likely result in different tests being run compared to an unbalanced run.

[wagenet]

Should specifying a partition with --load-balance not be allowed?

[tniezurawski]

Hmm. I think it should be allowed. --load-balance will "sort" the tests and put them into more-or-less(1) predictable partitions. In case you'd like to re-run your test container because it failed for some reason on the CI, then you'd run something like this from the command line:

ember exam --load-balance --parallel=3 "--split=6" "--partition=4" "--random=o2r8b0njazc"

You can then expect that exactly the same set of tests will be run on your local machine as on the CI. Which will give you a chance to reproduce an issue.

---

(1) If the tests are the same between runs then the set of tests landing in a container are the same. For incremental changes on a PR, the CI will most likely get a slightly different set of tests after balancing for each container (when you add/remove/edit names/rename files of tests).