5777 large compaction datafusion error

[m09526]

Make sure you have checked all steps below.

Issue

• My PR fully resolves the following issues. I've referenced an issue in the PR title, for example "Issue 1234 - My
  Feature". Note that before an issue is finished, you can still make a pull request by raising a separate issue
  for your progress.
  • Resolves https://github.com/gchq/sleeper/issues/XXX

Tests

• My PR adds the following tests based on our test strategy OR does not need testing for this extremely good reason:
  • MyUnitTest

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it, or I have linked to a
  separate issue for that below.
• If I have added new Java code, I have added Javadoc that explains it following our conventions and style.
• If I have added or removed any dependencies from the project, I have updated the NOTICES file.