[LIVY-1019] Deprecate Spark 2 in livy

[ArnavBalyan]

What changes were proposed in this pull request?

• Spark 2.x is at end of life and no longer supported, newer Spark versions (Spark 4) are now available.
• Deprecate Spark 2.x from Livy
• Also offboard Scala 211 support

How was this patch tested?

• CI and local builds are successful

[ArnavBalyan]

cc @gyogal @lmccay could you ptal thanks!

[gyogal]

Thank you for your PR @ArnavBalyan! This could be an important step of cleaning up unneeded features in an upcoming release. The change is quite big, so it will take some time to review. In the meantime, can I ask why you chose Spark version 3.3.4 in the main pom.xml and in README.md? The latest version should work with 3.5.6, but Spark 3.5.7 is already available as well.

[ArnavBalyan]

Thank you for your PR @ArnavBalyan! This could be an important step of cleaning up unneeded features in an upcoming release. The change is quite big, so it will take some time to review. In the meantime, can I ask why you chose Spark version 3.3.4 in the main pom.xml and in README.md? The latest version should work with 3.5.6, but Spark 3.5.7 is already available as well.

Hi @gyogal thanks so much, really appreciate it. For Spark, looks like 3.3.4 was the latest version that shipped hadoop 2 binaries. Moving to 3.5.7 may need us to bump to hadoop 3, was a bit unsure if this would have any side effects since this was getting a considerably large change. Let me know if we are comfortable directly moving to hadoop 3 happy to update. Thanks for taking a look

[gyogal]

For Spark, looks like 3.3.4 was the latest version that shipped hadoop 2 binaries. Moving to 3.5.7 may need us to bump to hadoop 3, was a bit unsure if this would have any side effects since this was getting a considerably large change. Let me know if we are comfortable directly moving to hadoop 3 happy to update. Thanks for taking a look

Oh right, it makes sense now, thanks! In #478 it has already been changed to version 3 in the spark3 profile, so I think that should be OK. Alternatively, it could be done in a follow-up change later on.

[ArnavBalyan]

For Spark, looks like 3.3.4 was the latest version that shipped hadoop 2 binaries. Moving to 3.5.7 may need us to bump to hadoop 3, was a bit unsure if this would have any side effects since this was getting a considerably large change. Let me know if we are comfortable directly moving to hadoop 3 happy to update. Thanks for taking a look

Oh right, it makes sense now, thanks! In #478 it has already been changed to version 3 in the spark3 profile, so I think that should be OK. Alternatively, it could be done in a follow-up change later on.

Thanks @gyogal I can bump it up as a follow up change if it looks good to you?

[ArnavBalyan]

cc @gyogal gentle reminder thanks! 😀

[gyogal]

@ArnavBalyan , I think both this and the Python 3 commit could be merged after branch-0.9 is created. That would mean that the upcoming 0.9 release would still retain support, but the cleanup work can be started in preparation for the release afterwards. Please let me know what you think!

[ArnavBalyan]

Hi @gyogal thanks for taking a look! Makes perfect sense, will wait until we do the release. Let me know if you need support with release

[gyogal]

Hi @ArnavBalyan , the main Livy version has been updated to 0.10-incubating-SNAPSHOT, however this caused some conflicts in your PR. Could you please resolve them? Once that is done, I think this could be merged into version 0.10-incubating.