Fix Iceberg REST compression defaults

[res-life]

Fixes #15274.

Description

Iceberg REST catalog write tests can resolve the Parquet compression codec to gzip, which is not supported by the GPU writer. The previous catalog-level fix is effective with Iceberg 1.9.x and 1.10.x, but Iceberg 1.6.x does not apply table-default.* properties when creating REST tables.

This change sets write.parquet.compression-codec=zstd and write.delete.parquet.compression-codec=zstd directly on every REST test table. It also removes the version-dependent catalog defaults from jenkins/spark-tests.sh and documents the compatibility behavior.

Validation:

• test_insert_into_aqe[unpartitioned] passed.
• test_delete_aqe[unpartitioned] passed with copy-on-write and merge-on-read.
• python -m py_compile integration_tests/src/main/python/iceberg/__init__.py passed.
• bash -n jenkins/spark-tests.sh passed.
• Scala 2.13 build validation is pending and will be completed before the PR is marked ready for review.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (test_insert_into_aqe[unpartitioned] and test_delete_aqe[unpartitioned])
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[res-life]

build

[greptile-apps]

Greptile Summary

This PR fixes an Iceberg REST catalog test failure where Iceberg 1.6.x ignores catalog-level table-default.* properties during table creation, causing the default Parquet codec to fall through to gzip — which the GPU writer does not support.

• Replaces the catalog-level PYSP_TEST_spark_sql_catalog_spark__catalog_table-default_* env-var workaround in spark-tests.sh (effective only with Iceberg ≥1.9.x) with per-table TBLPROPERTIES injected via _BASE_TBLPROPS in iceberg/__init__.py, which works across all tested Iceberg versions.
• The compression properties are added to the shared _BASE_TBLPROPS dict at module-level when ICEBERG_TEST_CATALOG_TYPE=rest, and _BASE_TBLPROPS_SQL is computed after the update, so both the SQL fragment and _build_tblprops() callers automatically pick up zstd for REST catalog tables.
• README.md is updated to document why the table-property approach is necessary and what properties are applied.

Confidence Score: 5/5

Safe to merge — targeted fix to test infrastructure replacing a broken catalog-level workaround with a per-table approach that works across all tested Iceberg versions.

All GPU write tests route through _build_tblprops() or _BASE_TBLPROPS_SQL, both of which pick up the new codec properties. The removed env vars in spark-tests.sh are fully superseded by the table-level properties.

No files require special attention.

Important Files Changed

Filename	Overview
integration_tests/src/main/python/iceberg/init.py	Adds conditional module-level mutation of _BASE_TBLPROPS to inject zstd compression codecs for REST catalog tables; logic is ordered correctly before _BASE_TBLPROPS_SQL is computed.
jenkins/spark-tests.sh	Removes two catalog-level table-default env vars that only worked with Iceberg ≥1.9.x; these are superseded by the table-level properties added in init.py.
integration_tests/README.md	Documentation updated to explain the table-property approach and the Iceberg 1.6.x incompatibility with catalog-level table-default settings.

_{Reviews (1): Last reviewed commit: "Fix Iceberg REST compression defaults" | Re-trigger Greptile}

[sameerz]

build