[FSTORE-1921] Improve documentation building process

[aversey]

This PR adds/fixes/changes...

• please summarize your changes to the code
• and make sure to include all changes to user-facing APIs

JIRA Issue: -

Priority for Review: -

Related PRs: -

How Has This Been Tested?

• Unit Tests
• Integration Tests
• Manual Tests on VM

Checklist For The Assigned Reviewer:

- [ ] Checked if merge conflicts with master exist
- [ ] Checked if stylechecks for Java and Python pass
- [ ] Checked if all docstrings were added and/or updated appropriately
- [ ] Ran spellcheck on docstring
- [ ] Checked if guides & concepts need to be updated
- [ ] Checked if naming conventions for parameters and variables were followed
- [ ] Checked if private methods are properly declared and used
- [ ] Checked if hard-to-understand areas of code are commented
- [ ] Checked if tests are effective
- [ ] Built and deployed changes on dev VM and tested manually
- [x] (Checked if all type annotations were added and/or updated appropriately)

[Copilot]

Pull request overview

This PR modernizes Python type hints across the codebase by replacing typing.Optional and typing.Union with their PEP 604 equivalents (X | None and X | Y), updates import patterns to use TYPE_CHECKING guards for circular dependencies, and improves documentation formatting by converting to Google-style docstrings with proper markdown formatting.

Key Changes:

• Replaced Optional[X] with X | None and Union[X, Y] with X | Y throughout the codebase
• Moved type-only imports under TYPE_CHECKING guards to avoid circular import issues
• Converted docstrings from custom format to Google-style with proper markdown
• Removed unnecessary else clauses after return statements
• Updated string formatting to use f-strings

Reviewed changes

Copilot reviewed 247 out of 364 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
python/hsfs/core/feature_descriptive_statistics.py	Updated type hints and moved imports under TYPE_CHECKING guard
python/hsfs/core/external_feature_group_engine.py	Fixed string concatenation in error messages
python/hsfs/core/explicit_provenance.py	Modernized type hints, simplified conditionals, and improved docstrings
python/hsfs/core/expectation_suite_engine.py	Updated type hints and moved imports under TYPE_CHECKING
python/hsfs/core/expectation_suite_api.py	Removed unnecessary typing imports and updated type hints
python/hsfs/core/expectation_engine.py	Added TYPE_CHECKING guard and simplified conditionals
python/hsfs/core/expectation_api.py	Updated docstrings and fixed punctuation
python/hsfs/core/delta_engine.py	Modernized type hints and removed unnecessary else clauses
python/hsfs/core/data_source_data.py	Updated type hints and simplified docstrings
python/hsfs/core/data_source.py	Simplified conditionals and updated type hints
python/hsfs/core/arrow_flight_client.py	Added TYPE_CHECKING guard and simplified boolean logic
python/hsfs/constructor/serving_prepared_statement.py	Updated type hints across all methods
python/hsfs/constructor/query.py	Comprehensive type hint updates and docstring improvements
python/hsfs/constructor/prepared_statement_parameter.py	Updated type hints for all properties
python/hsfs/constructor/join.py	Simplified type hints
python/hsfs/constructor/hudi_feature_group_alias.py	Updated type hints
python/hsfs/constructor/fs_query.py	Updated type hints throughout
python/hsfs/constructor/filter.py	Modernized type hints and simplified error messages
python/hsfs/constructor/external_feature_group_alias.py	Updated type hints
python/hsfs/builtin_transformations.py	Removed unnecessary list comprehensions
python/hsfs/init.py	Updated to f-string formatting
python/hopsworks_common/util.py	Comprehensive type hint updates and docstring improvements
python/hopsworks_common/user.py	Updated type hints in dataclass
python/hopsworks_common/usage.py	Simplified conditional logic
python/hopsworks_common/triggered_alert.py	Improved property docstrings
python/hopsworks_common/secret.py	Updated docstring formatting
python/hopsworks_common/search_results.py	Modernized type hints across search result classes
python/hopsworks_common/project.py	Updated method return types and improved docstrings
python/hopsworks_common/kafka_topic.py	Updated type hints and docstrings
python/hopsworks_common/kafka_schema.py	Improved docstring formatting
python/hopsworks_common/job_schedule.py	Fixed docstring formatting
python/hopsworks_common/job.py	Added Literal types for state/status and improved docstrings
python/hopsworks_common/git_*.py	Updated type hints and docstrings across all git-related modules
python/hopsworks_common/flink_cluster.py	Added Literal types and improved docstrings
python/hopsworks_common/execution.py	Updated type hints and docstrings
python/hopsworks_common/environment.py	Added TYPE_CHECKING guard and updated docstrings
python/hopsworks_common/engine/*.py	Updated type hints and docstrings
python/hopsworks_common/decorators.py	Removed unnecessary else clause
python/hopsworks_common/core/*.py	Comprehensive type hint and docstring updates
docs/transformation_statistics.md	Added new documentation file

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[aversey]

See logicalclocks/logicalclocks.github.io#532 for the related PR.

[vatj]

Nice and thorough work, I think I somewhat understand the ideas underlying the PR so I am happy with merging

[aversey]

The branch passed:

./workflows/hopsworks/test_secrets.py
./workflows/hopsworks/test_python_job_execution.py
./workflows/hopsworks/test_opensearch.py
./workflows/hopsworks/test_environment.py
./workflows/hopsworks/test_git.py
./workflows/feature_store/python_driver/test_get_feature_vectors.py::test_feature_vector_transformations
./workflows/feature_store/pyspark_driver/test_get_feature_vectors.py::test_feature_vector_transformations
./workflows/feature_store/python_driver/test_get_feature_vectors.py::test_manual_all_return_types
./workflows/feature_store/pyspark_driver/test_get_feature_vectors.py::test_manual_all_return_types
./workflows/feature_store/python_driver/test_batch.py
./workflows/feature_store/pyspark_driver/test_batch.py
./workflows/feature_store/python_driver/test_feature_group_insert.py
./workflows/feature_store/pyspark_driver/test_feature_group_insert.py::test_pyspark_feature_group_insert

But failed (it passed the other one, not HUDI):

workflows/feature_store/python_driver/test_feature_group_transformation_functions.py::test_many_to_many_on_demand_transformation_with_model_dependent_transformations[HUDI]

With:

WARNING  hopsworks_common.client.online_store_rest_client:vector_server.py:337 Online Store Rest Client is already initialised. To reset connection or/and override configuration, use reset_online_store_rest_client flag.
ERROR    hsfs.core.online_store_rest_client_api:online_store_rest_client_api.py:190 Received response from RonDB Rest Server with status code 401
ERROR    hsfs.core.online_store_rest_client_api:online_store_rest_client_api.py:193 Response: API key not authorized to access fg_tf_kiutf

I guess it is not an error introduced in this PR.

[aversey]

workflows/feature_store/python_driver/test_feature_pipeline.py::test_feature_pipeline[True-DELTA]

Is also failing (another one in test_feature_pipeline is ok). These passed:

./workflows/feature_store/python_driver/test_feature_logging.py::test_logging_one_row_entire_dataframe_and_some_seperate
./workflows/feature_store/pyspark_driver/test_feature_logging.py::test_logging_one_row_entire_dataframe_and_some_seperate
./workflows/feature_store/python_driver/test_feature_logging.py::test_implicit_logging_batch_data
./workflows/feature_store/pyspark_driver/test_feature_logging.py::test_implicit_logging_batch_data

[aversey]

./workflows/feature_store/python_driver/test_training_data.py
./workflows/feature_store/pyspark_driver/test_training_data.py
./workflows/feature_store/pyspark_driver/test_statistics.py
./workflows/feature_store/python_driver/test_statistics.py
./workflows/feature_store/python_driver/test_similarity_search.py::test_all_data_types

Have also passed.

[aversey]

It also passed:

./workflows/model_serving/python_driver/external/test_deploy_model_and_predict.py::test_deploy_and_predict_python_model_with_logging_with_kserve
./workflows/model_serving/python_driver/external/test_deploy_model_and_predict.py::test_deploy_and_call_endpoint_with_kserve_with_feature_logging
./workflows/model_serving/python_driver/internal/test_deploy_model_and_predict.py::test_deploy_and_test_endpoint_with_kserve_with_feature_logging
./workflows/model_serving/python_driver/internal/test_deploy_model_and_predict.py::test_deploy_and_predict_python_model_with_kserve_with_feature_logging