Skip to content

feat: opt-in pipeline fit optimisations, native bucketize, and sampled fitting support with caching - #69

Open
ConorWorthington wants to merge 30 commits into
ExpediaGroup:mainfrom
ConorWorthington:main
Open

feat: opt-in pipeline fit optimisations, native bucketize, and sampled fitting support with caching#69
ConorWorthington wants to merge 30 commits into
ExpediaGroup:mainfrom
ConorWorthington:main

Conversation

@ConorWorthington

Copy link
Copy Markdown
Contributor

Description

Provide a short description of the PR changes.

The below checklists come from the docs page on adding new transformers here

Keras Layer Checklist

Verify that:

  • The new Keras layer extends BaseLayer
  • The _call method has been implemented in the new layer.
  • The compatible_dtypes property is defined in the new layer.
  • The new layer is decorated with @tf.keras.utils.register_keras_serializable(package=kamae.__name__).
  • The new layer takes a name, input_dtype, and output_dtype as arguments to the constructor and that this is passed to the super constructor.
  • The Keras layer is serializable. I have implemented the get_config method.
  • There are unit tests of the new layer.
  • There is a specific test of layer serialisation added here.
  • The new layer is imported in the init.py file in the layers directory.

Spark Transformer/Estimator Checklist

Verify that:

  • The new Spark Transformer extends BaseTransformer.
  • If the new transform needs a fit method, a Spark Estimator has been implemented that extends BaseEstimator.
  • The instructions in the above docs page have been followed for the __init__ and setParams methods.
  • The transformer uses one of the input/output mixin classes from base.py.
  • If the new transformer requires more parameters that would need to be serialised to the Spark ML pipeline, there is a implemented parameter class by extending the Params class here.
  • The compatible_dtypes property has been implemented to specify the input/output data types that my transformer/estimator supports.
  • A Keras subclassed layer is returned in the transformer's get_tf_layer method.
  • There are unit tests of the new transform. In particular, there are parity tests between the Spark and Keras implementations.
  • The new transformer/estimator is imported in the init.py file in the transformers/estimators directory.

Finally, please verify that:

  • There is a new entry (alphabetical order) in the README table describing the new layer/transformer

ConorWorthington and others added 24 commits June 29, 2026 16:31
Plans are too slow to materialise - this is our attempt to speed it up
Wrap the moments aggregation in StandardScale, SingleFeatureArrayStandardScale
and ConditionalStandardScale estimators in a guarded persist/unpersist so the
array-size probe and the aggregation reuse a materialised result instead of
re-scanning the upstream lineage twice. Repair the incomplete persist edit in
ConditionalStandardScale._fit.

Add checkpointInterval / pruneInputColumns coverage to the pipeline tests and a
checkpoint directory to the spark_session fixture. Surface estimator fit errors
as RuntimeError chained from the original exception.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@ConorWorthington
ConorWorthington requested a review from a team as a code owner August 5, 2026 13:56
@ConorWorthington ConorWorthington changed the title feat: opt-in pipeline fit optimisations, native bucketize, and sampled fitting feat: opt-in pipeline fit optimisations, native bucketize, and sampled fitting support with caching Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant