new APIs to materialize params in the ZL_Compressor #311
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@Victor-C-Zhang has exported this pull request. If you are a Meta employee, you can view the originating Diff in D89093023. |
Victor-C-Zhang
force-pushed
the
export-D89093023
branch
from
c5c355b to
1eb04d7
Compare
Victor-C-Zhang
added a commit
to Victor-C-Zhang/openzl-1
that referenced
this pull request
Dec 22, 2025
Summary: Create a new API to materialize params from localparams. This relies on a new functionality associated with the ZL_Compressor to manage allocated objects. A materialized param is a new type of local parameter object that is fundamentally an object in memory. This object's constructor and destructor functions shall be specified in the new `ZL_MaterializerDesc` struct passed to a node registration. At the time of registration and subsequent node parameterizations^, the constructor function is called to create this new object in memory. This new object will live as long as the ZL_Compressor. When a CCtx uses this Compressor to compress, the associated node will be able to access the materialized object using the `ZL_Encoder_getMaterializedParam()` function. As a consequence of these two requirements > the generation *must* be deterministic and hermetic > the ref ZL_CCtx may provide concurrent access to materialized objects We may optimize construction of materialized params by eliding duplicate calls of the same function on the same params. Instead, we may just return two pointers to an object we create once. Wow! Equality of LocalParams uses its built-in hash function. ^ not all parameterizations will necessarily result in a construction. See next diff in stack. Differential Revision: D89093023
Victor-C-Zhang
force-pushed
the
export-D89093023
branch
from
1eb04d7 to
702119b
Compare
Victor-C-Zhang
added a commit
to Victor-C-Zhang/openzl-1
that referenced
this pull request
Dec 22, 2025
Summary: Create a new API to materialize params from localparams. This relies on a new functionality associated with the ZL_Compressor to manage allocated objects. A materialized param is a new type of local parameter object that is fundamentally an object in memory. This object's constructor and destructor functions shall be specified in the new `ZL_MaterializerDesc` struct passed to a node registration. At the time of registration and subsequent node parameterizations^, the constructor function is called to create this new object in memory. This new object will live as long as the ZL_Compressor. When a CCtx uses this Compressor to compress, the associated node will be able to access the materialized object using the `ZL_Encoder_getMaterializedParam()` function. As a consequence of these two requirements > the generation *must* be deterministic and hermetic > the ref ZL_CCtx may provide concurrent access to materialized objects We may optimize construction of materialized params by eliding duplicate calls of the same function on the same params. Instead, we may just return two pointers to an object we create once. Wow! Equality of LocalParams uses its built-in hash function. ^ not all parameterizations will necessarily result in a construction. See next diff in stack. Differential Revision: D89093023
Victor-C-Zhang
force-pushed
the
export-D89093023
branch
from
702119b to
8bcdeb2
Compare
Victor-C-Zhang
added a commit
to Victor-C-Zhang/openzl-1
that referenced
this pull request
Dec 22, 2025
Summary: Create a new API to materialize params from localparams. This relies on a new functionality associated with the ZL_Compressor to manage allocated objects. A materialized param is a new type of local parameter object that is fundamentally an object in memory. This object's constructor and destructor functions shall be specified in the new `ZL_MaterializerDesc` struct passed to a node registration. At the time of registration and subsequent node parameterizations^, the constructor function is called to create this new object in memory. This new object will live as long as the ZL_Compressor. When a CCtx uses this Compressor to compress, the associated node will be able to access the materialized object using the `ZL_Encoder_getMaterializedParam()` function. As a consequence of these two requirements > the generation *must* be deterministic and hermetic > the ref ZL_CCtx may provide concurrent access to materialized objects We may optimize construction of materialized params by eliding duplicate calls of the same function on the same params. Instead, we may just return two pointers to an object we create once. Wow! Equality of LocalParams uses its built-in hash function. ^ not all parameterizations will necessarily result in a construction. See next diff in stack. Differential Revision: D89093023
Victor-C-Zhang
added a commit
to Victor-C-Zhang/openzl-1
that referenced
this pull request
Dec 22, 2025
Summary: Create a new API to materialize params from localparams. This relies on a new functionality associated with the ZL_Compressor to manage allocated objects. A materialized param is a new type of local parameter object that is fundamentally an object in memory. This object's constructor and destructor functions shall be specified in the new `ZL_MaterializerDesc` struct passed to a node registration. At the time of registration and subsequent node parameterizations^, the constructor function is called to create this new object in memory. This new object will live as long as the ZL_Compressor. When a CCtx uses this Compressor to compress, the associated node will be able to access the materialized object using the `ZL_Encoder_getMaterializedParam()` function. As a consequence of these two requirements > the generation *must* be deterministic and hermetic > the ref ZL_CCtx may provide concurrent access to materialized objects We may optimize construction of materialized params by eliding duplicate calls of the same function on the same params. Instead, we may just return two pointers to an object we create once. Wow! Equality of LocalParams uses its built-in hash function. ^ not all parameterizations will necessarily result in a construction. See next diff in stack. Differential Revision: D89093023
Summary: Create a new API to materialize params from localparams. This relies on a new functionality associated with the ZL_Compressor to manage allocated objects. A materialized param is a new type of local parameter object that is fundamentally an object in memory. This object's constructor and destructor functions shall be specified in the new `ZL_MaterializerDesc` struct passed to a node registration. At the time of registration and subsequent node parameterizations^, the constructor function is called to create this new object in memory. This new object will live as long as the ZL_Compressor. When a CCtx uses this Compressor to compress, the associated node will be able to access the materialized object using the `ZL_Encoder_getMaterializedParam()` function. As a consequence of these two requirements > the generation *must* be deterministic and hermetic > the ref ZL_CCtx may provide concurrent access to materialized objects We may optimize construction of materialized params by eliding duplicate calls of the same function on the same params. Instead, we may just return two pointers to an object we create once. Wow! Equality of LocalParams uses its built-in hash function. ^ not all parameterizations will necessarily result in a construction. See next diff in stack. Differential Revision: D89093023
Victor-C-Zhang
force-pushed
the
export-D89093023
branch
from
8bcdeb2 to
cd91531
Compare
Victor-C-Zhang
added a commit
to Victor-C-Zhang/openzl-1
that referenced
this pull request
Dec 22, 2025
Summary: Create a new API to materialize params from localparams. This relies on a new functionality associated with the ZL_Compressor to manage allocated objects. A materialized param is a new type of local parameter object that is fundamentally an object in memory. This object's constructor and destructor functions shall be specified in the new `ZL_MaterializerDesc` struct passed to a node registration. At the time of registration and subsequent node parameterizations^, the constructor function is called to create this new object in memory. This new object will live as long as the ZL_Compressor. When a CCtx uses this Compressor to compress, the associated node will be able to access the materialized object using the `ZL_Encoder_getMaterializedParam()` function. As a consequence of these two requirements > the generation *must* be deterministic and hermetic > the ref ZL_CCtx may provide concurrent access to materialized objects We may optimize construction of materialized params by eliding duplicate calls of the same function on the same params. Instead, we may just return two pointers to an object we create once. Wow! Equality of LocalParams uses its built-in hash function. ^ not all parameterizations will necessarily result in a construction. See next diff in stack. Differential Revision: D89093023
Victor-C-Zhang
added a commit
to Victor-C-Zhang/openzl-1
that referenced
this pull request
Dec 24, 2025
Summary: Create a new API to materialize params from localparams. This relies on a new functionality associated with the ZL_Compressor to manage allocated objects. A materialized param is a new type of local parameter object that is fundamentally an object in memory. This object's constructor and destructor functions shall be specified in the new `ZL_MaterializerDesc` struct passed to a node registration. At the time of registration and subsequent node parameterizations^, the constructor function is called to create this new object in memory. This new object will live as long as the ZL_Compressor. When a CCtx uses this Compressor to compress, the associated node will be able to access the materialized object using the `ZL_Encoder_getMaterializedParam()` function. As a consequence of these two requirements > the generation *must* be deterministic and hermetic > the ref ZL_CCtx may provide concurrent access to materialized objects We may optimize construction of materialized params by eliding duplicate calls of the same function on the same params. Instead, we may just return two pointers to an object we create once. Wow! Equality of LocalParams uses its built-in hash function. ^ not all parameterizations will necessarily result in a construction. See next diff in stack. Differential Revision: D89093023
Victor-C-Zhang
added a commit
to Victor-C-Zhang/openzl-1
that referenced
this pull request
Dec 26, 2025
Summary: Create a new API to materialize params from localparams. This relies on a new functionality associated with the ZL_Compressor to manage allocated objects. A materialized param is a new type of local parameter object that is fundamentally an object in memory. This object's constructor and destructor functions shall be specified in the new `ZL_MaterializerDesc` struct passed to a node registration. At the time of registration and subsequent node parameterizations^, the constructor function is called to create this new object in memory. This new object will live as long as the ZL_Compressor. When a CCtx uses this Compressor to compress, the associated node will be able to access the materialized object using the `ZL_Encoder_getMaterializedParam()` function. As a consequence of these two requirements > the generation *must* be deterministic and hermetic > the ref ZL_CCtx may provide concurrent access to materialized objects We may optimize construction of materialized params by eliding duplicate calls of the same function on the same params. Instead, we may just return two pointers to an object we create once. Wow! Equality of LocalParams uses its built-in hash function. ^ not all parameterizations will necessarily result in a construction. See next diff in stack. Differential Revision: D89093023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Create a new API to materialize params from localparams. This relies on a new functionality associated with the ZL_Compressor to manage allocated objects.
A materialized param is a new type of local parameter object that is fundamentally an object in memory. This object's constructor and destructor functions shall be specified in the new
ZL_MaterializerDescstruct passed to a node registration.At the time of registration and subsequent node parameterizations^, the constructor function is called to create this new object in memory. This new object will live as long as the ZL_Compressor. When a CCtx uses this Compressor to compress, the associated node will be able to access the materialized object using the
ZL_Encoder_getMaterializedParam()function.^ not all parameterizations will necessarily result in a construction. See next diff in stack.
Differential Revision: D89093023