feat(capi): add xet_capi C ABI crate for hf-xet#881
Draft
assafvayner wants to merge 30 commits into
Draft
Conversation
…add stream task_id
Previously xet_op_take_* freed the op internally on success/error but not on the wrong-variant path, an inconsistent contract that made double-frees easy (freeing an already-consumed op joins an already-joined worker thread and panics with ESRCH). Switch to a single predictable rule: take_* never frees; the caller always frees every op exactly once with xet_op_free. This is the RAII/defer-friendly model the language bindings want.
Each performs the same upload -> commit -> download-by-hash round-trip against a real HF Xet repo via the C API, using a Hub token-refresh URL for auth. All four were tested end-to-end against assafvayner/xet-c-api-test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
xet_capi, a new workspace crate exposing the fullxet::xet_session::XetSessionsurface over a C ABI, sohf-xetcan be consumed from C / C++ / CGo. It follows the same thin-wrapper pattern as the existing Python (hf_xet), wasm (hf_xet_wasm), and Node (xet_pkg_napi) bindings.Design: opaque handles freed explicitly; async transfers via handle-based polling (
XetOp—poll+ typedtake_*), backed by std threads running the existing_blockingAPIs; no callbacks cross the ABI in either direction. Errors areXetStatuscodes plus an opaqueXetErrorout-param. Reports are opaque with accessor functions; scalar progress/dedup are flat#[repr(C)]structs. The headerinclude/hf_xet.his cbindgen-generated and committed.Surface
xet_session_new/free,xet_init_logging, and threeXetAuthConfig-driven builders (upload commit, file-download group, stream group).XetFileInfo, download-to-path, finish, progress, abort, task_id; ordered/unordered streaming with pull-basednext, progress, cancel, task_id.XetOppoll + typedtake_*(file-metadata / commit-report / download-report / bytes / chunk / void / error).XetProgress/XetDedupMetrics;XetBytes;XetError/XetStatus.Build & safety
crate-type = ["cdylib", "staticlib", "rlib"]; release build produceslibxet_capi.{a,dylib}.extern "C"fn is wrapped in a panic guard (ffi_guard/catch_unwind) so panics never unwind across the ABI; all pointer args null-checked.build.rs; the committed header is guarded by an up-to-date test and a symbol-presence test.Test Plan
cargo test -p xet_capi— 15 tests pass (default features)cargo test -p xet_capi --features simulation— 16 pass, includinge2e_upload_then_download_via_ffi(real upload → download round-trip through the C ABI over alocal://CAS)cargo clippy -p xet_capi --all-targets -- -D warningsand--features simulation— cleancargo +nightly fmt -p xet_capi --check— cleancargo build -p xet_capi --release— produces staticlib + cdylibc_smoke_compiles—tests/smoke.ccompiles against the generated headerinclude/hf_xet.hagainst intended C consumer usage