Add an S3-to-GPU kTLS reference implementation - #406
Open
kjmph wants to merge 5 commits into
Open
Conversation
Add a minimal-dependency reference path for streaming S3 objects over TLS 1.3 RX kTLS through a NUMA-local CUDA-pinned ring into final GPU allocations. The implementation models a database scan: object metadata comes from a catalog snapshot, the TCP/TLS/kTLS pool is established without HTTP requests or payload warmup, and measured iterations validate exact device contents. It includes retry accounting, receive-only mode, and an explicitly tuned g7e.48xlarge profile intended to be retuned elsewhere. On g7e.48xlarge, the path sustains roughly 1.04–1.07 Tbps during transfer and about 0.88 Tbps end-to-end for a 303.8 GB scan, with zero transfer errors and approximately 1 ms of final H2D drain. A current ENA driver, jumbo path MTU, and a regional S3 gateway endpoint are recommended. The order-2 ENA RX-page descriptor-collapse experiment is optional and non-upstream.
On systems with 4 KiB base pages, a jumbo receive can span three ENA
descriptors and backing pages. This increases receive descriptor
processing and page-pool traffic.
Add an opt-in large_rx_page module parameter that backs each RX
descriptor with an order-2 compound page. A 16 KiB allocation provides
enough contiguous space for a jumbo frame in one posted receive buffer.
Track the allocation order and span per ring and use them consistently
for page-pool configuration, DMA mapping, descriptor lengths, skb
accounting, and teardown. Keep high-order allocations intact instead of
using deferred buffer fragmentation and reuse.
Reject the mode when Page Pool support or 4 KiB base pages are
unavailable, and prevent XDP and AF_XDP attachment while it is active.
Treat failure to populate the initial RX rings as fatal. Add ethtool
statistics for allocation geometry, descriptor usage, allocation
failures, and skb fragment counts.
This trades higher-order allocation pressure and increased skb truesize
for reduced receive-side descriptor and page processing. The feature is
experimental, disabled by default, and can be enabled with:
options ena large_rx_page=1
Normalized configurable paths. Made whole-script sudo drop privileges safely. Added access/identity checks and reliable temp cleanup. Fixed artifact ignores across the checkout.
Merge the host-agnostic runner updates with the latest reference path. - resolve invoking-user identity and canonicalize runtime paths - rebuild only when the CUDA source is newer than the binary - preconnect and HEAD-prime every kTLS socket without transferring payload - preserve and restore TCP, IRQ, and NIC queue settings - distinguish catalog, prime, and measured-query HEAD requests - document patched ENA installation on Ubuntu and Amazon Linux
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a minimal-dependency reference path for streaming S3 objects over TLS 1.3 RX kTLS through a NUMA-local CUDA-pinned ring into final GPU allocations.
The implementation models a database scan: object metadata comes from a catalog snapshot, the TCP/TLS/kTLS pool is established without HTTP requests or payload warmup, and measured iterations validate exact device contents. It includes retry accounting, receive-only mode, and an explicitly tuned g7e.48xlarge profile intended to be retuned elsewhere.
On g7e.48xlarge, the path sustains roughly 1.04–1.07 Tbps during transfer and about 0.88 Tbps end-to-end for a 303.8 GB scan, with zero transfer errors and approximately 1 ms of final H2D drain.
A current ENA driver, jumbo path MTU, and a regional S3 gateway endpoint are recommended. The order-2 ENA RX-page descriptor-collapse experiment is optional and non-upstream.