ENH: Out-of-core architecture rewrite and filter optimizations#1568
Open
joeykleingers wants to merge 29 commits into
Open
ENH: Out-of-core architecture rewrite and filter optimizations#1568joeykleingers wants to merge 29 commits into
joeykleingers wants to merge 29 commits into
Conversation
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
b4ef97f to
99b49ed
Compare
joeykleingers
requested review from
JDuffeyBQ,
imikejackson,
jmarquisbq,
mmarineBlueQuartz and
nyoungbq
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
2 times, most recently
from
b4ef97f to
bb09048
Compare
This was referenced Mar 24, 2026
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
4 times, most recently
from
102c436 to
b4c1358
Compare
joeykleingers
changed the title
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
6 times, most recently
from
2bd614a to
110c054
Compare
joeykleingers
changed the title
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
4 times, most recently
from
35aecd0 to
3a88bbf
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
bdfed87 to
6fbfc8d
Compare
…ntationAnalysis
Replace per-element DataStore access with chunked bulk I/O
(copyIntoBuffer/copyFromBuffer) across 60+ algorithm files to eliminate
virtual dispatch overhead and HDF5 chunk thrashing when arrays are backed
by out-of-core storage.
--- Architecture ---
DispatchAlgorithm pattern (Direct/Scanline):
11 algorithms gain a base dispatcher class that selects between an
in-core Direct implementation and an OOC Scanline variant at runtime
based on IsOutOfCore()/ForceOocAlgorithm():
SimplnxCore: ComputeBoundaryCells, ComputeFeatureNeighbors,
ComputeKMedoids, ComputeSurfaceAreaToVolume, ComputeSurfaceFeatures,
DBSCAN, MultiThresholdObjects, QuickSurfaceMesh, SurfaceNets
OrientationAnalysis: BadDataNeighborOrientationCheck, ComputeIPFColors
ComputeGBCDPoleFigure dispatches directly from its filter executeImpl().
Connected Component Labeling (CCL) pattern:
4 algorithms gain a two-pass CCL variant as an OOC alternative to
random-access BFS/DFS flood-fill:
SimplnxCore: FillBadData (BFS/CCL), IdentifySample (BFS/CCL)
OrientationAnalysis: EBSDSegmentFeatures, CAxisSegmentFeatures
The CCL engine in SegmentFeatures::executeCCL() scans voxels in Z-Y-X
order with a 2-slice rolling buffer and UnionFind equivalence tracking,
giving sequential I/O access patterns. Supports Face and FaceEdgeVertex
connectivity with optional periodic boundaries.
--- New utility infrastructure ---
- UnionFind (src/simplnx/Utilities/UnionFind.hpp):
Vector-based disjoint set with union-by-rank and path-halving.
- SliceBufferedTransfer (src/simplnx/Utilities/SliceBufferedTransfer.hpp):
Z-slice buffered tuple transfer for propagating neighbor voxel data
used by ErodeDilate, FillBadData, MinNeighbors, and ReplaceElements.
- TupleTransfer batch API (Filters/Algorithms/TupleTransfer.hpp):
Batch bulk I/O methods for QuickSurfaceMesh and SurfaceNets mesh
generation attribute transfer.
- SegmentFeaturesTestUtils.hpp:
Shared test builder functions for segmentation filter test suites.
--- Bulk I/O conversions (existing algorithms) ---
Core utilities:
DataArrayUtilities (ImportFromBinaryFile, AppendData, CopyData,
mirror ops), DataGroupUtilities (RemoveInactiveObjects),
ClusteringUtilities (RandomizeFeatureIds), GeometryHelpers
(FindElementsContainingVert, FindElementNeighbors),
AlignSections (Z-slice OOC transfer path),
ImageRotationUtilities (source slab caching for nearest-neighbor),
TriangleUtilities (bulk-load triangles/labels for winding repair),
H5DataStore (streaming row-batch FillOocDataStore replacing full-
dataset allocation)
SimplnxCore algorithms:
AlignSectionsFeatureCentroid, ComputeEuclideanDistMap,
ComputeFeatureCentroids, ComputeFeatureClustering, ComputeFeatureSizes,
CropImageGeometry, ErodeDilateBadData, ErodeDilateCoordinationNumber,
ErodeDilateMask, RegularGridSampleSurfaceMesh, RequireMinimumSizeFeatures,
ReplaceElementAttributesWithNeighborValues, ScalarSegmentFeatures,
WriteAvizoRectilinearCoordinate, WriteAvizoUniformCoordinate
OrientationAnalysis algorithms:
AlignSectionsMisorientation, AlignSectionsMutualInformation,
ComputeAvgCAxes, ComputeAvgOrientations, ComputeCAxisLocations,
ComputeFeatureNeighborCAxisMisalignments,
ComputeFeatureReferenceCAxisMisorientations,
ComputeFeatureReferenceMisorientations, ComputeGBCD,
ComputeGBCDMetricBased, ComputeKernelAvgMisorientations,
ComputeTwinBoundaries, ConvertOrientations, MergeTwins,
NeighborOrientationCorrelation, RotateEulerRefFrame, WriteGBCDGMTFile,
WriteGBCDTriangleData, WritePoleFigure
EBSD readers:
ReadAngData, ReadCtfData, ReadH5Ebsd, ReadH5EspritData
--- Test infrastructure ---
- UnitTestCommon: ExpectedStoreType()/RequireExpectedStoreType() helpers,
TestFileSentinel reference-counted decompression, CompareDataArrays
rewritten with chunked bulk I/O for OOC-safe comparison.
- 29 test files updated with OOC dual-path testing:
ForceOocAlgorithmGuard + GENERATE(from_range(k_ForceOocTestValues))
runs every test case in both in-core and forced-OOC modes.
… bugs Add CreateResolvedDataStore utility that runs the IOCollection format resolver before creating a DataStore, matching the path filter actions use. Update test builder functions to call it so that test-constructed arrays become OOC stores when the OOC plugin is active. Fix three bugs in the OOC ComputeAvgOrientations Rodrigues average: - Allow featureId 0 in accumulation (matching architecture branch) - Start normalization loop from featureId 0 - Add missing continue for zero-count features to avoid divide-by-zero Fix stale GetIOCollection API call in UnitTestCommon (shared_ptr to ref). Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
…mized algorithms Adds extensive documentation across all out-of-core optimized filter algorithms explaining what each algorithm does and why the OOC variant works the way it does. Targets readers with no prior OOC knowledge. - Headers: Doxygen @Class, @brief, @param on all classes, methods, InputValues structs, and member variables - Source files: file-level overviews, Doxygen on operator()(), and inline comments explaining rolling windows, buffer strategies, dispatch logic, and OOC rationale - Filter docs: Algorithm sections with In-Core/Out-of-Core/Performance subsections added to ~45 filter markdown files - Key utilities: SliceBufferedTransfer.hpp and TupleTransfer.hpp documented as core OOC infrastructure
…writes WritePoleFigure was missed in the d4f2cce OOC-optimization sweep and was timing out (>300 s) on OOC-backed inputs under the OocOnly test path. Four Catch2 tests exercising it (Discrete, Discrete-Masked, Color, Color-Masked) were failing with Timeout on OOC-Release ctest. Hot paths replaced: 1. Cell-phase + Euler-angle gather loops (algorithm body). Two passes per phase over `numPoints` cells that used `phases[i]` + `eulerAngles[i * 3 + {0,1,2}]` per-element reads — one HDF5 hit per element on OOC stores, 4N hits per cell. Replaced with a chunk-sequential stream: 64K-tuple `std::vector` buffers filled via `copyIntoBuffer()` once per chunk; the inner loops iterate over the in-memory buffers. Peak auxiliary memory bounded to ~1 MB regardless of input size; NOT an O(N) bulk allocation. 2. Intensity-plot write-back (`std::copy(..., array.begin())` on three `Float64Array` outputs per phase). Per-element operator[] writes on OOC-backed `Float64Array` targets are one HDF5 write per pixel; a 512x512 image emits 262K hits per plot, ~786K per phase. Replaced with `copyFromBuffer()` per output array — one write per image. 3. Composite RGB image pack (for `figures.size() == 3` phases). The loop that interleaved RGBA source bytes into RGB-packed `UInt8Array` output wrote three `imageData[...] = ...` elements per pixel. Now builds the packed RGB buffer in a local `std::vector<uint8>` first, then emits one `copyFromBuffer()` call for the whole image. No algorithm changes. `metaDataArrayRef[phase] = ...` kept as-is — that's a single write keyed by phase count, not a hot loop. Test results on OOC-Release (InCore continues to pass — no in-core regression): WritePoleFigureFilter-Discrete 2.88 s (was: Timeout) WritePoleFigureFilter-Discrete-Masked 4.39 s (was: Timeout) WritePoleFigureFilter-Color 3.29 s (was: Timeout) WritePoleFigureFilter-Color-Masked 4.74 s (was: Timeout) All four now pass within the 300 s ctest timeout. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Add download_test_data entries for fill_bad_data_exemplars.tar.gz and identify_sample_exemplars.tar.gz to SimplnxCore test CMakeLists * Add download_test_data entries for segment_features_exemplars.tar.gz to both SimplnxCore and OrientationAnalysis test CMakeLists * Fixes CI failures where FillBadData, IdentifySampleFilter, ScalarSegmentFeatures, CAxisSegmentFeatures, and EBSDSegmentFeatures tests could not locate their exemplar archives Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Replace per-row copyIntoBuffer/copyFromBuffer pair (one pair per (z,y) scanline) with a K=32 Z-slice batched slab I/O pattern * Read K full source Z-slices per bulk call, extract the crop region via std::memcpy per row, and write K destination Z-slices in one bulk call * Working-set bound is O(n^(2/3)): (k_ZSliceBatch + 2) * X * Y * numComps * sizeof(T) bytes Tests: 16/16 pass on OOC build. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Remove the O(n) dense neighborsVoxelIndex array in favor of sparse parallel vectors (changedVoxels + neighborVoxelIdxs), saving ~16 GB on CT_align-scale volumes * Hoist slabBuf above the convergence while-loop so it is no longer re-allocated every iteration * Delete dead class RequireMinimumSizeFeaturesTransferDataImpl * Add ChunkedTransferWorker<T> doing Z-batched bulk I/O for the transfer phase, dispatched via ExecuteParallelFunction with type-based dispatch and parallelized per cell-level array via ParallelTaskAlgorithm; 64 MB/task/array slab budget Tests: 1/1 pass. Unit-test time unchanged at 0.23 s. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Replace the dense triNewIndex (8 B per triangle) with a triMask bitset (1 bit per triangle) plus a sparse triPrefixSum popcount table for ~6x memory reduction on the triangle side * Keep vertNewIndex as a dense 8 B per vertex map to preserve the invariant that triangle 0's three fresh vertices are assigned compact indices 0, 1, 2 (other filters depend on this ordering) * Stream all passes with chunked bulk I/O; vertex copies use bulk source reads + per-vertex dest writes (required by the ordering invariant), triangle copies use bulk reads and bulk writes Tests: 4/4 pass. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Mirror the nearest-neighbor slab-cache pattern in the trilinear interpolation path with a +/-2 Z-slice margin so all 8 corner neighbors remain resident, avoiding per-voxel random reads * Slide the slab cache window when consecutive output slices shift the needed source Z range — memmove the surviving slices in the buffer and read only the delta slices instead of re-reading the full slab * Parallelize the inner output-row loop for both the nearest-neighbor and trilinear paths via ParallelDataAlgorithm; threads share the cached slab (read-only) and write disjoint Y-row ranges of the local output slice buffer * Replace per-element at()/setValue() calls in the node-geometry convert path with 16 K-vertex chunked copyIntoBuffer/copyFromBuffer bulk I/O CT_align (1.97 B-voxel trilinear rotation): 133 s -> 20 s (~6.6x). Tests: 14/14 pass on both in-core and OOC builds. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Replace per-triangle getFaceCoordinates() random reads with a chunked pipeline: bulk-read 65K triangle connectivity indices per pass, determine the referenced vertex-index span, and bulk-load that vertex range into a local buffer * Parallelize the area compute on the local buffer (reads/writes touch plain C++ arrays only, so threads are safe — no DataStore access inside the parallel region) * Bulk-write the chunk's area output in one call * Guard against pathological meshes whose vertex indexing spans more than 16M entries per chunk with a serial per-triangle fallback; filter-generated meshes stay well under this cap CT_align (mesh-scale triangle areas): 26 s -> <1 s (~26x). Tests: 1/1 pass on both in-core and OOC builds. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
… inputs Bump the FeatureId (and RectGrid element-size) bulk-read chunk size from 64K to 256K tuples. The voxel-counting pass is I/O-bound on OOC-backed stores; larger chunks reduce copyIntoBuffer() round-trip overhead on datasets with tens of thousands of chunks while keeping per-chunk working-set memory bounded (1 MB for the int32 buffer, and an additional 1 MB for the float32 element-size buffer on the RectGrid path). CT_align (1.97 B voxels, Image path): 14 s -> 13 s. Tests: 9/9 pass on the OOC build. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Rewrite the markdown Algorithm section to explain the crop as a 3D subarray copy from first principles, teach the Z-slice-batched bulk I/O strategy step-by-step, and quantify why batching by K Z-slices collapses HDF5 chunk-op overhead * Add a Doxygen block on CropImageGeomDataArray describing the per-pass pipeline (bulk read slab -> in-memory extract -> bulk write) and the O(slab), non-O(volume) memory bound Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Rewrite the Algorithm section so a reader unfamiliar with the filter can follow the two-phase pipeline end-to-end: * Phase 1 (feature removal): motivate why small features get pruned, describe the 64K-tuple chunked scan, and explain the "skip write when chunk unchanged" optimization * Phase 2 (gap fill by majority-vote): teach the rolling 3-slice buffer scan, the sparse parallel vectors that replace the old O(n) dense index array, the per-array ChunkedTransferWorker with its +/-1 Z-margin slab read + interior-only write-back, and the outer ParallelTaskAlgorithm across arrays * Add a memory-footprint summary clarifying that every data structure is O(slice) or O(iteration bad count), never O(volume) Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Add a new Algorithm section that teaches the filter from scratch: * Explain conceptually which triangles are kept (all three vertices inside the user-specified node-type range) and what the output geometry looks like (compact vertex list, compact triangle list, remapped connectivity) * Document the downstream-invariant that forces vertNewIndex to stay a dense per-vertex map (triangle 0's three fresh vertices land at new indices 0..2 in traversal order) * Explain the triMask bitset + triPrefixSum sparse popcount table that replaces the legacy dense triangle map for ~6.4x memory savings, and how remapIndex() turns an O(1) table lookup plus a small popcount into each triangle's compact new index * Walk the six streaming passes (vertex-ok mask, triangle scan + vertex-index assignment, prefix-sum build, vertex copy, triangle remap copy, per-vertex/per-triangle attached-array copy) * Summarize the memory footprint so the vertNewIndex dominance is clear on very large meshes Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Add a comprehensive Algorithm section covering both the node-geometry and image-geometry paths from first principles: * Describe how every supported transform (rotation, scale, manual matrix, etc.) collapses to a single 4x4 homogeneous matrix M and how M composes with prior transforms * Node geometries: walk the 16K-vertex chunked read -> multiply -> write pipeline and explain why in-place topology+attribute data is correct * Image geometries: teach the re-gridding problem (why output voxels need to look up source values via M^-1), and contrast nearest- neighbor vs. trilinear interpolation * Z-slice slab cache: analytically deriving the per-output-slice source-Z range and the +/-2 trilinear margin * Sliding-window slab updates via memmove + delta copyIntoBuffer reads when consecutive output slices overlap heavily * Intra-slice parallelism via ParallelDataAlgorithm with thread safety argued from shared-read + disjoint-write access patterns and per-thread pValues scratch Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Add an Algorithm section that walks the chunked pipeline step-by-step for a reader unfamiliar with the optimization: * Establish the closed-form per-triangle math (0.5 * |(A-B) x (A-C)|) so there is no confusion about the compute * Quantify the naive access pattern (six OOC chunk-cache hits per triangle, hundreds of millions of virtual dispatches on CT-scale meshes) to motivate the chunking * Walk the five-step per-chunk pipeline: bulk triangle connectivity read -> analyze vertex-index span -> span-bounded bulk vertex coords read -> parallel compute on plain buffers -> bulk area write * Explain the 16M-vertex span cap and the serial per-triangle fallback for pathological meshes * Summarize memory footprint (bounded O(chunk), not O(mesh)) Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Rewrite the Algorithm section to fully teach the filter: * State what the three output arrays (NumElements, Volume, EquivalentDiameter) represent and show the spherical/circular diameter formulas * Image Geometry path: explain the uniform-voxel-volume shortcut that lets the filter skip per-voxel volume computations, then walk the 256K-tuple chunked count pass and the per-feature output pass; cover the 2D fallback rules and the two-empty-dimensions preflight error * RectGrid path: contrast with the Image case, describe the lockstep FeatureIds + elementSizes chunked read, and explain why Kahan summation is needed to avoid float32 rounding error on billion-voxel volumes * Justify the 256K chunk size choice based on HDF5 chunk-lookup overhead vs. L2 cache residency * Summarize memory footprint Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Three sites in the algorithm multiplied double-precision resolution and angle values by k_PiOver180F (a float constant). Float promotion to double preserves the quantized float bits rather than recovering the true M_PI/180.0, introducing a ~1e-10 deviation from the legacy SIMPL algorithm (which uses double k_PiOver180). Over ~756k triangles times ~2300 symmetry operations the deviation flipped two near-boundary triangles in/out of the selected set, shifting a handful of distribution bin values by ~3e-2. Switch the three multiplications to the existing k_PiOver180D double constant so the resolution thresholds and fixed-misorientation angle are computed at full double precision. The stored 6_6_find_gbcd_metric_based.tar.gz exemplar was generated by the original float-precision DREAM3D FindGBCDMetricBased filter and no longer matches the simplnx algorithm after this fix. Publish a fresh exemplar from the double-precision legacy pipeline and repoint the tests at it. * Rename archive and top-level folder from 6_6_find_gbcd_metric_based to compute_gbcd_metric_based (drops the legacy 6_6_ prefix in accordance with current archive-naming conventions). * Drop the 6_6_ prefix from the stored .dat exemplar filenames; input .dream3d filename follows the folder name. * ComputeGBPDMetricBasedTest's InValid section reuses the GBCD archive (for crystal-structures and mesh input); update its paths too. * CMakeLists.txt download_test_data entry bumped to the new archive name and SHA512. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
…x bool-mask bulk I/O Three logically related changes that finish reconciling the rebased branch with Nathan Young's PR BlueQuartzSoftware#1590 (ENH: Standardize 2D Image Handling) and fix one resulting OOC perf cliff: 1. Wholesale port of PR BlueQuartzSoftware#1590's two algorithm rewrites into the renamed in-core dispatch variants: - ComputeFeatureNeighborsDirect.cpp gets Nathan's templated ComputeFeatureNeighborsFunctor<ImageDimensionStateT> and ProcessVoxels dispatcher in place of the OOC-commit-era custom in-core logic. - IdentifySampleBFS.cpp gets Nathan's templated IdentifySampleFunctor plus the corresponding ProcessVoxels dispatch. The Scanline OOC variant of ComputeFeatureNeighbors is updated to reference the namespaced VoxelNeighbors<Image3D>:: constants while preserving its Z-slice rolling-window bulk-I/O structure. 2. Reapply PR BlueQuartzSoftware#1590's constexpr/const cleanups across the algorithm files where the rebase took --theirs (the OOC commit version) at the 2aa00ee conflict and dropped Nathan's small adjustments: SimplnxCore: ComputeBoundaryCellsDirect, ErodeDilateBadData, ErodeDilateCoordinationNumber, ErodeDilateMask, ReplaceElementAttributesWithNeighborValues, RequireMinimumSizeFeatures OrientationAnalysis: BadDataNeighborOrientationCheckWorklist, NeighborOrientationCorrelation The pattern is uniform: promote the inlined `6` neighbor-array sizes to use VoxelNeighbors<Image3D>::k_FaceNeighborCount via a local k_NumFaceNeighbors alias, make neighborVoxelIndexOffsets const, make faceNeighborInternalIdx constexpr, make isValidFaceNeighbor const where it is not mutated, drop the now-unused DataGroup.hpp include, and const-ify NeighborOrientationCorrelation's orientationOps. ComputeFeatureNeighborsFilter.md picks up Nathan's all-dimension note about user-set spacing for shared surface area calculation. 3. Fix a per-element OOC fallback in BadDataNeighborOrientationCheckScanline that was triggered whenever the input mask was a BoolArray rather than a UInt8Array. The previous code routed bool masks through maskCompare->isTrue / maskCompare->setValue per voxel per Z-slice, causing chunk thrashing under chunked OOC storage. The Small_IN100 pipeline test (a 189x201x117 volume with a bool mask produced by MultiThresholdObjects) ran in 4.7 s on simplnx-Rel but 3+ minutes on simplnx-ooc-Rel. AbstractDataStore<bool> already exposes copyIntoBuffer/copyFromBuffer just like AbstractDataStore<uint8>; the comment claiming otherwise was stale. Resolve a typed AbstractDataStore<bool>* alongside the existing uint8 store pointer and route both load and write-back through bulk I/O, with a small per-slice std::unique_ptr<bool[]> scratch buffer bridging between the algorithm's uint8 slice buffers and the bool data store's typed bulk API. With this change Small_IN100 OOC drops to 4.6 s (~1.6x in-core, in line with normal OOC overhead). Tests updated: - IdentifySampleTest.cpp adopts Nathan's PR BlueQuartzSoftware#1590 hand-built 2D Empty Z/Y/X Non-Square regression tests plus the parameterized identify_sample_v2 exemplar test and the SIMPL Backwards Compatibility test, all wrapped with the OOC dual-path pattern (ForceOocAlgorithmGuard + GENERATE(from_range(k_ForceOocTestValues))). The pre-existing 200x200x200 large-scale OOC validation test is retained. Verified: simplnx-Rel and simplnx-ooc-Rel preset builds both clean. All 43 affected-filter tests pass on simplnx-Rel; all 86 affected-filter tests pass on simplnx-ooc-Rel (regex covering ComputeFeatureNeighbors, IdentifySample, BadDataNeighborOrientation, ComputeBoundaryCells, ErodeDilate*, NeighborOrientationCorrelation, ReplaceElementAttributesWithNeighborValues, RequireMinimumSizeFeatures).
* Replace CreateDataStore + CreateResolvedDataStore with a single resolver-aware CreateDataStore(DataStructure, DataPath, ...) that always consults the registered format resolver. Old explicit-format overload deleted. * Replace CreateListStore similarly so NeighborList backing storage is OOC-eligible when the OOC plugin is loaded and thresholds permit. * Inline action-layer caller in ArrayCreationUtilities::CreateArray using GetIOCollection().createDataStoreWithType directly. * Migrate 23 CreateResolvedDataStore call sites (mechanical rename). * Migrate 13 cell-level test fixtures that were silently in-memory in OOC builds to the resolver-aware path so OOC builds actually exercise OOC stores. * Migrate 6 in-memory non-test callers (ComputeFeatureCentroids scratch buffers, HDF5 readers in DataStoreIO and DatasetIO) to direct std::make_shared<DataStore<T>> since they have no DataStructure context. * Migrate 2 NeighborListIO HDF5 readers to std::make_shared<ListStore<T>> for the same reason (in-core branch of the import pipeline). * Wire CreateNeighbors action helper through the resolver-aware CreateListStore. * Rewrite IOFormat.cpp tests to exercise the resolver path. ImageGeom and RectGridGeom findElementSizes now route through the new CreateDataStore so the voxel-sizes array can go OOC for very large structured grids. RectGridGeom's inner loop also refactored from per-voxel setValue calls to per-axis precompute + Z-slice copyFromBuffer to avoid catastrophic OOC perf when the array is OOC-backed. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
… Format tests
* DataIOCollection's constructor registers the OOC format via
SimplnxOoc::registerIOManager under SIMPLNX_USE_OOC, so getManager("HDF5-OOC")
resolves in the compile-time-switch OOC build.
* IOFormat: guard the in-core large-data-format preference tests to
#ifndef SIMPLNX_USE_OOC (the OOC-build defaults are covered by SimplnxOoc's
DataFormatPreferenceTest) and update the "not configured" assertion to the
seeded k_InMemoryFormat default.
Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
The rebase onto upstream/develop brought in the parallelized in-core ComputeFeatureSizes (tbb::combinable thread-local accumulation), while the OOC branch had replaced that loop with serial chunked bulk I/O. Rather than discard either, split the algorithm into the established Direct/Scanline dispatch pattern so each storage backing uses its optimal strategy. * ComputeFeatureSizesDirect: in-core parallel accumulation (the upstream ParallelDataAlgorithm + tbb::combinable Kahan-summation implementation) * ComputeFeatureSizesScanline: out-of-core chunked copyIntoBuffer streaming (renamed from the former single ComputeFeatureSizes implementation) * ComputeFeatureSizes: thin DispatchAlgorithm<Direct, Scanline> dispatcher selecting on whether the FeatureIds array is out-of-core * Register both new algorithm units in the SimplnxCore CMakeLists * Exercise both paths in the existing tests via ForceOocAlgorithmGuard + GENERATE(from_range(k_ForceOocTestValues)) Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
The in-core build previously forced SIMPLNX_TEST_ALGORITHM_PATH to InCoreOnly whenever SIMPLNX_USE_OOC was OFF, on the assumption that no out-of-core paths exist to test. That is no longer true: the Direct/Scanline dispatch classes are always compiled into the plugins, and forcing the Scanline (OOC) path runs it against in-core data via copyIntoBuffer (a plain std::copy here), staying fast while verifying the OOC algorithm matches the in-core result. * Only coerce the nonsensical OocOnly (1) to InCoreOnly (2) when OOC is off * Allow Both (0) so a single in-core build can validate both algorithm paths Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
5 tasks
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
d4f634b to
c04347c
Compare
joeykleingers
changed the title
DatasetIO::createEmptyDataset created the dataset with no creation property list, so out-of-core arrays large enough to take the two-step streaming write (createEmptyDataset + hyperslab writes) were always written contiguous and uncompressed, even when WriteOptions requested compression. The single-shot writeSpan path already applied it. * Build the dataset creation property list via BuildChunkedDeflateDcpl in createEmptyDataset, matching writeSpan * Preserves the existing fall-throughs to contiguous storage for compression level 0 and for arrays below the small-array threshold Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
The "Recovery file with all in-core data" test hardcoded StoreType::InMemory and relied on ambient preferences, so it failed whenever forceOocData was set: under forceOoc, the recovery file's inline arrays correctly load as out-of-core stores backed by the recovery file itself. * Assert the expected store type via RequireExpectedStoreType, which tracks the active large-data preferences (OutOfCore under forceOoc, InMemory otherwise) * Correct stale comments that claimed OOC was not compiled in Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Add MemoryBudgetManager::maxBudgetBytes() (max(min(total-6GiB, 0.95*total), 1GiB)) and make setBudgetBytes() clamp the upper bound and report whether it clamped. Deduplicate the platform total-RAM ifdef through Memory::GetTotalMemory(). Apply the --memory-budget override to the manager in nxrunner so the cap and the override actually take effect in headless runs (previously the override was written only to Preferences, which the manager never reads in CLI mode). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rewrites the out-of-core (OOC) architecture in simplnx — replacing the old chunk-based API with a new bulk I/O design built around
copyIntoBuffer/copyFromBufferonAbstractDataStore— and adds the OOC-optimized algorithm variants for 30+ filters that build on it. The core infrastructure and the filter optimizations (previously tracked separately in #1575) are consolidated onto this branch, so this PR now contains both layers:DispatchAlgorithm-based in-core (Direct/BFS) vs OOC (Scanline/CCL) variants for 30+ filters, with benchmarks.Core OOC Architecture
Core Architecture Changes
AbstractDataStore/IDataStore(loadChunk,getNumberOfChunks,getChunkLowerBounds,getChunkUpperBounds,getChunkShape)copyIntoBuffer/copyFromBufferpure virtual bulk I/O methods toAbstractDataStorewith implementations inDataStore,EmptyDataStore, andHDF5ChunkedStore(in SimplnxOoc plugin)StoreTypeenum (InMemory,OutOfCore,Empty) toIDataStore;IsOutOfCore()now checksStoreTypeinstead ofgetChunkShape()HDF5ChunkedStoreperforms I/O via HDF5 hyperslab selections with Z-slice-aligned default chunk shape{1,Y,X}for 3D datacopyFromBufferfast path: skips read-modify-write for tuple-aligned writescopyIntoBufferfast path: direct span-basedreadTuplesfor tuple-aligned readsDatasetIOgainsreadTuples/writeTuplesfor direct hyperslab-based bulk tuple I/ONew Core Utilities
DispatchAlgorithm— Runtime dispatch between in-core (Direct) and OOC (Scanline/CCL) algorithm variants based on data store typeSliceBufferedTransfer— Type-dispatched Z-slice buffered tuple copy utility that eliminates per-element OOC overhead during morphological transfer phasesUnionFind— Vector-based disjoint set data structure with union-by-rank and path-halving compression for chunk-sequential CCL algorithmsSegmentFeaturesOOC path — Z-slice CCL-based connected component labeling withUnionFindequivalence tracking, replacing BFS/DFS flood fill for OOC dataAlignSectionsOOC path — Bulk slice read/write withAlignSectionsTransferDataOocImplDataArrayUtilitiesbulk I/O —ImportFromBinaryFile,AppendData,CopyData, and mirrorswap_rangesupdated with chunked bulk I/O (runtime OOC check preserves original in-core performance)OOC Store Management
DataIOCollection/IDataIOManager— Updated for OOC store lifecycle managementImportH5ObjectPathsAction— OOC-aware file import with recovery metadataDataStoreIO— Detect OOC recovery attributes inReadDataStorefor safe data restorationFilter Optimizations
Adds out-of-core (OOC) optimized algorithm variants for 30+ filters, using
DispatchAlgorithmto select between in-core (Direct/BFS) and OOC (Scanline/CCL) code paths at runtime based on data store type. A preparatory rename commit gives git rename tracking so that GitHub shows meaningful diffs against the original algorithm code.Algorithm Rename for Git Tracking
Renames 13 algorithm files to their in-core variant names before any logic changes, so that when dispatch variants are introduced, GitHub shows proper diffs against the original code instead of "new file" with no context.
Bug Fixes
OOC import of legacy SIMPL files with multi-dimensional component arrays
Legacy SIMPL
.dream3dfiles store multi-dimensional component arrays (e.g., GBCD with componentShape[10,10,10,20,20,2]) with HDF5 physical dimensions in reversed order relative to theComponentDimensionsattribute.Two fixes address this at different layers:
AbstractOocStore::readHdf5(SimplnxOoc): Detects shape mismatch between logical and physical dimensions before the streaming import path. Falls back to flat bulk read (H5S_ALL) when shapes differ, preserving correct byte order.ImportH5ObjectPathsAction::backfillReadOnlyOocStores(simplnx): The read-only reference store optimization creates stores pointing directly at the source file. For mismatched arrays, the N-D hyperslabs would be out-of-bounds. Detects the mismatch and creates a writable OOC store populated viareadHdf5(which triggers the flat-read fallback) instead of a read-only reference.Filter Optimization Details
Group B — Face-Neighbor Filters (5 filters)
Split into Direct (in-core) and Scanline (OOC) algorithm classes using
DispatchAlgorithm. Scanline variants use Z-slice rolling windows (prev/cur/next) for cross-slice neighbor access with zero per-element OOC overhead.Filters:
ComputeBoundaryCells,ComputeSurfaceFeatures,ComputeFeatureNeighbors,ComputeSurfaceAreaToVolume,BadDataNeighborOrientationCheckGroup C — Morphological / Neighbor Replacement (5 filters)
Z-slice rolling buffers for all 6 face-neighbor reads from RAM.
SliceBufferedTransferfor type-dispatched bulk tuple copy.Filters:
ErodeDilateBadData,ErodeDilateCoordinationNumber,ErodeDilateMask,ReplaceElementAttributesWithNeighborValues,NeighborOrientationCorrelationGroup D — CCL Segmentation (5 filters)
Chunk-sequential Connected Component Labeling using
UnionFindequivalence tracking, replacing BFS/DFS flood fill for OOC data.Filters:
ScalarSegmentFeatures,EBSDSegmentFeatures,CAxisSegmentFeatures,FillBadData,IdentifySampleGroup E — AlignSections Family (4 filters)
Bulk slice read/write via
AlignSectionsTransferDataOocImpl. Per-filter OOCfindShiftswith 2-slice buffers and bulk mask reads.Filters:
AlignSectionsMisorientation,AlignSectionsMutualInformation,AlignSectionsFeatureCentroid,AlignSectionsListFilterQuickSurfaceMesh
DispatchAlgorithm<QuickSurfaceMeshDirect, QuickSurfaceMeshScanline>. Scanline eliminates the O(volume)nodeIdsarray (7.5 GB for 1000³) with rolling 2-plane node buffers (16 MB). Two-pass architecture: counting pass + mesh creation pass. All output arrays (triangle connectivity, faceLabels, vertex coordinates, nodeTypes) buffered per z-slice and flushed withcopyFromBuffer. BatchquickSurfaceTransferBatchAPI added toTupleTransferfor bulk source-read/dest-write of cell and feature data.SurfaceNets
DispatchAlgorithm<SurfaceNetsDirect, SurfaceNetsScanline>. Scanline is a complete reimplementation (881 lines) eliminating the O(n)Cell[]array — uses O(surface) hash map + vertex vectors with slice-by-slice FeatureIds reading. All output arrays (vertices, nodeTypes, triangle connectivity, faceLabels) buffered and flushed withcopyFromBuffer. BatchsurfaceNetsTransferBatchAPI added toTupleTransferfor bulk I/O.Mesh Infrastructure (RepairTriangleWinding + GeometryHelpers)
RepairTriangleWinding: Bulk-reads triangle face list and faceLabels into local buffers; all BFS work operates on local memory; modified triangles written back viacopyFromBuffer.FindElementsContainingVert/FindElementNeighbors(GeometryHelpers.hpp): Chunked bulk I/O with 65K-element chunks for sequential passes. Random neighbor lookups check if candidate is in the current chunk (cache hit) before falling back to per-elementcopyIntoBuffer. Together with RepairTriangleWinding buffering, this reduced SurfaceNets Winding from 515s to 2.9s.Clustering Filters (3 filters)
DBSCAN:DispatchAlgorithm<DBSCANDirect, DBSCANScanline>— chunked grid construction, on-demand per-grid-cell coordinate reads incanMerge. 653s → 12s (54x)ComputeKMedoids:DispatchAlgorithm<Direct, Scanline>— chunkedfindClusters, per-clusteroptimizeClusterswith O(max_cluster_size) peak memory. 74s → 13s (5.7x)ComputeFeatureClustering: Single implementation with feature-level array caching. 203s → 77s (2.6x)Pipeline Prerequisite Filters (2 filters)
MultiThresholdObjects:DispatchAlgorithm<Direct, Scanline>— eliminates O(n)tempResultVectorin OOC pathConvertOrientations: Single implementation with chunked bulk I/O in macro-generated Convertor classes (4096-tuple chunks)Together these reduced the AlignSectionsMisorientation pipeline test from 635s to 5.9s (107x).
OrientationAnalysis Misc (10 filters)
ComputeTwinBoundaries: Bulk-read all face/feature/ensemble arrays into local vectors. 179s → 44s (4x)ComputeKernelAvgMisorientations: Slab-based bulk I/O with cached CrystalStructuresComputeAvgCAxes: Already OOC-optimized (chunked reads, cached feature output). Compute-bound.ReadH5Ebsd:copyFromBufferin CopyData template, phase copy, Euler interleaving. 463s → 241s (1.9x)ComputeGBCDPoleFigure:DispatchAlgorithm<Direct, Scanline>— Direct caches full GBCD, Scanline caches only the phase-of-interest slice (bounded by bin resolution, not cell count). 853s → 0.9s (948x)ComputeFeatureReferenceCAxisMisorientations: Z-slice buffered I/O for all cell-level arrays (featureIds, cellPhases, quats, output). Cached ensemble/feature-level arrays (crystalStructures, avgCAxes). 196s → 5.4s (36x)ComputeFeatureNeighborCAxisMisalignments: Bulk-read all feature-level arrays (featurePhases, featureAvgQuat, crystalStructures) and buffered avgCAxisMisalignment output.MergeTwins: Chunked bulk I/O for voxel-level parent ID fill and assignment loop. Feature-level featureParentIds cached locally for lookup. 67s → 1.8s (37x)ReadCtfData: BulkcopyFromBufferfor all cell arrays (phases, euler angles, bands, error, MAD, BC, BS, X, Y). Euler angle interleave uses chunked 64K buffer. Crystal structures cached locally for hex correction. 231s → 0.25sReadAngData: Same bulkcopyFromBufferpattern. Phase validation done in-place on EbsdLib buffer before single bulk write. Euler interleave chunked.Pipeline-Critical Filters (6 filters)
Optimizations targeting the filters responsible for OOC pipeline timeouts (4 of 5 timed-out pipelines blocked by
ComputeIPFColors):ComputeIPFColors:DispatchAlgorithm<ComputeIPFColorsDirect, ComputeIPFColorsScanline>. Direct keeps parallelParallelDataAlgorithmfor in-core; Scanline uses chunked sequential bulk I/O (65K-tuple chunks) with locally cached crystal structures.ForceOocAlgorithmGuardadded to test. 1,937ms → 90ms (21.5x)ComputeFeatureSizes:DispatchAlgorithm<ComputeFeatureSizesDirect, ComputeFeatureSizesScanline>— Direct keeps the parallel in-core accumulation (tbb::combinable Kahan summation); Scanline uses chunkedcopyIntoBufferfor featureIds (ImageGeom path) and featureIds + elemSizes (RectGrid path, Kahan summation preserved). 813ms → 28ms (29x) on the OOC/Scanline path.ComputeAvgOrientations: Chunked featureIds/phases/quats reads, locally cached crystal structures and avgQuats (feature-level). BulkcopyFromBufferfor output arrays.ComputeFeatureReferenceMisorientations: Chunked all cell-level arrays (featureIds, phases, quats, GB distances, output misorientations). Locally cached crystal structures, avgQuats, and center quaternions (all feature/ensemble-level). 106ms → 1ms (106x)ComputeFeatureCentroids: ReplacedAbstractDataStoreintermediate arrays (sum, center, count, rangeX/Y/Z) with plainstd::vector— eliminates ~119M virtual dispatch calls per run. Chunked featureIds reads. Inline coordinate computation from spacing/origin. 39,724ms → 25ms (1,589x)RequireMinimumSizeFeatures: Three-part optimization:removeSmallFeatures: Chunked featureIds read/write (65K-tuple batches)assignBadVoxels: 3-slice rolling slab buffer for neighbor voting scan (O(slice) memory), sparse changed-voxel tracking to skip full-volume transfer when few/no voxels changed. 14,592ms → 142ms (103x)RemoveInactiveObjects(shared utility inDataGroupUtilities.cpp): Chunked featureIds renumbering withcopyIntoBuffer/copyFromBuffer. 5,573ms → 50ms (111x)Additional Filters
ComputeEuclideanDistMap: Bulk-read featureIds and distance stores into local vectors; flood-fill operates on local memory; bulk-write output. 116s → 1.1s (105x)AppendImageGeometry: Bulk I/O for mirror operations (scanline-based reversal instead of per-tuple swaps). 469s → 113s (4.2x)GBCD Filter Group (5 filters)
All five GBCD filters optimized for OOC with zero cell-level O(n) allocations, cancel checking, and progress messaging:
ComputeGBCDPoleFigure:DispatchAlgorithm<Direct, Scanline>withForceOocAlgorithmGuardin test. Scanline caches only the phase-of-interest GBCD slice viacopyIntoBuffer.WriteGBCDGMTFile: Phase-of-interest GBCD slice cached viacopyIntoBuffer; crystal structures cached locally.WriteGBCDTriangleData: Chunked triangle I/O (8K chunks), feature-level euler cache, buffered file output viafmt::format_to+fmt::memory_buffer.ComputeGBCD: Feature-level caching (eulers, phases, crystalStructures), chunked triangle array reads per 50K-triangle iteration, GBCD output accumulated in local buffer (bounded by phases × bins) then written back viacopyFromBuffer.ComputeGBCDMetricBased: Eliminated O(n)triIncludedallocation (replaced with per-chunk sequential area accumulation). Feature-level caching (phases, eulers, crystalStructures, featureFaceLabels). Chunked triangle I/O in totalFaceArea scan. Raw pointer access in parallel TrianglesSelector worker.HDF5 Import + Pole Figure Filters (3 filters)
FillOocDataStore(shared infrastructure): Streaming chunked HDF5 hyperslab reads +copyFromBuffer, with zero O(n) temp allocations — batched reads even for partial hyperslabs. Benefits all HDF5 import paths.ReadH5EspritData:copyFromBufferbulk writes from raw HDF5 reader buffers, replacing 9+ per-elementoperator[]writes per point.WritePoleFigure: Chunked iteration over eulerAngles/phases/mask per-phase using bounded buffers (no O(n) pre-caching); bulk-write intensity and image outputs viacopyFromBuffer.ReadHDF5Dataset: Cancel checking + per-dataset progress messages.WritePoleFigureTestandReadHDF5DatasetTestoptimized withcopyIntoBuffer.Core Utilities + Geometry Filters
ImportFromBinaryFile:copyFromBufferinstead of per-element writes. ReadRawBinary Case1: 1076s → 29s (37x)CropImageGeometry: Row-based bulk I/O. 27s → 2.6s (10x)RandomizeFeatureIds(ClusteringUtilities): Chunked bulk I/O for both overloads — benefits all callers (segmentation filters, SharedFeatureFace, MergeTwins).AppendData/CopyData/mirror swaps: Runtime OOC check — chunked bulk I/O for OOC, original code for in-core (verified zero in-core regression)TupleTransfer: AddedquickSurfaceTransferBatchandsurfaceNetsTransferBatchbatch APIs with bulkcopyIntoBuffer/copyFromBufferfor source reads and destination writes. Used by QuickSurfaceMeshScanline and SurfaceNetsScanline.Cancel + Progress Messaging
All in-core and OOC algorithm variants now have:
m_ShouldCancelchecks at the top of major outer loopsThrottledMessenger-based progress reporting with descriptive phase messages and percentage completionOOC Performance Results
All benchmarks on arm64 Release build with
forceOocData = true.Mesh Generation Filters (full ctest wall-clock, OOC build)
Groups B–E (200³ dataset, filter.execute() only)
Pipeline-Critical Filters (filter.execute() only, OOC build)
OrientationAnalysis Filters (full ctest wall-clock, OOC build)
GBCD Filter Group (full ctest wall-clock)
HDF5 Import + Pole Figure Filters (full ctest wall-clock)
Additional Optimizations (full ctest wall-clock)
Test Infrastructure
CompareDataArraysrewritten to usecopyIntoBufferin 40K-element chunks instead of per-elementoperator[]CompareFloatArraysWithNans,CompareArrays, andCompareDataArraysByComponent(UnitTestCommon.hpp) rewritten with chunkedcopyIntoBufferreads (40K elements/chunk). Per-elementoperator[]access caused extreme slowdowns on OOC-backed arrays — this alone reduced the ComputeGBCD test from 1500s (timeout) to ~10s (the filter itself runs in ~3s).RotateSampleRefFrameuses slab-basedcopyIntoBuffer/copyFromBufferinRotateImageGeometryWithNearestNeighbor(no O(n) allocation);RotateEulerRefFrameuses chunked 65K-tuple I/O (19.5s → 4.8s, 4x). Together these reduced ReadH5Ebsd from 241s to 2.1s (117x).ForceOocAlgorithmGuardcoverage in all optimized filter tests for both algorithm pathsSIMPLNX_TEST_ALGORITHM_PATHCMake option (0=Both, 1=OOC-only, 2=InCore-only) for build-specific test path controlRelated PRs
Test Plan