Feat/xenium celldega regular grid - #415
Open
cornhundred wants to merge 17 commits into
Open
Conversation
Adds the viewer-independent core of the celldega_regular_grid_v1 profile: the tile geometry and the tile -> row-group -> file numbering. - x-major tile ids (tile_x * num_tiles_y + tile_y), matching Celldega's RowGroupTileReader, but reimplemented rather than imported so spatialdata-io gains no dependency on a viewer package (Celldega already depends on spatialdata-io, so importing it would be circular). A conformance test pins the two formulas together. - Half-open tile bounds, with the dataset's upper edge clamped into the last tile so points on x_max/y_max are not dropped. - Out-of-grid coordinates raise instead of being silently clamped, since they indicate a mismatched transform rather than a rounding artefact. - Zero-padded chunk filenames: Celldega indexes the manifest's files array by position while dask globs and sorts lexicographically, so unpadded names would make dask reorder partitions past 10 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FeatureCatalog assigns genes codes [0, n_genes) in the table's var_names order,
so a gene's feature_code is also its CBG row-group index -- no second lookup and
no browser-side string join. Non-gene features (controls, unassigned codewords)
are retained but coded above every gene and flagged, never folded into a real
gene. Verified on Xenium pancreas: 377 genes + 164 controls.
write_points_regular_grid keeps every canonical column and the index, and adds
display_xy (fixed_size_list<uint32>[2], already interleaved for deck.gl) and
feature_code. Rows are grouped by tile, one row group per logical tile including
empty ones, split across zero-padded chunk files.
Notable details:
- statistics are disabled: the tile formula is the spatial index, so no client
reads column-chunk min/max, and they inflate the footer the browser fetches.
- coordinates are cast to float64 before the affine: under numpy 2's NEP-50
promotion, float32 * python float stays float32 and loses pixel precision.
- the rewrite is idempotent. Re-running on an already-optimized element replaces
the render columns rather than appending duplicates, which previously produced
an unreadable file ("Multiple matches for FieldRef") because pandas also
round-trips fixed_size_list back as a variable-length list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds display_geometry (list<list<fixed_size_list<uint32>[2]>>: polygon -> rings -> interleaved integer pixel vertices) and cell_code, alongside the untouched canonical WKB geometry. - Cells are assigned to exactly one tile by centroid in display pixel space; a polygon whose outline crosses a tile boundary is not duplicated. Covered by a fixture case with exactly that shape. - display_geometry is marked lossy in the manifest: exterior ring only, largest part of a MultiPolygon, matching Celldega's current behaviour. - The nested layout is walked by a test that mirrors getPolygonDataFromChunk (polygon offset -> ring offset -> coordinate index), so a layout change that would break the browser fails here instead. - GeoParquet 'geo' metadata is preserved via the same conversion to_parquet uses. GeoDataFrame.to_arrow() emits GeoArrow extension metadata but not the 'geo' key, and without it the rewritten file stops being readable by geopandas.read_parquet and by SpatialData. Verified against Xenium pancreas: 140,702 cells, all single-ring Polygons, shapes index matches table obs_names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One row group per gene, so selecting a gene fetches a single row group instead of touching any transcript data. Schema matches Celldega's existing CBG reader exactly: cell_id / expression / gene, with gene_to_row_group and num_genes in the parquet schema metadata. Deliberate difference from Celldega's own writer: every catalog gene gets a row group in catalog order, including all-zero genes, so that 'row group index == feature_code' holds and one integer addresses both a transcript's gene and its expression vector. gene_to_row_group is still written for clients that look it up rather than assume it. Explicit stored zeros are dropped -- a sparse matrix can carry them and they are not expression. Validated against Xenium pancreas: 377 genes, 2,607,168 values, 16.2 MB, 0.2s; INS/GCG/ACTA2 cell sets and values match the AnnData table exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured on Xenium pancreas (8.07M transcripts, 250px tiles), total overhead of the two render columns plus row-group fragmentation, against the 249.1 MB SpatialData default write: snappy 345.1 MB +38.5% 7.0s zstd 261.4 MB +4.9% 7.3s brotli 250.4 MB +0.5% 50.1s snappy barely compresses display_xy at all; zstd removes most of the overhead for no meaningful write cost, and brotli is 7x slower for another 4 points. Verified parquet-wasm 0.7.1 (celldega's pinned version) can read zstd: ZSTD = 5 is in its Compression enum and the zstd shims are compiled into the wasm bundle. This matters because an unsupported codec would fail only in the browser. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
add_spatial_tiling() tiles an existing store in place; xenium_spatially_tiled() goes from raw Xenium to a tiled store in one call. Both are single calls; the one-shot path is internally read -> write -> tile because the tiling rewrites written parquet. The manifest reuses Celldega's landscape_parameters.json keys (use_row_groups, tile_grid, row_group_files, technology, image_info) so its reader consumes it unchanged, and declares paths relative to the profile directory (../../points/transcripts/points.parquet) so Celldega can be pointed at that directory as base_url with no reader change. validate_manifest() fails loudly on the mistakes that would otherwise surface as a blank viewport: row-group counts disagreeing with the tile grid, file counts that cannot hold the declared row groups, CBG mappings pointing past the end, and declared files that do not exist. The re-runnability test caught the same idempotency bug in the shapes writer that was previously fixed for points: re-tiling appended duplicate display_geometry / cell_code columns rather than replacing them. Verified on Xenium pancreas: 15.0s to tile, 7,535 row groups across 19 files, read_zarr still returns 8,073,840 points / 140,702 shapes / (140702, 377) table, and canonical geometry is byte-identical after the reorder (including the 159 pre-existing invalid polygons, which are 10x source data, not corruption). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Normative description of celldega_regular_grid_v1: coordinate system, grid and tile numbering, row-group and multi-part file numbering, empty-tile handling, the render columns, feature and cell codes, CBG mapping, manifest layout, transport requirements, invalidation rules and a conformance checklist. Written viewer-independently with Celldega named as the reference client, and carrying the measurements behind the non-obvious choices (tile size, multi-file splitting, zstd) so the rationale survives the prototype. Also declares the 'columns' projection in the writer manifest fragments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… profile Writes a browser-ready WebP pyramid as Parquet row groups in Celldega's existing ImageRowGroupReader layout: zoom/tile_x/tile_y/image_data, one tile per row group, zoom_info in the schema metadata. Zoom numbering follows DeepZoom and was verified tile-for-tile against 'vips dzsave' on the same input: levels 0..11 for a 1300x700 image, level 11 = 3x2, level 10 = 2x1. So the output is interchangeable with tiles Celldega already produces. On Xenium pancreas it independently derives max_pyramid_zoom=16 and image_dimensions 34155x13770, matching the DegaFiles reference exactly. Encoding uses Pillow rather than libvips, so spatialdata-io gains no system dependency. Two memory measures matter on real images: the display window is taken from the smallest multiscale level rather than by scanning a 500-megapixel plane, and the 8-bit conversion runs in row blocks instead of materializing a float copy of the whole image. The canonical OME-Zarr image is untouched; this is explicitly a display cache, and the source element, dimensions, dtype, window, gamma, downsampling method and WebP settings are all recorded for invalidation. Also aligns the manifest with a real DegaFiles landscape_parameters.json (max_pyramid_zoom, image_dimensions, image_format, use_int_index, segmentation_approach, top-level tile_size). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ted memory Grouping rows by tile is a global sort -- a row at the end of the input can belong to the first tile -- so streaming the read alone is not enough. The streaming path makes two passes: stream the element one partition at a time and spill each row into a temporary file chosen by its destination chunk file, then sort each spill file independently and write its chunk. Peak memory scales with one partition plus one spill file rather than the dataset. Enabled automatically for partitioned (dask) elements; single-partition elements keep the single-pass path. A test asserts the two produce identical row groups, so they cannot silently diverge. Chunks pin the categorical dictionary to the element's full category list, since partitions observing different feature subsets would otherwise convert to incompatible Arrow dictionary types and could not be written to one file. Measured on Xenium Prime human skin (74,011,892 transcripts, 5,006 genes, 9.2x the pancreas): 94s to tile at 3.0 GB peak RSS, on a 17 GB machine where the in-memory path was estimated at 17-20 GB. 14,022 row groups across 36 files, all rows preserved, manifest validates, read_zarr returns the full element. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ixed-path assets Renames the profile from celldega_regular_grid_v1 to grid_files_v1. Naming it after one client contradicted the viewer-independent intent; nothing about the layout is Celldega-specific. Adds the files a client reads at fixed paths rather than through the manifest, which is what lets the profile directory stand in for a DegaFiles root so no client needs to know it is looking at a SpatialData store: - cell_metadata.parquet: per-cell centroids and names. Cells are the overview representation, so every centroid is needed up front. Centroids are not stored anywhere in a SpatialData store (the Xenium reader puts no x/y_centroid in obs), so they are derived from the shape geometry -- the same centroids already computed for tile assignment. Row order matches the table's obs order, so a cell's position here is its cell_code. - micron_to_image_transform.csv: the real micron-to-pixel affine. Coordinates are already in display pixels so this places nothing, but it drives the scale bar, and writing identity would make the scale bar wrong by the pixel size. - cell_clusters/: placeholder single-group clustering, since SpatialData does not require one and the Xenium reader does not load one. Without the file the viewer's category machinery fails on a 404. Also drops the design doc from the branch (it moves to the PR discussion) and removes supports_points_writer_hook, which was unused and untested. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The render columns no longer go into the canonical elements. Canonical points and shapes are still re-ordered into tile row groups -- that is what makes spatial subsetting cheap from Python -- but keep only their own columns. display_xy and feature_code go to <profile>/trx, display_geometry and cell_code to <profile>/cell_seg, with the same row-group layout. This fixes two separate problems with one change: 1. SpatialData.write() round-trip. A nested Arrow column cannot survive dask's parquet round-trip: it either fails outright on a schema mismatch or silently comes back as a string. A tiled store could therefore not be rewritten. Now it can, and a test asserts it. 2. Column projection. parquet-wasm corrupts the IPC stream whenever 'columns' is passed (0.7.1 and 0.7.2, apache-arrow 15 and 18, scalar and nested alike, even an empty array), which is what left the viewer showing only the image layer. A standalone render file makes projection unnecessary: reading every column of it already transfers only what is drawn. Also in this change: - All image channels are written, not just the first. Xenium morphology_focus has four; each gets its own pyramid and image_info entry with a distinct colour. Channel names are sanitised for paths, since names like 'ATP1A1/CD45/E-Cadherin' would otherwise create nested directories. - Render files are written before the canonical rewrite. Rewriting canonical replaces the files the lazy dask frame still points at, so the other order made the second read fail. - The streaming path reported render_only=False regardless; both paths now report it, with a test covering each. - Re-tiling drops render columns left in canonical by the previous layout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cornhundred
force-pushed
the
feat/xenium-celldega-regular-grid
branch
from
September 6, 2026 22:43
b4d94db to
d133ced
Compare
…e intensity Two fixes found by rendering the profile in a viewer. meta_gene.parquet had the wrong shape. A client reads it by convention, taking the gene list from the parquet *index* and colours from a 'color' column, alongside mean/std/max/non-zero. The profile wrote name/feature_code/is_gene as plain columns with no index and no colour, so the gene list came back empty: the viewer showed no transcript controls at all and logged nothing, because nothing had failed. It now matches that layout, with per-gene statistics computed column-wise from the sparse matrix (never densified -- a 5,006-gene panel would be several GB), and carries feature_code/is_gene alongside as profile additions. Image intensity is no longer stretched by default. The previous 1st-99.9th percentile window was applied on top of the viewer's own intensity slider and blew out the mid-tones: on a Xenium DAPI tile it took the mean from 1.3 to 37.4 with 1.2% of pixels saturated. The default is now the full dtype range, a linear mapping matching Celldega's pipeline, which saves raw values and leaves brightening to the slider. display_min/display_max still override it and are recorded in the manifest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pandas writes a named index as a column of that name. Our index was named 'name', so the parquet had a 'name' column and no '__index_level_0__' -- and a client looks for the gene list only under '__index_level_0__'. The gene list therefore still came back empty, with no error, exactly as before the schema fix. Leaving the index unnamed makes pandas emit '__index_level_0__', matching a DegaFiles meta_gene.parquet, whose index is also unnamed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… feature_code A client derives its integer gene id from a feature's row position in meta_gene.parquet, then colours transcripts by indexing an array built the same way. Sorting the frame by name interleaved the 164 control features among the 377 genes, so feature_code no longer matched that position and most colour lookups returned undefined -- which renders transparent rather than erroring. The catalog order is already the normative one (genes first, in var_names order, so a gene's feature_code is also its CBG row group), so writing the frame unsorted makes row position, feature_code and CBG row group all agree. Test uses a deliberately non-alphabetical var_names order with a control that sorts in the middle, so a reintroduced sort fails it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…integers Rounding to whole pixels put every transcript on an integer lattice. The displacement is scientifically negligible -- mean 0.08 um against Xenium's ~0.1-0.3 um localisation precision -- but visually obvious, because regular quantisation creates structure the eye reads as real. float32 was chosen over fixed-point integers despite storing less densely. On Xenium pancreas the render file measures: whole pixels (uint32) 27.6 MB 1 px max err 0.150 um fixed-point x4 (uint32) 43.3 MB 1/4 px max err 0.038 um float32 78.5 MB ~0.004 px max err 0 Fixed-point is denser, but requires the client to apply a scale read from the manifest; a client that ignores it renders everything silently offset by that factor. float32 needs no client-side arithmetic and cannot be misread. The extra ~50 MB is about 1% of a 3.5 GB store, which is a good trade for removing a whole class of silent-wrongness. Applies to display_geometry as well, so polygon vertices are no longer quantised either, and deck.gl still receives the Arrow buffer with no copy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hook was prototyped in core and then dropped, since nothing used it: tiling operates on an already-written store. This comment was the last reference to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only the defaults are Xenium (element names, the technology string); the profile itself reads generic SpatialData elements, which is why it lives in spatialdata-io rather than beside the Xenium reader. Uses a store with different element names, a different micron-to-pixel scale, and a control feature absent from var_names, so a Xenium assumption creeping into the writer fails here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.