Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- Malformed files no longer crash the reader with a raw JDK exception when decoding VarBin, Dict, Bitpacked, ALP, Sparse, Chunked, or Struct columns — every case now fails as `VortexException`. ([ef982992](https://github.com/dfa1/vortex-java/commit/ef982992))
- Same hardening for RunEnd, Constant, zone-map stats, and Pco columns — every case now fails as `VortexException`. ([12d7466c](https://github.com/dfa1/vortex-java/commit/12d7466c))
- A constant Utf8/Binary column (`vortex.constant`) no longer eagerly allocates and copies `n` repetitions of its scalar on decode; it now broadcasts lazily in O(1) like every other constant type, closing an integer-overflow/OOM risk from a large row count. ([987fe412](https://github.com/dfa1/vortex-java/commit/987fe412))

### Changed

- `VarBinArray`'s representations (`OffsetMode`, `DictMode`, `ChunkedMode`, `ViewMode`, `SlicedMode`, `ConstantMode`) are now top-level classes (`VarBinOffsetArray`, `VarBinDictArray`, `VarBinChunkedArray`, `VarBinViewArray`, `VarBinSlicedArray`, `VarBinConstantArray`) instead of nested types, and `VarBinArray` is `non-sealed`, matching its sibling array-family interfaces. **Breaking** for any code referencing the old nested names directly. ([7e0d6e75](https://github.com/dfa1/vortex-java/commit/7e0d6e75))

### Added

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import io.github.dfa1.vortex.reader.array.MaterializedShortArray;
import io.github.dfa1.vortex.reader.array.ShortArray;
import io.github.dfa1.vortex.reader.array.VarBinArray;
import io.github.dfa1.vortex.reader.array.VarBinOffsetArray;

import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
Expand Down Expand Up @@ -128,7 +129,7 @@ static BoolArray bools(Arena arena, boolean... vs) {
return new MaterializedBoolArray(DType.BOOL, vs.length, seg.asReadOnly());
}

/// Builds a UTF-8 [VarBinArray] (`OffsetMode`, I64 offsets) from the given strings.
/// Builds a UTF-8 [VarBinArray] (`VarBinOffsetArray`, I64 offsets) from the given strings.
static VarBinArray utf8(Arena arena, String... vs) {
byte[][] rows = new byte[vs.length][];
for (int i = 0; i < vs.length; i++) {
Expand All @@ -137,7 +138,7 @@ static VarBinArray utf8(Arena arena, String... vs) {
return varbin(arena, DType.UTF8, rows);
}

/// Builds a binary [VarBinArray] (`OffsetMode`, I64 offsets) from the given byte rows.
/// Builds a binary [VarBinArray] (`VarBinOffsetArray`, I64 offsets) from the given byte rows.
static VarBinArray binary(Arena arena, byte[]... rows) {
return varbin(arena, DType.BINARY, rows);
}
Expand All @@ -156,6 +157,6 @@ private static VarBinArray varbin(Arena arena, DType dtype, byte[]... rows) {
pos += rows[i].length;
offsets.setAtIndex(ValueLayout.JAVA_LONG, i + 1, pos);
}
return new VarBinArray.OffsetMode(dtype, rows.length, bytes.asReadOnly(), offsets, PType.I64);
return new VarBinOffsetArray(dtype, rows.length, bytes.asReadOnly(), offsets, PType.I64);
}
}
8 changes: 4 additions & 4 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,18 +136,18 @@ decoder falls into one of three shapes:
| `vortex.null` | n/a | n/a | no per-row data |
| `vortex.bytebool` | Zero-copy | Zero-copy | mmap slice |
| `vortex.zigzag` | Lazy | Lazy | `LazyZigZagXxxArray` (I8/I16/I32/I64); broadcast → `LazyConstantXxxArray`, ADR 0010 + 0015 |
| `vortex.constant` | Lazy | Lazy | `LazyConstantXxxArray` (primitive + bool + decimal) + `VarBinArray.ConstantMode` (Utf8/Binary); per-row broadcast, no buffer, ADR 0015 |
| `vortex.constant` | Lazy | Lazy | `LazyConstantXxxArray` (primitive + bool + decimal) + `VarBinConstantArray` (Utf8/Binary); per-row broadcast, no buffer, ADR 0015 |
| `vortex.ext` | Zero-copy | Zero-copy | wraps storage |
| `vortex.runend` | Lazy | Lazy | `LazyRunEndXxxArray` (primitive + bool); Utf8/Binary stays Materialized (offset rebasing), ADR 0015 |
| `vortex.varbin` | Zero-copy | Zero-copy | bytes + offsets slices |
| `vortex.varbinview` | Lazy | Lazy | `VarBinArray.ViewMode` — keeps views + data buffers as mmap slices |
| `vortex.varbinview` | Lazy | Lazy | `VarBinViewArray` — keeps views + data buffers as mmap slices |
| `vortex.alp` | Lazy | Lazy | `LazyAlpXxxArray`; broadcast → `LazyConstantXxxArray`; patched stays Materialized, ADR 0010 + 0015 |
| `vortex.alprd` | Lazy | Lazy | `LazyAlpRdDoubleArray`/`LazyAlpRdFloatArray` — left/right + patches on access |
| `vortex.dict` | Lazy | Lazy | `DictXxxArray` (numeric) + `VarBinArray.DictMode` (string), ADR 0012 |
| `vortex.dict` | Lazy | Lazy | `DictXxxArray` (numeric) + `VarBinDictArray` (string), ADR 0012 |
| `vortex.sparse` | Lazy | Lazy | `LazySparseXxxArray` (primitive + bool); Utf8/Binary stays Materialized, ADR 0015 |
| `vortex.sequence` | Zero-copy | Zero-copy | synthetic (no data) |
| `vortex.struct` | Zero-copy | Zero-copy | `StructArray` wraps fields |
| `vortex.chunked` | Lazy | Lazy | `ChunkedXxxArray` (primitive/Bool) + `VarBinArray.ChunkedMode` (Utf8/Binary), ADR 0012 |
| `vortex.chunked` | Lazy | Lazy | `ChunkedXxxArray` (primitive/Bool) + `VarBinChunkedArray` (Utf8/Binary), ADR 0012 |
| `vortex.fsst` | Materialized | Materialized | symbol-table decompression |
| `vortex.list` | Lazy | Lazy | `ListArray` wraps elements + offsets children; shape inherits from child |
| `vortex.listview` | Lazy | Lazy | `ListViewArray` wraps elements + offsets + sizes children |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
import io.github.dfa1.vortex.reader.array.ShortArray;
import io.github.dfa1.vortex.reader.array.StructArray;
import io.github.dfa1.vortex.reader.array.VarBinArray;
import io.github.dfa1.vortex.reader.array.VarBinSlicedArray;
import io.github.dfa1.vortex.reader.layout.Layout;
import io.github.dfa1.vortex.reader.layout.LayoutDecodeContext;
import io.github.dfa1.vortex.reader.layout.ZonedStatsSchema;
Expand Down Expand Up @@ -777,7 +778,7 @@ private static Array sliceArray(Array full, long offset, long length, DType dtyp
case ByteArray a -> new OffsetByteArray(dtype, length, a, offset);
case BoolArray a -> new OffsetBoolArray(dtype, length, a, offset);
case NullArray ignored -> new NullArray(dtype, length);
case VarBinArray a -> new VarBinArray.SlicedMode(dtype, length, a, offset);
case VarBinArray a -> new VarBinSlicedArray(dtype, length, a, offset);
case StructArray s -> {
// A shared nested struct column is decoded once over the full range, then sliced
// per chunk by slicing each field into the same window. Field dtypes come from the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
/// Stitches the per-chunk arrays of one logical column into a single view, dispatching on the
/// column's [DType]. Each family gets its zero-copy composite shape (ADR 0012): primitive and
/// boolean chunks fold into the `ChunkedXxxArray` records, variable-length chunks into
/// [VarBinArray.ChunkedMode], and list chunks into a stitched [ListArray] whose bulk element data
/// [VarBinChunkedArray], and list chunks into a stitched [ListArray] whose bulk element data
/// stays zero-copy (the child element arrays are themselves combined recursively) while only the
/// small outer offsets table is rebuilt so per-chunk offsets — which each reset to zero — become one
/// cumulative table.
Expand Down Expand Up @@ -46,8 +46,8 @@ public static Array combine(DType dtype, long totalRows, List<Array> chunks,
}
Array data = switch (dtype) {
case DType.Bool ignored -> ChunkedBoolArray.of(dtype, totalRows, chunks);
case DType.Utf8 ignored -> VarBinArray.ChunkedMode.of(dtype, totalRows, chunks, arena);
case DType.Binary ignored -> VarBinArray.ChunkedMode.of(dtype, totalRows, chunks, arena);
case DType.Utf8 ignored -> VarBinChunkedArray.of(dtype, totalRows, chunks, arena);
case DType.Binary ignored -> VarBinChunkedArray.of(dtype, totalRows, chunks, arena);
case DType.List list -> combineLists(list, totalRows, chunks, arena);
case DType.Primitive prim -> combinePrimitive(prim.ptype(), dtype, totalRows, chunks);
default -> throw new VortexException("unsupported dtype for chunked layout: " + dtype);
Expand Down Expand Up @@ -77,7 +77,7 @@ private static Array combinePrimitive(PType ptype, DType dtype, long totalRows,
///
/// An entirely-null chunk decodes to a [NullArray] rather than a [ListArray] (e.g. a
/// `vortex.null` flat, or `vortex.constant` with a null scalar, #269). Mirroring the sibling
/// [VarBinArray.ChunkedMode] fix, such a chunk contributes `n` zero-length list rows: it adds no
/// [VarBinChunkedArray] fix, such a chunk contributes `n` zero-length list rows: it adds no
/// elements, so it is skipped by the recursive element combine, and its `n` outer offsets simply
/// repeat the running element count. Row-level nullability is preserved separately by the
/// caller's validity bitmap.
Expand Down
Loading