Skip to content

fix(segment): propagate forward-store writer open failure instead of crashing#579

Open
mrcs64 wants to merge 1 commit into
alibaba:mainfrom
mrcs64:fix/forward-store-open-error-propagation
Open

fix(segment): propagate forward-store writer open failure instead of crashing#579
mrcs64 wants to merge 1 commit into
alibaba:mainfrom
mrcs64:fix/forward-store-open-error-propagation

Conversation

@mrcs64

@mrcs64 mrcs64 commented Jul 9, 2026

Copy link
Copy Markdown

Problem

Opening and closing several collections concurrently in one process can crash. Under file-descriptor pressure, arrow::io::FileOutputStream::Open fails, so ChunkedFileWriter::Open returns nullptr. MemForwardStore::Open stores that null in writer_ but returns Status::OK() anyway, so the store looks initialized. Inserts proceed, and on close MemForwardStore::flush() dereferences the null writer_ at memory_forward_store.cc:283, crashing with SIGSEGV.

SegmentImpl::init_memory_components() compounds it: it assigns memory_store_ to the member before Open() succeeds. A failed init then leaves a non-null, half-built memory_store_ that satisfies the if (!memory_store_) re-init guard in internal_insert() and gets flushed on close.

Reproduction

Spawn N threads. Each creates and opens a collection in its own directory (one dense HNSW field plus an FTS field), upserts a few docs, runs a query, then drops it. Lowering ulimit -n induces the writer-open failure. This reliably SIGSEGVs, with the backtrace:

MemForwardStore::flush
SegmentImpl::flush
CollectionImpl::flush_unsafe
CollectionImpl::close_unsafe
CollectionImpl::~CollectionImpl

Single-threaded runs, and runs with a high fd limit, are unaffected.

Fix

  • MemForwardStore::Open returns an error when the writer could not be created, instead of Status::OK().
  • MemForwardStore::flush returns an error if writer_ is null, so close() cannot dereference it.
  • SegmentImpl::init_memory_components rolls back partial state (memory_store_ and the vector indexers) on any failure, so a failed init leaves memory_store_ null and is never flushed on close.

Test

Adds a case to mem_store_test.cc that opens a MemForwardStore at a path whose parent directory does not exist. Before the fix, Open() returned OK with a null writer and later crashed; now it returns an error.

Related

Under much heavier concurrent-open load a separate crash appears inside RocksDB's filter registry (__hash_table::find("rocksdb.BuiltinBloomFilter")). That is independent of this change and is tracked in #580.

@CLAassistant

CLAassistant commented Jul 9, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

…crashing

MemForwardStore::Open() stored the result of ChunkedFileWriter::Open() (a
unique_ptr that is null when the underlying arrow FileOutputStream cannot be
created, e.g. under file-descriptor pressure from many concurrent collection
opens) but returned Status::OK() unconditionally. The failure was swallowed:
the store reported success with a null writer_, and a later flush()/close()
dereferenced it, crashing with SIGSEGV at memory_forward_store.cc:283.

init_memory_components() also assigned memory_store_ to the member before
Open() succeeded, so a failed init left a non-null, half-built memory_store_
that satisfied the `if (!memory_store_)` re-init guard and got flushed on close.

- MemForwardStore::Open(): return an error when the writer cannot be opened.
- MemForwardStore::flush(): guard against a null writer_ before writing.
- SegmentImpl::init_memory_components(): roll back partial components on failure
  via AILEGO_DEFER so a failed init leaves memory_store_ null.

Adds a mem_store_test case asserting Open() reports an error when the writer
cannot be created.
@mrcs64 mrcs64 force-pushed the fix/forward-store-open-error-propagation branch from 7bf358c to b5495d5 Compare July 10, 2026 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants