fix(segment): propagate forward-store writer open failure instead of crashing#579
Open
mrcs64 wants to merge 1 commit into
Open
fix(segment): propagate forward-store writer open failure instead of crashing#579mrcs64 wants to merge 1 commit into
mrcs64 wants to merge 1 commit into
Conversation
…crashing MemForwardStore::Open() stored the result of ChunkedFileWriter::Open() (a unique_ptr that is null when the underlying arrow FileOutputStream cannot be created, e.g. under file-descriptor pressure from many concurrent collection opens) but returned Status::OK() unconditionally. The failure was swallowed: the store reported success with a null writer_, and a later flush()/close() dereferenced it, crashing with SIGSEGV at memory_forward_store.cc:283. init_memory_components() also assigned memory_store_ to the member before Open() succeeded, so a failed init left a non-null, half-built memory_store_ that satisfied the `if (!memory_store_)` re-init guard and got flushed on close. - MemForwardStore::Open(): return an error when the writer cannot be opened. - MemForwardStore::flush(): guard against a null writer_ before writing. - SegmentImpl::init_memory_components(): roll back partial components on failure via AILEGO_DEFER so a failed init leaves memory_store_ null. Adds a mem_store_test case asserting Open() reports an error when the writer cannot be created.
7bf358c to
b5495d5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Opening and closing several collections concurrently in one process can crash. Under file-descriptor pressure,
arrow::io::FileOutputStream::Openfails, soChunkedFileWriter::Openreturnsnullptr.MemForwardStore::Openstores that null inwriter_but returnsStatus::OK()anyway, so the store looks initialized. Inserts proceed, and on closeMemForwardStore::flush()dereferences the nullwriter_atmemory_forward_store.cc:283, crashing with SIGSEGV.SegmentImpl::init_memory_components()compounds it: it assignsmemory_store_to the member beforeOpen()succeeds. A failed init then leaves a non-null, half-builtmemory_store_that satisfies theif (!memory_store_)re-init guard ininternal_insert()and gets flushed on close.Reproduction
Spawn N threads. Each creates and opens a collection in its own directory (one dense HNSW field plus an FTS field), upserts a few docs, runs a query, then drops it. Lowering
ulimit -ninduces the writer-open failure. This reliably SIGSEGVs, with the backtrace:Single-threaded runs, and runs with a high fd limit, are unaffected.
Fix
MemForwardStore::Openreturns an error when the writer could not be created, instead ofStatus::OK().MemForwardStore::flushreturns an error ifwriter_is null, soclose()cannot dereference it.SegmentImpl::init_memory_componentsrolls back partial state (memory_store_and the vector indexers) on any failure, so a failed init leavesmemory_store_null and is never flushed on close.Test
Adds a case to
mem_store_test.ccthat opens aMemForwardStoreat a path whose parent directory does not exist. Before the fix,Open()returned OK with a null writer and later crashed; now it returns an error.Related
Under much heavier concurrent-open load a separate crash appears inside RocksDB's filter registry (
__hash_table::find("rocksdb.BuiltinBloomFilter")). That is independent of this change and is tracked in #580.