LogMergePolicy has no explicit delete-driven merge selection

[salvatorecampagna]

Description

I noticed this while investigating an index under sustained updates where deleted documents accumulated without being reclaimed.

LogMergePolicy.findMerges can leave heavily deleted segments unmerged when their level has too few segments to form a merge. New segments may reach a level too slowly to keep up with accumulated deletes, even under continued indexing.

The policy requires mergeFactor segments to form an initial candidate, but may merge fewer of them to stay within the size limit.

calibrateSizeByDeletes scales estimated segment sizes by (1 - delRatio). This can change level grouping and make previously oversized candidates eligible. Deletes therefore can trigger a merge indirectly through size-based selection, but there is no separate trigger based on the amount of reclaimable deletes.

Behavior

Selection tests using stock Lucene defaults. Each cell indicates whether a merge was selected.

LogByteSizeMergePolicy (mergeFactor=10, maxMergeMB=2048, minMergeMB=16):

Segments	0% deleted	20%	30%	50%	99%
2, 6, or 9	No	No	No	No	No
10 or 64	Yes	Yes	Yes	Yes	Yes

TieredMergePolicy (segmentsPerTier=8, maxMergedSegmentMB=5120, deletesPctAllowed=20):

Segments	0% deleted	20%	30%	50%	99%
2, 6, or 9	No	No	Yes	Yes	Yes
10 or 64	Yes	Yes	Yes	Yes	Yes

Reading across each row, LogByteSizeMergePolicy gives the same answer at every tested delete fraction in this setup. TieredMergePolicy changes its answer with the delete fraction alone.

With nine segments at 99% deletes, LogByteSizeMergePolicy selects nothing; TieredMergePolicy selects a merge. TieredMergePolicy uses its index-wide deletion budget, selecting merges at the first tested percentage above deletesPctAllowed=20.

I'll push a companion PR with a couple of tests that reproduce this behavior and link it here.

Reclamation latency

As a simplified illustration, with 50 MB flush segments and full mergeFactor = f merges, segment size at level k is 50 MB × f^k. Producing f such segments requires f^(k+1) flushes:

Level	Segment size	Full level size	Flushes required	Flushes (f = 10)
0	50 MB	500 MB	f	10
1	500 MB	5 GB	f²	100
2	5 GB	50 GB	f³	1,000

This assumes no size reduction from deletes or compression changes and a sufficiently large maximum merge size. The default 2048 MB limit already truncates merges of the 500 MB segments at level 1.

At a constant flush rate, higher levels take geometrically longer to fill and can hold tens of gigabytes while still below the segment-count threshold. Deletes left unreclaimed during that wait inflate storage relative to live data, increasing production storage requirements and costs; the exact recoverable bytes depend on the data and codec.

Discussion

#905 already established a delete-reclamation exception to TieredMergePolicy's anti-thrashing rule. That policy also allows singleton merges that reclaim deletes. Should the same principle apply to LogMergePolicy?

Two possible approaches are:

• Smaller adjacent merges: allow fewer than mergeFactor segments when a level holds substantial reclaimable data.
• Singleton rewrites: allow individual delete-heavy segments to be rewritten.

Both preserve adjacency and trade additional write amplification for shorter reclamation latency.

Related: #13226 reports similar symptoms under TieredMergePolicy, though the cause may differ.