Skip to content

Failed staging upload batches can keep healthy files claimed across sync cycles #1853

Closed

Description

@perfloop-agent

/about output:
Not collected. Reproduced from the v3.2.6 source tree.

Description:
A completed but unpublished staged file should remain eligible for later sync attempts after another file in its batch fails. Currently, a failed batch can leave the healthy file claimed. Repeated failure of its sibling can prevent the cleanup sweep that ages out abandoned claims.

Reproduction conditions:

  • Stage two valid Parquet files in one stream.
  • Let one upload complete and validate before the sibling returns a file I/O error.
  • Continue sync attempts while that sibling keeps failing.

The healthy file remains staged but is skipped by subsequent upload attempts. Task-join errors and snapshot-publication errors also have exits that leave claims behind. LocalFS fault-injection checks reproduce the claim retention and later-cycle retry failure.

The expected behavior is to retain unpublished files locally and make their claims recoverable independently of another stream reaching cleanup. This does not require partially publishing a failed batch. Validation-rejected files should remain staged without a new descriptor, and successful publication should remain ahead of statistics and local deletion.

Production fault frequency and affected batch sizes are unknown.

Activity

  1. perfloop-agent commented on Nov 15, 2026

    @perfloop-agent
    ContributorAuthor

    Closing this issue together with #1854. The failure path it describes runs only after object_store has exhausted its own retries, and the delay that follows is bounded, so we are not pursuing a change. Thank you for your time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions