fix(disk buffer): retry reads for published writer progress

[jjh5887]

Summary

Fix a rare disk_v2 reader stall by checking published writer progress before waiting for another writer notification.

If the reader reaches EOF on the current writer file but the ledger shows that the writer has already published unread records, the reader now briefly backs off and retries the read instead of waiting indefinitely for another notification. Writer-progress waits are also bounded so periodic re-checks remain a fallback safety net.

Vector configuration

N/A. This changes internal disk_v2 buffer reader behavior.

The production shape that motivated this is a low-volume source using a disk buffer, where the writer remains open and a single record may arrive after a long idle period.

How did you test this PR?

• Added reader_receives_record_after_cold_start_write, which starts the reader on an empty disk buffer, keeps the writer open, idles briefly, then writes and flushes one record.
• Ran cargo fmt --package vector-buffers.
• Ran cargo test -p vector-buffers variants::disk_v2::tests::cold_start::reader_receives_record_after_cold_start_write --no-default-features.
• Ran cargo test -p vector-buffers --no-default-features -- --test-threads=1.
• Ran cargo clippy -p vector-buffers --tests --no-default-features -- -D warnings.
• Ran make check-fmt.
• Ran make check-markdown.
• Ran cargo vdev check changelog-fragments --merge-base upstream/master.
• Ran make check-clippy.

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Related: buffers: ignored reader_exits_cleanly_when_writer_done_and_in_flight_acks test #23456
• Related: Events remain stuck in buffer, not flushed to Kafka sink when source finishes #23605

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[jjh5887]

I have read the CLA Document and I hereby sign the CLA

[pront]

Hi @jjh5887, thank you for this fix!

We recently merged a lot of buffers fixes. I suspect this bug was fixed already. You can test with the latest nightly to confirm. Thanks!