thanos compactor removing metrics if upload failed with error "upload s3 object: We encountered an internal error, please try again"

[bobykus31]

Thanos, Prometheus and Golang version used:

thanos --version
thanos, version 0.39.2 (branch: HEAD, revision: 36681af)
build user: root@b918afaaed61
build date: 20250721-06:26:54
go version: go1.24.0
platform: linux/amd64
tags: netgo

Object Storage Provider:
SeaweedFS

What happened:

Oct 30 07:43:36 thanos-compact3 thanos[7906]: ts=2025-10-30T07:43:36.004499301Z caller=compact.go:570 level=error msg="retriable error" err="compaction: group 0@12539261079432647024: upload of 01K8T12404GZVN4RJKSNG9KJ4F failed: upload chunks: upload file /var/lib/thanos/compact/0@12539261079432647024/01K8T12404GZVN4RJKSNG9KJ4F/chunks/000001 as 01K8T12404GZVN4RJKSNG9KJ4F/chunks/000001: upload s3 object: We encountered an internal error, please try again."

The 01K8T12404GZVN4RJKSNG9KJ4F is removed from local storage

find /var/lib/thanos/ -name 01K8T12404GZVN4RJKSNG9KJ4F

and from object storage

s3cmd ls s3://promproxy-metrics/01K8T12404GZVN4RJKSNG9KJ4F/

What you expected to happen:

block remain on local storage and re uploaded

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Oct 30 07:42:04 thanos-compact3 thanos[7906]: ts=2025-10-30T07:42:04.817693785Z caller=handler.go:87 level=info caller=compact.go:790 time=2025-10-30T07:42:04.817648339Z msg="Found overlapping blocks during compaction" ulid=01K8T12404GZVN4RJKSNG9KJ4F

Oct 30 07:43:06 thanos-compact3 thanos[7906]: ts=2025-10-30T07:43:06.135855794Z caller=handler.go:87 level=info caller=compact.go:530 time=2025-10-30T07:43:06.135828382Z msg="compact blocks" count=3 mint=1760436000000 maxt=1760443200000 ulid=01K8T12404GZVN4RJKSNG9KJ4F sources="[01K7HCWRQ1112185MT68RBB2KG 01K7HD4X5J5NQ1ZKCP4W96M04Q 01K7H6988QEB2GFKW1HK1TYTQ2]" duration=1m1.341975081s

Oct 30 07:43:06 thanos-compact3 thanos[7906]: ts=2025-10-30T07:43:06.538376553Z caller=compact.go:1265 level=info group="0@{monitor=\"production\", replica=\"prom2\"}" groupKey=0@12539261079432647024 msg="compacted blocks" new=[01K8T12404GZVN4RJKSNG9KJ4F] duration=1m1.744532399s duration_ms=61744 overlapping_blocks=true blocks="[/var/lib/thanos/compact/0@12539261079432647024/01K7HCWRQ1112185MT68RBB2KG /var/lib/thanos/compact/0@12539261079432647024/01K7HD4X5J5NQ1ZKCP4W96M04Q /var/lib/thanos/compact/0@12539261079432647024/01K7H6988QEB2GFKW1HK1TYTQ2]"

Oct 30 07:43:36 thanos-compact3 thanos[7906]: ts=2025-10-30T07:43:36.004499301Z caller=compact.go:570 level=error msg="retriable error" err="compaction: group 0@12539261079432647024: upload of 01K8T12404GZVN4RJKSNG9KJ4F failed: upload chunks: upload file /var/lib/thanos/compact/0@12539261079432647024/01K8T12404GZVN4RJKSNG9KJ4F/chunks/000001 as 01K8T12404GZVN4RJKSNG9KJ4F/chunks/000001: upload s3 object: We encountered an internal error, please try again."

Environment:

cat /etc/issue
Ubuntu 24.04.2 LTS

uname -a
Linux thanos-compact3 6.8.0-60-generic...

[shivansh-gohem]

Hi @bobykus31, thanks for the detailed report!

I'd like to work on fixing this issue. The problem is clear: when S3 upload fails with a transient/retriable error, the compacted block is being deleted from both local and object storage, causing data loss.

Here's my approach:

1. Analyze the error handling logic in cmd/compact/compact.go around the deletion code
2. Distinguish between transient (retriable) and permanent errors
3. Ensure blocks are NOT deleted on transient S3 errors
4. Add test cases to verify the block persists when upload fails with transient errors
5. Add retry logic or proper cleanup handling

Before I dive in, a few questions:

• Should failed uploads be retried automatically, or should the block remain in local storage for the next compaction cycle to retry?
• Are there existing error handling patterns in the codebase for distinguishing transient vs permanent S3 errors?

I don't have access to SeaweedFS, but I'll use Go mocks and existing Thanos test infrastructure to reproduce and verify the fix.

Looking forward to working on this!

[bobykus31]

Thank you very much, @shivansh-gohem, for your initiative!

Since the error message says

msg="retriable error"

supposedly the failed uploads is to be retried automatically.

No idea about existing error handling patterns in the codebase.

From SeaweedFS side the error is

500 Internal Server Error

just FYI, which is reported to SeaweedFS developers in the bug reports

seaweedfs/seaweedfs#7060