Facing TSDB Error, What does it means ? How to debug it ?

[maheshrajrp]

Facing the following TSDB Error, Any idea how to debug it ? Documentation isn't providing much of information related to this.

ts=2025-02-28T07:52:57.162930402Z caller=ingester.go:2830 level=error msg="unable to open TSDB" err="failed to compact TSDB: /data/tsdb/anonymous: invalid magic number 0" user=anonymous
ts=2025-02-28T07:52:57.162961701Z caller=ingester.go:2865 level=error msg="error while opening existing TSDBs" err="unable to open TSDB for user anonymous: failed to compact TSDB: /data/tsdb/anonymous: invalid magic number 0"
ts=2025-02-28T07:52:57.163063599Z caller=mimir.go:914 level=error msg="module failed" module=ingester-service err="starting module ingester-service: invalid service state: Failed, expected: Running, failure: opening existing TSDBs: unable to open TSDB for user anonymous: failed to compact TSDB: /data/tsdb/anonymous: invalid magic number 0"
ts=2025-02-28T07:52:57.163153497Z caller=module_service.go:120 level=info msg="module stopped" module=ingester-ring
ts=2025-02-28T07:52:57.163118398Z caller=module_service.go:120 level=info msg="module stopped" module=active-groups-cleanup-service
ts=2025-02-28T07:52:57.163204695Z caller=memberlist_client.go:742 level=info msg="leaving memberlist cluster"
ts=2025-02-28T07:52:57.163209995Z caller=module_service.go:120 level=info msg="module stopped" module=runtime-config
ts=2025-02-28T07:52:57.479398231Z caller=module_service.go:120 level=info msg="module stopped" module=memberlist-kv
ts=2025-02-28T07:52:57.479751522Z caller=server_service.go:55 level=info msg="server stopped"
ts=2025-02-28T07:52:57.479767822Z caller=module_service.go:120 level=info msg="module stopped" module=server
ts=2025-02-28T07:52:57.479802721Z caller=module_service.go:120 level=info msg="module stopped" module=usage-stats
ts=2025-02-28T07:52:57.47981942Z caller=module_service.go:120 level=info msg="module stopped" module=sanity-check
ts=2025-02-28T07:52:57.482065164Z caller=module_service.go:120 level=info msg="module stopped" module=activity-tracker
ts=2025-02-28T07:52:57.482108163Z caller=mimir.go:901 level=info msg="Application stopped"
ts=2025-02-28T07:52:57.48222126Z caller=log.go:134 level=error msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:957\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"

[deniszh]

Looks like related to #2815

[maheshrajrp]

I'm more interested in understanding the issue. Like I've faced this issue two, and I don't know what to do. Eventually, end up reinstalling mimir completely.

[deniszh]

Ah, right, looks like it's not covered in runbook. In theory, you just need to delete corrupted (or all) data in /data/tsdb/anonymous

[maheshrajrp]

Thanks for responding, @deniszh. I'm not sure if this is corrupted. This is a fresh instance of Mimir. Typically I run the mimir setup in AKS with Azure Disk PVC, considering the nodeAffinity caused by Azure Disk, I was trying to switch to Azure File which caused this issue.

Basically I create a new setup with a working helm chart just the storageClass was modified causing this issue to happen. Maybe, I'll need to raise a separate issue.

Thanks.