Mimir - "failed to create memberlist - no private IP address found, and explicit IP not provided"

[girishms-sentient]

Describe the bug

Getting an error while configuring the mimir helm chart - "failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"

Environment

• Infrastructure: Amazon EKS
• Deployment tool: Helm

Additional Context

Logs:

caller=mimir.go:804 level=error msg="module failed" module=memberlist-kv err="invalid service state: Failed, expected: Running, failure: service &{0xc00087d4a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 false  7946  false [172.20.82.242] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false 0x3721400  false {    false  }} 0x3721400  [{ringDesc 0xdddd60}]} 0xc0009a8980 0x3721400 0xc000676620 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xdddd60}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc00005f5c0 0xc0007ec420 0xc0007ec4e0 0xc0007ec5a0 0xc0007ec660 0xc0007ec8a0 0xc0007ec720 0xc0007ec960 0xc0007ec7e0 0xc0007eeac0 0xc0007eeb00 0xc0007ecae0 0xc0007ecba0 0xc0007ecd20 0xc0007ecc60 0xc000013ee0 0xc0007be460 0xc000013ed0 0xc000013ed8 0xc0007eeb40 0xc0007eeb80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"

ts=2023-07-17T11:58:30.090844941Z caller=mimir.go:804 level=error msg="module failed" module=ring err="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc00087d4a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 false  7946  false [172.20.82.242] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false 0x3721400  false {    false  }} 0x3721400  [{ringDesc 0xdddd60}]} 0xc0009a8980 0x3721400 0xc000676620 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xdddd60}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc00005f5c0 0xc0007ec420 0xc0007ec4e0 0xc0007ec5a0 0xc0007ec660 0xc0007ec8a0 0xc0007ec720 0xc0007ec960 0xc0007ec7e0 0xc0007eeac0 0xc0007eeb00 0xc0007ecae0 0xc0007ecba0 0xc0007ecd20 0xc0007ecc60 0xc000013ee0 0xc0007be460 0xc000013ed0 0xc000013ed8 0xc0007eeb40 0xc0007eeb80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"

ts=2023-07-17T11:58:30.090956548Z caller=mimir.go:804 level=error msg="module failed" module=distributor-service err="failed to start distributor-service, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc00087d4a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 false  7946  false [172.20.82.242] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false 0x3721400  false {    false  }} 0x3721400  [{ringDesc 0xdddd60}]} 0xc0009a8980 0x3721400 0xc000676620 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xdddd60}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc00005f5c0 0xc0007ec420 0xc0007ec4e0 0xc0007ec5a0 0xc0007ec660 0xc0007ec8a0 0xc0007ec720 0xc0007ec960 0xc0007ec7e0 0xc0007eeac0 0xc0007eeb00 0xc0007ecae0 0xc0007ecba0 0xc0007ecd20 0xc0007ecc60 0xc000013ee0 0xc0007be460 0xc000013ed0 0xc000013ed8 0xc0007eeb40 0xc0007eeb80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"

Helm Values:

        mimir:
          structuredConfig:
            ingester:
              ring:
                final_sleep: 0s
                num_tokens: 512
                tokens_file_path: /data/tokens
                heartbeat_period: 2s
                heartbeat_timeout: 10s
                unregister_on_shutdown: true
                kvstore:
                  store: memberlist
                replication_factor: 2
                zone_awareness_enabled: true

            memberlist:
              abort_if_cluster_join_fails: false
              compression_enabled: false
              join_members:
              - {{ include "mimir.fullname" . }}-gossip-ring
              
            limits:
              compactor_blocks_retention_period: 604800s
              ingestion_rate: 500000
              max_global_series_per_metric: 9000000
              max_global_series_per_user: 9000000
              max_label_names_per_series: 60

            distributor:
              instance_limits:
                max_ingestion_rate: 0
                max_inflight_push_requests: 0
              remote_timeout: 30s
            blocks_storage:
              backend: s3
              bucket_store:
                max_chunk_pool_bytes: 12884901888 # 12GiB
              s3:
                endpoint: s3.us-west-2.amazonaws.com
                bucket_name: central-logging-mimir-bucket-block-storage
                insecure: true
              tsdb:
                dir: /data/tsdb

            alertmanager_storage:
              backend: s3
              s3:
                endpoint: s3.us-west-2.amazonaws.com
                bucket_name: central-logging-mimir-bucket-alertmanager-storage

            ruler_storage:
              backend: s3
              s3:
                endpoint: s3.us-west-2.amazonaws.com
                bucket_name: central-logging-mimir-bucket-ruler-storage

        serviceAccount:
          create: true
          name: mimir
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx

kubectl get po -n mimir
NAME                                        READY   STATUS             RESTARTS        AGE
grafana-688dd59f5-j9r5r                     1/1     Running            0               5h24m
mimir-alertmanager-0                        1/1     Running            0               2d20h
mimir-alertmanager-1                        0/1     CrashLoopBackOff   8 (3m12s ago)   19m
mimir-chunks-cache-0                        2/2     Running            0               2d21h
mimir-chunks-cache-1                        2/2     Running            0               2d21h
mimir-compactor-0                           0/1     CrashLoopBackOff   8 (2m57s ago)   19m
mimir-distributor-64949669b-n5vxf           1/1     Running            0               2d20h
mimir-distributor-64949669b-th9ww           1/1     Running            0               2d20h
mimir-distributor-84f9544554-wn45x          0/1     CrashLoopBackOff   8 (2m56s ago)   19m
mimir-index-cache-0                         2/2     Running            0               2d21h
mimir-ingester-zone-a-0                     0/1     CrashLoopBackOff   8 (3m17s ago)   19m
mimir-ingester-zone-b-0                     1/1     Running            0               2d20h
mimir-ingester-zone-c-0                     1/1     Running            0               2d20h
mimir-metadata-cache-0                      2/2     Running            0               2d21h
mimir-nginx-688d787b84-cjwnw                1/1     Running            0               2d21h
mimir-overrides-exporter-6986bd595b-trd8c   1/1     Running            0               19m
mimir-querier-574fd5fd45-l644v              0/1     CrashLoopBackOff   8 (3m9s ago)    19m
mimir-querier-7c774cff67-672wb              1/1     Running            0               2d20h
mimir-query-frontend-6487bfcb56-kjrwj       1/1     Running            0               19m
mimir-query-scheduler-67577d4565-kgj54      1/1     Running            0               19m
mimir-query-scheduler-67577d4565-ns6r9      1/1     Running            0               18m
mimir-results-cache-0                       2/2     Running            0               2d21h
mimir-results-cache-1                       2/2     Running            0               2d21h
mimir-rollout-operator-57759f5cfc-x466v     1/1     Running            0               2d21h
mimir-ruler-5776845d7c-wlx86                0/1     CrashLoopBackOff   8 (3m21s ago)   19m
mimir-ruler-69c96bc7b5-x9nzm                1/1     Running            0               2d20h
mimir-store-gateway-zone-a-0                0/1     CrashLoopBackOff   8 (3m9s ago)    19m
mimir-store-gateway-zone-b-0                1/1     Running            0               2d20h
mimir-store-gateway-zone-c-0                1/1     Running            0               2d20h

kubectl get svc -n mimir
NAME                             TYPE           CLUSTER-IP       EXTERNAL-IP                                                                  PORT(S)                      AGE
grafana                          LoadBalancer   172.20.112.150   k8s-mimir-grafana-xxxxxxx.amazonaws.com    80:30763/TCP                 5h24m
mimir-alertmanager               ClusterIP      172.20.223.132   <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-alertmanager-headless      ClusterIP      None             <none>                                                                       8080/TCP,9095/TCP,9094/TCP   2d21h
mimir-chunks-cache               ClusterIP      None             <none>                                                                       11211/TCP,9150/TCP           2d21h
mimir-compactor                  ClusterIP      172.20.91.81     <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-distributor                ClusterIP      172.20.148.133   <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-distributor-headless       ClusterIP      None             <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-gossip-ring                ClusterIP      None             <none>                                                                       7946/TCP                     2d21h
mimir-index-cache                ClusterIP      None             <none>                                                                       11211/TCP,9150/TCP           2d21h
mimir-ingester-headless          ClusterIP      None             <none>                                                                       9095/TCP                     2d21h
mimir-ingester-zone-a            ClusterIP      172.20.187.130   <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-ingester-zone-b            ClusterIP      172.20.99.45     <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-ingester-zone-c            ClusterIP      172.20.55.88     <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-metadata-cache             ClusterIP      None             <none>                                                                       11211/TCP,9150/TCP           2d21h
mimir-nginx                      LoadBalancer   172.20.26.220    k8s-mimir-mimirngi-xxxxx.amazonaws.com   80:30458/TCP                 2d21h
mimir-overrides-exporter         ClusterIP      172.20.116.112   <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-querier                    ClusterIP      172.20.119.97    <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-query-frontend             ClusterIP      172.20.255.190   <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-query-scheduler            ClusterIP      172.20.249.162   <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-query-scheduler-headless   ClusterIP      None             <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-results-cache              ClusterIP      None             <none>                                                                       11211/TCP,9150/TCP           2d21h
mimir-ruler                      ClusterIP      172.20.105.78    <none>                                                                       8080/TCP                     2d21h
mimir-store-gateway-headless     ClusterIP      None             <none>                                                                       9095/TCP                     2d21h
mimir-store-gateway-zone-a       ClusterIP      172.20.60.237    <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-store-gateway-zone-b       ClusterIP      172.20.144.229   <none>                                                                       8080/TCP,9095/TCP            2d21h
mimir-store-gateway-zone-c       ClusterIP      172.20.82.241    <none>                                                                       8080/TCP,9095/TCP            2d21h

Could you please assist me with resolving this issue?

### Tasks

### Tasks

[dimitarvdimitrov]

do the pods in the EKS cluster get public or private addresses? Mimir expects that pods run with at least one private interface