feat(clickhouse): add volume permissions support

charts/clickhouse/MANUAL_TESTING.md

@@ -0,0 +1,188 @@
+# Manual Testing Guide for Volume Permissions Fix (Issue #623)
+
+## The Problem (Issue #623)
+
+ClickHouse fails to start with persistent volumes due to permission denied errors:
+```
+filesystem error: in create_directories: Permission denied ["/var/lib/clickhouse/preprocessed_configs"]
+```
+
+**Root Cause**: Persistent volumes mount as `root:root` (0:0), but ClickHouse runs as user `101:101`. The container cannot create directories in the mounted volume.
+
+## Solution Implemented
+
+Added volume permissions init container that runs as root and fixes ownership before ClickHouse starts:
+```yaml
+initContainers:
+- name: volume-permissions
+  image: docker.io/bitnami/bitnami-shell:11-debian-11-r118
+  command: ["/bin/bash", "-ec"]
+  args:
+    - |
+      mkdir -p /var/lib/clickhouse
+      chown -R 101:101 /var/lib/clickhouse
+      find /var/lib/clickhouse -mindepth 1 -maxdepth 1 -not -name ".snapshot" -not -name "lost+found" | xargs -r chown -R 101:101
+  securityContext:
+    runAsUser: 0  # Root needed to change ownership
+  volumeMounts:
+    - name: data-volumeclaim-template
+      mountPath: /var/lib/clickhouse
+```
+
+## Manual Testing Instructions
+
+### Prerequisites
+- Kubernetes cluster (minikube, kind, or cloud cluster)
+- Helm 3.x installed
+- kubectl configured
+
+### Test 1: Reproduce the Issue (Failure Case)
+
+```bash
+# Install WITHOUT volume permissions (reproduces issue #623)
+helm install test-fail . \
+  --set volumePermissions.enabled=false \
+  --set persistence.enabled=true
+
+# Watch the failure
+kubectl get pods -w
+kubectl logs chi-test-fail-clickhouse-cluster-0-0-0 -f
+```
+
+**Expected Result**: ClickHouse pod fails with permission denied errors.
+
+### Test 2: Verify the Fix (Success Case)
+
+```bash
+# Install WITH volume permissions (the fix)
+helm install test-success . \
+  --set volumePermissions.enabled=true \
+  --set persistence.enabled=true
+
+# Watch success
+kubectl get pods -w
+kubectl logs chi-test-success-clickhouse-cluster-0-0-0 -f
+```
+
+**Expected Result**: ClickHouse starts successfully!
+
+### Test 3: Verify Init Container Runs
+
+```bash
+# Check init container is present in pod spec
+kubectl describe pod chi-test-success-clickhouse-cluster-0-0-0
+
+# Check init container logs
+kubectl logs chi-test-success-clickhouse-cluster-0-0-0 -c volume-permissions
+```
+
+**Expected Output**:
+```
++ mkdir -p /var/lib/clickhouse
++ chown -R 101:101 /var/lib/clickhouse
++ find /var/lib/clickhouse -mindepth 1 -maxdepth 1 -not -name .snapshot -not -name lost+found
++ xargs -r chown -R 101:101
+```
+
+### Test 4: Existing Claims (Real Issue #623 Scenario)
+
+This reproduces the exact scenario from the GitHub issue:
+
+```bash
+# Create existing PVCs first
+kubectl apply -f - <<EOF
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: signoz-clickhouse
+spec:
+  accessModes: ["ReadWriteOnce"]
+  resources:
+    requests:
+      storage: 5Gi
+EOF
+
+# Install with existing claim (issue #623 exact scenario)
+helm install test-existing . \
+  --set volumePermissions.enabled=true \
+  --set persistence.enabled=true \
+  --set persistence.existingClaim=signoz-clickhouse
+```
+
+## Verification Commands
+
+### Check ClickHouse is working
+```bash
+# Port forward to ClickHouse
+kubectl port-forward svc/test-success-clickhouse 8123:8123
+
+# Test ClickHouse HTTP interface
+curl http://localhost:8123/ping
+# Should return "Ok."
+```
+
+### Check file permissions inside pod
+```bash
+# Exec into ClickHouse pod
+kubectl exec -it chi-test-success-clickhouse-cluster-0-0-0 -- /bin/bash
+
+# Check ownership of directories
+ls -la /var/lib/clickhouse/
+# Should show: drwxr-xr-x 101 101 ... (ClickHouse user owns the directory)
+
+# Verify ClickHouse can create files
+touch /var/lib/clickhouse/test-file
+# Should work without permission errors
+```
+
+## Expected Results Summary
+
+| Test Case | Settings | Expected Result |
+|-----------|----------|-----------------|
+| **Test 1** | `volumePermissions.enabled=false` | ❌ ClickHouse fails with permission denied |
+| **Test 2** | `volumePermissions.enabled=true` | ✅ ClickHouse starts successfully |
+| **Test 3** | Check init containers | ✅ volume-permissions init container runs and fixes ownership |
+| **Test 4** | With existing PVCs | ✅ Works with existing claims (resolves issue #623) |
+
+## Key Points for Testing
+
+1. **CRITICAL**: Use `--set volumePermissions.enabled=true` (NOT `--set clickhouse.volumePermissions.enabled=true`)
+
+2. **Template Validation**: You can verify the fix renders correctly:
+   ```bash
+   helm template test . \
+     --set volumePermissions.enabled=true \
+     --set persistence.enabled=true | grep -A 20 "volume-permissions"
+   ```
+
+3. **Init Container Order**: The volume-permissions init container runs BEFORE the ClickHouse container starts.
+
+4. **Security Context**: The init container runs as `runAsUser: 0` (root) to change ownership, while ClickHouse runs as user `101:101`.
+
+## Troubleshooting
+
+### Common Issues:
+- **Init container fails**: Check if security policies (PSP/PSA) allow `runAsUser: 0`
+- **Image pull fails**: Ensure `bitnami/bitnami-shell:11-debian-11-r118` is accessible
+- **Still permission errors**: Check that `volumePermissions.enabled=true` is correctly set
+
+### Debug Commands:
+```bash
+# Check pod events
+kubectl describe pod <clickhouse-pod-name>
+
+# Check all init container statuses
+kubectl get pod <clickhouse-pod-name> -o jsonpath='{.status.initContainerStatuses}'
+
+# Check volume mounts
+kubectl get pod <clickhouse-pod-name> -o yaml | grep -A 10 volumeMounts
+```
+
+## Success Criteria
+
+✅ **Issue #623 is RESOLVED when**:
+1. `volumePermissions.enabled=true` allows ClickHouse to start successfully
+2. No more "Permission denied" errors for `/var/lib/clickhouse/preprocessed_configs`
+3. Init container successfully runs `chown -R 101:101 /var/lib/clickhouse`
+4. ClickHouse can create directories and files in persistent volumes
+5. Solution works with both new PVCs and existing claims

charts/clickhouse/templates/_helper.tpl

@@ -126,6 +126,29 @@ Return the initContainers image name for UDF
 {{- end -}}
 {{- end -}}
 
+{{/*
+Return the volume permissions image name
+*/}}
+{{- define "clickhouse.volumePermissions.image" -}}
+{{- $registryName := default .Values.volumePermissions.image.registry .Values.global.imageRegistry -}}
+{{- $repositoryName := .Values.volumePermissions.image.repository -}}
+{{- $tag := .Values.volumePermissions.image.tag | toString -}}
+{{- $digest := .Values.volumePermissions.image.digest | toString -}}
+{{- if $registryName -}}
+    {{- if $digest -}}
+        {{- printf "%s/%s@%s" $registryName $repositoryName $digest -}}
+    {{- else -}}
+        {{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
+    {{- end -}}
+{{- else -}}
+    {{- if $digest -}}
+        {{- printf "%s@%s" $repositoryName $digest -}}
+    {{- else -}}
+        {{- printf "%s:%s" $repositoryName $tag -}}
+    {{- end -}}
+{{- end -}}
+{{- end -}}
+
 {{/*
 Return the proper clickhouse image name
 */}}
@@ -167,14 +190,17 @@ nodePort: null
 Return the proper Image Registry Secret Names.
 */}}
 {{- define "clickhouse.imagePullSecrets" -}}
-{{- if or .Values.global.imagePullSecrets .Values.imagePullSecrets }}
+{{- if or .Values.global.imagePullSecrets .Values.imagePullSecrets .Values.volumePermissions.image.pullSecrets }}
 imagePullSecrets:
 {{- range .Values.global.imagePullSecrets }}
   - name: {{ . }}
 {{- end }}
 {{- range .Values.imagePullSecrets }}
   - name: {{ . }}
 {{- end }}
+{{- range .Values.volumePermissions.image.pullSecrets }}
+  - name: {{ . }}
+{{- end }}
 {{- end }}
 {{- end }}
 

charts/clickhouse/templates/clickhouse-instance/clickhouse-instance.yaml

@@ -193,6 +193,42 @@ spec:
 
           {{- if .Values.initContainers.enabled }}
           initContainers:
+            {{- if and .Values.volumePermissions.enabled (or .Values.persistence.enabled .Values.templates.volumeClaimTemplates) }}
+            - name: volume-permissions
+              image: {{ include "clickhouse.volumePermissions.image" . }}
+              imagePullPolicy: {{ default "" .Values.volumePermissions.image.pullPolicy | quote }}
+              command:
+                - /bin/bash
+              args:
+                - -ec
+                - |
+                  mkdir -p /var/lib/clickhouse
+                  chown -R {{ .Values.securityContext.runAsUser }}:{{ .Values.securityContext.fsGroup }} /var/lib/clickhouse
+                  find /var/lib/clickhouse -mindepth 1 -maxdepth 1 -not -name ".snapshot" -not -name "lost+found" | xargs -r chown -R {{ .Values.securityContext.runAsUser }}:{{ .Values.securityContext.fsGroup }}
+              {{- if .Values.volumePermissions.containerSecurityContext.enabled }}
+              securityContext: {{- omit .Values.volumePermissions.containerSecurityContext "enabled" | toYaml | nindent 16 }}
+              {{- end }}
+              {{- if .Values.volumePermissions.resources }}
+              resources: {{- toYaml .Values.volumePermissions.resources | nindent 16 }}
+              {{- end }}
+              volumeMounts:
+              {{- if and (not .Values.templates.volumeClaimTemplates) (not .Values.persistence.enabled) }}
+                - name: data-volume
+                  mountPath: /var/lib/clickhouse
+              {{- end }}
+              {{- if and (.Values.persistence.enabled) (.Values.persistence.existingClaim) }}
+                - name: existing-volumeclaim
+                  mountPath: /var/lib/clickhouse
+              {{- else if .Values.persistence.enabled }}
+                - name: data-volumeclaim-template
+                  mountPath: /var/lib/clickhouse
+              {{- else if .Values.templates.volumeClaimTemplates }}
+                {{- range .Values.templates.volumeClaimTemplates }}
+                - name: {{ .name }}
+                  mountPath: /var/lib/clickhouse
+                {{- end }}
+              {{- end }}
+            {{- end }}
             {{- if .Values.initContainers.udf.enabled }}
             - name: {{ include "clickhouse.fullname" . }}-udf-init
               image: {{ include "clickhouse.initContainers.udf.image" . }}

charts/clickhouse/templates/clickhouse-operator/secret.yaml

@@ -3,7 +3,7 @@ apiVersion: v1
 kind: Secret
 metadata:
   name: {{ include "clickhouseOperator.fullname" . }}
-  namespace: {{ .Release.Namespace }}
+  namespace: {{ .Values.namespace | default .Release.Namespace }}
   labels: {{ include "clickhouseOperator.labels" . | nindent 4 }}
 type: Opaque
 data: