diff --git a/deploy/helm/README.md b/deploy/helm/README.md index 3a29f1c7..5048ba16 100644 --- a/deploy/helm/README.md +++ b/deploy/helm/README.md @@ -40,13 +40,20 @@ For a dev/latest install, set `operator.image.useLatest=true`, `console.image.us After the chart is running you can create a cluster by applying a `KafscaleCluster` resource (see `config/samples/` for an example). The console service is exposed as a ClusterIP by default; enable ingress by toggling `.Values.console.ingress`. -### External Broker Access +### External Kafka Access -Broker Services are created by the operator. For external clients, configure the -`KafscaleCluster` spec to expose a LoadBalancer or NodePort and set the advertised -address/port so Kafka clients learn the reachable endpoint. +**Recommended:** enable the chart proxy (`proxy.enabled=true`) and set +`proxy.advertisedHost` to the address external clients should use. The proxy is +the single Kafka entrypoint and supports broker scaling without client +reconfiguration. For production, use `proxy.service.type=LoadBalancer`. For +local clusters (kind), pin `proxy.service.nodePort` — see +`deploy/helm/kafscale/README.md`. -See `docs/operations.md` for a full example and TLS termination guidance. +**Optional:** broker Services are created by the operator. To expose brokers +directly (bypassing the proxy), configure `KafscaleCluster` `spec.brokers.service` +and `advertisedHost` / `advertisedPort`. + +See `docs/operations.md` for full examples and TLS termination guidance. ### MCP @@ -61,6 +68,10 @@ The MCP service is optional and disabled by default. Enable it with `mcp.enabled | `operator.etcdEndpoints` | List of etcd endpoints the operator will connect to. | `["http://etcd:2379"]` | | `console.service.type` | Kubernetes service type for the console. | `ClusterIP` | | `console.ingress.*` | Optional ingress configuration for exposing the console. | disabled | +| `proxy.enabled` | Deploy the Kafka-aware proxy for external client access. | `false` | +| `proxy.advertisedHost` | Address Kafka clients should connect to (public DNS or node IP). | `""` | +| `proxy.service.type` | Proxy Service type (`LoadBalancer`, `NodePort`, `ClusterIP`). | `LoadBalancer` | +| `proxy.service.nodePort` | Pin NodePort when `type=NodePort` (range `30000–32767`). | `""` | | `mcp.enabled` | Deploy the MCP service. | `false` | | `mcp.namespace.name` | Namespace to deploy the MCP service into. | `kafscale-mcp` | | `mcp.ingress.*` | Optional ingress configuration for exposing the MCP service. | disabled | diff --git a/deploy/helm/kafscale/README.md b/deploy/helm/kafscale/README.md index e2706e44..31905602 100644 --- a/deploy/helm/kafscale/README.md +++ b/deploy/helm/kafscale/README.md @@ -17,7 +17,9 @@ limitations under the License. # KafScale Helm Chart -Helm chart for deploying KafScale components including the operator, console, proxy, LFS proxy, and MCP server. +Helm chart for deploying KafScale components including the operator, console, +Kafka proxy, and MCP server. LFS is configured on the `KafscaleCluster` CRD +(`spec.lfsProxy`) and deployed by the operator. ## Prerequisites @@ -47,8 +49,7 @@ helm upgrade --install kafscale ./deploy/helm/kafscale \ |-----------|-------------|---------| | **Operator** | KafScale cluster operator | Enabled | | **Console** | Web-based management UI | Enabled | -| **Proxy** | Kafka protocol proxy | Disabled | -| **LFS Proxy** | Large File Support proxy | Disabled | +| **Proxy** | Kafka protocol proxy (recommended external entrypoint) | Disabled | | **MCP** | Model Context Protocol server | Disabled | ## Quick Start Examples @@ -59,32 +60,30 @@ helm upgrade --install kafscale ./deploy/helm/kafscale \ helm upgrade --install kafscale ./deploy/helm/kafscale ``` -### With LFS Proxy and MinIO +### With Kafka Proxy (external access) ```bash helm upgrade --install kafscale ./deploy/helm/kafscale \ - --set lfsProxy.enabled=true \ - --set lfsProxy.http.enabled=true \ - --set lfsProxy.s3.bucket=kafscale \ - --set lfsProxy.s3.endpoint=http://minio:9000 \ - --set lfsProxy.s3.accessKey=minioadmin \ - --set lfsProxy.s3.secretKey=minioadmin \ - --set lfsProxy.s3.forcePathStyle=true + --set proxy.enabled=true \ + --set proxy.replicaCount=2 \ + --set proxy.service.type=LoadBalancer \ + --set proxy.advertisedHost=kafka.example.com \ + --set proxy.etcdEndpoints[0]=http://kafscale-etcd-client.kafscale.svc.cluster.local:2379 ``` ### LFS Demo Stack -Deploy the full LFS demo stack with browser UI: +Deploy the browser demo UI (`lfsDemos`) and enable LFS on your cluster CRD: ```bash helm upgrade --install kafscale ./deploy/helm/kafscale \ -n kafscale-demo --create-namespace \ - -f ./deploy/helm/kafscale/values-lfs-demo.yaml \ - --set lfsProxy.s3.endpoint=http://minio:9000 \ - --set lfsProxy.s3.accessKey=minioadmin \ - --set lfsProxy.s3.secretKey=minioadmin + -f ./deploy/helm/kafscale/values-lfs-demo.yaml ``` +Then apply a `KafscaleCluster` with `spec.lfsProxy.enabled=true` and S3 settings. +See the [LFS on KafscaleCluster](#lfs-on-kafscalecluster) section below. + ## Values Files | File | Description | @@ -102,14 +101,12 @@ See [values.yaml](values.yaml) for the full list of configurable parameters. |---------|-------------| | `operator.*` | KafScale operator settings | | `console.*` | Console UI settings | -| `proxy.*` | Kafka proxy settings | -| `lfsProxy.*` | LFS proxy settings | -| `lfsProxy.http.*` | HTTP API settings | -| `lfsProxy.http.cors.*` | CORS configuration | -| `lfsProxy.s3.*` | S3 storage backend | -| `lfsProxy.ingress.*` | HTTP ingress | -| `lfsDemos.*` | Demo applications | +| `proxy.*` | Kafka proxy settings (external entrypoint) | +| `lfsDemos.*` | Optional LFS browser demo UI | | `mcp.*` | MCP server settings | +| `.podSecurityContext` | Pod-level PSA `restricted` defaults (UID/GID `10001`, `runAsNonRoot`, `seccompProfile`) | +| `.containerSecurityContext` | Container hardening (`readOnlyRootFilesystem`, dropped capabilities) | +| `proxy.affinity` | Pod affinity override; when empty the chart applies soft hostname anti-affinity | ### Proxy Service @@ -136,9 +133,12 @@ proxy: nodePort: 30092 ``` -## LFS Proxy +## LFS on KafscaleCluster -The LFS Proxy implements the claim-check pattern for large Kafka messages: +Large File Support (LFS) uses the claim-check pattern: blobs land in S3 and Kafka +carries a pointer. The operator deploys an LFS proxy Deployment when you enable +`spec.lfsProxy` on your `KafscaleCluster` (not via Helm `lfsProxy.*` values — +that chart surface was removed in v1.6). ``` ┌─────────┐ ┌───────────┐ ┌─────────┐ @@ -153,59 +153,40 @@ The LFS Proxy implements the claim-check pattern for large Kafka messages: └───────────┘ ``` -### Enable HTTP API +Example (`KafscaleCluster` fragment): ```yaml -lfsProxy: - enabled: true - http: +spec: + lfsProxy: enabled: true - port: 8080 - cors: + http: enabled: true - allowOrigins: ["*"] + port: 8080 + s3: + bucket: my-lfs-bucket + region: us-east-1 + credentialsSecretRef: s3-credentials ``` -### S3 Configuration - -```yaml -lfsProxy: - s3: - bucket: my-lfs-bucket - region: us-east-1 - endpoint: "" # Leave empty for AWS S3 - existingSecret: s3-credentials # Recommended for production -``` - -For detailed LFS proxy documentation, see [docs/lfs-proxy/helm-deployment.md](../../../docs/lfs-proxy/helm-deployment.md). +`bucket` and `region` are required. Do not use the blocklisted default bucket +name `kafscale-lfs`. ### HTTP API Specification (OpenAPI/Swagger) -The LFS Proxy HTTP API is documented using OpenAPI 3.0: - | Resource | Location | |----------|----------| | **OpenAPI Spec** | [`api/lfs-proxy/openapi.yaml`](../../../api/lfs-proxy/openapi.yaml) | -| **Swagger UI** | Import the spec into [Swagger Editor](https://editor.swagger.io) or [Stoplight](https://stoplight.io) | - -**API Endpoints:** +| **Proxy spec (embedded)** | [`cmd/proxy/openapi.yaml`](../../../cmd/proxy/openapi.yaml) | | Endpoint | Method | Description | |----------|--------|-------------| | `/lfs/produce` | POST | Upload blob to S3, produce pointer to Kafka | | `/lfs/download` | POST | Get presigned URL or stream blob from S3 | -| `/readyz` | GET | Kubernetes readiness probe | -| `/livez` | GET | Kubernetes liveness probe | -| `/metrics` | GET | Prometheus metrics (port 9095) | - -**Example: View API spec locally:** -```bash -# Using Swagger UI Docker -docker run -p 8081:8080 -e SWAGGER_JSON=/spec/openapi.yaml \ - -v $(pwd)/api/lfs-proxy:/spec swaggerapi/swagger-ui +| `/readyz` | GET | Readiness probe | +| `/livez` | GET | Liveness probe | -# Open http://localhost:8081 -``` +For local development without Kubernetes, see +[`deploy/docker-compose/README.md`](../../docker-compose/README.md). ## Browser Demo (E72) @@ -269,58 +250,27 @@ helm upgrade --install kafscale ./deploy/helm/kafscale \ Note: if you set `global.imageRegistry`, individual component image repositories inherit it. -## Monitoring - -### Enable ServiceMonitor - -```yaml -lfsProxy: - metrics: - enabled: true - serviceMonitor: - enabled: true - interval: 30s -``` - -### Enable PrometheusRule - -```yaml -lfsProxy: - metrics: - prometheusRule: - enabled: true -``` - ## Security ### Credentials Best Practices -1. **Use existing secrets** instead of inline values: +1. **Use Kubernetes secrets** for S3 credentials on the `KafscaleCluster`: ```bash kubectl create secret generic s3-creds \ --from-literal=AWS_ACCESS_KEY_ID=xxx \ --from-literal=AWS_SECRET_ACCESS_KEY=xxx ``` ```yaml - lfsProxy: - s3: - existingSecret: s3-creds + spec: + lfsProxy: + s3: + credentialsSecretRef: s3-creds ``` -2. **Enable API key** for HTTP endpoints: - ```yaml - lfsProxy: - http: - apiKey: "your-secure-key" - ``` +2. **Enable HTTP API key** via `spec.lfsProxy.http.apiKeySecretRef`. -3. **Restrict CORS origins** in production: - ```yaml - lfsProxy: - http: - cors: - allowOrigins: ["https://app.example.com"] - ``` +3. **Restrict external Kafka access** with `proxy.service.loadBalancerSourceRanges` + or cloud LB annotations. ## Uninstall @@ -330,7 +280,6 @@ helm uninstall kafscale -n kafscale-system ## Documentation -- [LFS Proxy Helm Deployment](../../../docs/lfs-proxy/helm-deployment.md) -- [LFS Proxy Data Flow](../../../docs/lfs-proxy/data-flow.md) -- [LFS SDK Documentation](../../../docs/lfs-proxy/sdk-solution.md) -- [Operations Guide](../../../docs/operations.md) +- [Operations Guide](../../../docs/operations.md) — external access, proxy, TLS +- [Docker Compose (local LFS)](../../docker-compose/README.md) +- [OpenAPI spec](../../../api/lfs-proxy/openapi.yaml) diff --git a/deploy/helm/kafscale/values-lfs-demo.yaml b/deploy/helm/kafscale/values-lfs-demo.yaml index 13f8d443..b00a0684 100644 --- a/deploy/helm/kafscale/values-lfs-demo.yaml +++ b/deploy/helm/kafscale/values-lfs-demo.yaml @@ -16,20 +16,21 @@ # ============================================================================= # LFS Demo Stack Values # ============================================================================= -# This values file enables the LFS proxy with HTTP endpoint and browser demo. +# This values file enables the LFS browser demo UI (lfsDemos). +# +# LFS itself is configured on your KafscaleCluster CRD (spec.lfsProxy), not via +# Helm lfsProxy values — see deploy/helm/kafscale/README.md. # # Usage: # helm upgrade --install kafscale ./deploy/helm/kafscale \ -# -f ./deploy/helm/kafscale/values-lfs-demo.yaml \ -# --set lfsProxy.s3.bucket=my-bucket \ -# --set lfsProxy.s3.endpoint=http://minio:9000 \ -# --set lfsProxy.s3.accessKey=minioadmin \ -# --set lfsProxy.s3.secretKey=minioadmin +# -f ./deploy/helm/kafscale/values-lfs-demo.yaml +# +# Then apply a KafscaleCluster with spec.lfsProxy.enabled=true and S3 settings. # # For local development with port-forward: -# kubectl port-forward svc/kafscale-lfs-proxy 8080:8080 & -# kubectl port-forward svc/kafscale-lfs-demo-e72 3000:80 & -# open http://localhost:3000 +# kubectl port-forward svc/-lfs-proxy 8080:8080 & +# kubectl port-forward svc/kafscale-lfs-demo-e72 30072:80 & +# open http://localhost:30072 # ============================================================================= # Enable LFS Proxy with HTTP endpoint diff --git a/docs/operations.md b/docs/operations.md index c85a3d17..5413aa35 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -72,6 +72,8 @@ helm upgrade --install kafscale deploy/helm/kafscale \ - **TLS** – Terminate TLS at your ingress or service mesh; broker/console TLS env flags are not wired in v1. - **Admin APIs** – Create/Delete Topics are enabled by default. Set `KAFSCALE_ALLOW_ADMIN_APIS=false` on broker pods to disable them, and gate external access via mTLS, ingress auth, or network policies. - **Network policies** – If your cluster enforces policies, allow the operator + brokers to reach etcd and S3 endpoints and lock everything else down. +- **Pod Security (restricted)** – Chart-templated workloads (operator, proxy, console, MCP) ship PSA `restricted`-compatible `securityContext` defaults (`runAsNonRoot`, dropped capabilities, `readOnlyRootFilesystem`, `seccompProfile: RuntimeDefault`, UID/GID `10001`). Override per component via `.podSecurityContext` and `.containerSecurityContext`. The proxy mounts a writable `/tmp` `emptyDir` for LFS verify temp files. +- **Pod placement** – Operator-managed broker and etcd StatefulSets, and the chart proxy Deployment (when `proxy.affinity` is empty), use soft (`preferredDuringSchedulingIgnoredDuringExecution`) pod anti-affinity on `kubernetes.io/hostname` so replicas spread across nodes on multi-node clusters while still scheduling on single-node dev clusters. Set `proxy.affinity` explicitly to replace the chart default; broker/etcd affinity overrides via the CR are tracked separately ([#164](https://github.com/KafScale/platform/issues/164)). - **ACLs (v1.5)** – Optional, basic ACL enforcement is available at the broker. Identity comes from Kafka `client.id` until SASL is introduced. Configure via `KAFSCALE_ACL_ENABLED` plus either `KAFSCALE_ACL_JSON` or `KAFSCALE_ACL_FILE`. Use `KAFSCALE_ACL_FAIL_OPEN=true` to allow traffic when the ACL config is missing/invalid (default is fail-closed). - **Principal source** – Set `KAFSCALE_PRINCIPAL_SOURCE` to `client_id` (default), `remote_addr`, or `proxy_addr`. Use `proxy_addr` with `KAFSCALE_PROXY_PROTOCOL=true` to derive principals from a trusted TCP proxy (PROXY protocol v1/v2). - **Auth denials** – Broker logs emit a rate-limited `authorization denied` entry with principal/action/resource context. @@ -376,11 +378,16 @@ Cost-optimized (accepts a larger loss window after crash): ### Proxy - `KAFSCALE_PROXY_ADDR` – Proxy listen address (host:port). +- `KAFSCALE_PROXY_HEALTH_ADDR` – Health/readiness listen address (default `:9094`; exposes `/readyz` and `/livez`). - `KAFSCALE_PROXY_ADVERTISED_HOST` – Address Kafka clients should connect to. - `KAFSCALE_PROXY_ADVERTISED_PORT` – Advertised port (default `9092`). - `KAFSCALE_PROXY_ETCD_ENDPOINTS` – Etcd endpoints for metadata snapshots. - `KAFSCALE_PROXY_ETCD_USERNAME`, `KAFSCALE_PROXY_ETCD_PASSWORD` – Etcd auth for proxy. - `KAFSCALE_PROXY_BACKENDS` – Optional comma-separated broker list (`host:port`) for backend routing. +- `KAFSCALE_PROXY_BACKEND_CACHE_TTL_SEC` – Seconds to cache backend metadata before refreshing from etcd (default `60`). +- `KAFSCALE_PROXY_BACKEND_BACKOFF_MS` – Backoff between backend connection retries (default `500`). +- `KAFSCALE_PROXY_BACKEND_RETRIES` – Backend connection retry count (default `6`). +- `KAFSCALE_PROXY_LFS_ENABLED` – Enable the LFS HTTP API on the unified proxy (default `false`). ### Console @@ -390,33 +397,28 @@ Cost-optimized (accepts a larger loss window after crash): - `KAFSCALE_CONSOLE_BROKER_METRICS_URL` – Broker Prometheus endpoint. - `KAFSCALE_UI_USERNAME`, `KAFSCALE_UI_PASSWORD` – Console login credentials. -## External Broker Access +## External Kafka Access By default, brokers advertise the in-cluster service DNS name. That works for -clients running inside Kubernetes, but external clients must connect to a -reachable address. Configure both the broker Service exposure and the advertised -address so clients learn the external endpoint from metadata responses. +clients running inside Kubernetes. External clients need a reachable address in +Metadata responses. -Broker exposure settings (KafscaleCluster `spec.brokers`): -- `advertisedHost` / `advertisedPort` – Address Kafka clients should connect to. -- `service.type` – `ClusterIP`, `LoadBalancer`, or `NodePort`. -- `service.annotations` – Cloud provider LB annotations. -- `service.loadBalancerIP` / `service.loadBalancerSourceRanges` – Static IP + CIDR allowlist. -- `service.externalTrafficPolicy` – `Cluster` or `Local`. -- `service.kafkaNodePort` / `service.metricsNodePort` – Optional NodePort overrides. +**Recommended:** deploy the Kafka-aware proxy (`proxy.enabled=true`). When the +proxy is enabled, clients connect to the **proxy Service**, not individual broker +pods. The proxy answers Metadata/FindCoordinator with a single stable endpoint, +then forwards all other Kafka requests to brokers. This keeps clients connected as +broker pods scale or rotate. -### Kafka Proxy (external scaling) +**Optional:** expose brokers directly via `KafscaleCluster` `spec.brokers` Service +settings (LoadBalancer or NodePort). Use this only when you intentionally bypass +the proxy (traffic isolation, pinned producers, or legacy integrations). Broker +NodePort does not replace the proxy entrypoint in scaled deployments. -For external clients plus broker churn, deploy the Kafka-aware proxy. It answers -Metadata/FindCoordinator requests with a single stable endpoint (the proxy -service), then forwards all other Kafka requests to the brokers. This keeps -clients connected even as broker pods scale or rotate. The proxy is the -recommended external access layer and enables automated horizontal scaling -without exposing individual broker pods. +The Helm chart does **not** ship a Kafka Ingress resource. External TLS uses +LoadBalancer annotations or an external TCP gateway (see +[Proxy TLS via LoadBalancer](#proxy-tls-via-loadbalancer-recommended)). -Use the broker Service settings above when you intentionally expose dedicated -brokers (for example, isolating traffic or pinning producers to specific nodes). -That path is more controllable but requires explicit endpoint management. +### Kafka Proxy (recommended) Recommended settings: - Run 2+ proxy replicas behind a LoadBalancer service. @@ -442,7 +444,35 @@ helm upgrade --install kafscale deploy/helm/kafscale \ --set proxy.etcdEndpoints[0]=http://kafscale-etcd-client.kafscale.svc.cluster.local:2379 ``` -Helm chart docs: `deploy/helm/README.md`. +For local clusters (kind, minikube), pin the proxy NodePort so host port mappings +stay stable. Set `proxy.service.type=NodePort` and `proxy.service.nodePort` to a +value in `30000–32767`, then set `proxy.advertisedHost` to the node IP (or the +mapped host address). See `deploy/helm/kafscale/README.md` for details. + +Example (kind / local dev): + +```bash +helm upgrade --install kafscale deploy/helm/kafscale \ + --namespace kafscale --create-namespace \ + --set proxy.enabled=true \ + --set proxy.service.type=NodePort \ + --set proxy.service.nodePort=30092 \ + --set proxy.advertisedHost=127.0.0.1 \ + --set proxy.advertisedPort=30092 \ + --set proxy.etcdEndpoints[0]=http://kafscale-etcd-client.kafscale.svc.cluster.local:2379 +``` + +Helm chart docs: `deploy/helm/kafscale/README.md`. + +### Direct broker exposure (optional) + +Broker exposure settings (KafscaleCluster `spec.brokers`): +- `advertisedHost` / `advertisedPort` – Address Kafka clients should connect to. +- `service.type` – `ClusterIP`, `LoadBalancer`, or `NodePort`. +- `service.annotations` – Cloud provider LB annotations. +- `service.loadBalancerIP` / `service.loadBalancerSourceRanges` – Static IP + CIDR allowlist. +- `service.externalTrafficPolicy` – `Cluster` or `Local`. +- `service.kafkaNodePort` / `service.metricsNodePort` – Optional NodePort overrides. Example (GKE/AWS/Azure load balancer): diff --git a/docs/quickstart.md b/docs/quickstart.md index c44e94e6..128626a7 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -156,10 +156,11 @@ kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic orders --from-b ``` Note: Kafka clients that default to idempotent producers or transactions must disable them explicitly. -External clients: configure `spec.brokers.advertisedHost` / `advertisedPort` and -`spec.brokers.service` in your `KafscaleCluster` so Kafka clients learn a -reachable endpoint. See `docs/operations.md` and `deploy/helm/README.md` for -examples. +External clients: enable the Kafka proxy (`proxy.enabled=true`) and set +`proxy.advertisedHost` / `proxy.advertisedPort` so clients learn a stable +endpoint. For local clusters, pin `proxy.service.nodePort` (see +`docs/operations.md`). Direct broker exposure via `spec.brokers.service` is +optional when you intentionally bypass the proxy. ## Next Steps