Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# RustFS Kubernetes Operator

A Kubernetes operator for [RustFS](https://rustfs.com/) object storage, written in Rust with [kube-rs](https://github.com/kube-rs/kube). It reconciles a **`Tenant` custom resource** (`rustfs.com/v1alpha1`), validates referenced credential and KMS Secrets, and applies RBAC, Services, and StatefulSets so RustFS runs as an erasure-coded cluster inside your cluster.
A Kubernetes operator for [RustFS](https://rustfs.com/) object storage, written in Rust with [kube-rs](https://github.com/kube-rs/kube). It reconciles a **`Tenant` custom resource** (`rustfs.com/v1alpha1`), validates referenced credential and KMS Secrets, and applies RBAC, Services, and StatefulSets so RustFS runs inside your cluster, from single-node single-disk development tenants to erasure-coded distributed clusters.

**Status:** v0.1.0 pre-release — under active development.

Expand Down
12 changes: 5 additions & 7 deletions console-web/app/(dashboard)/tenants/new/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ const DEFAULT_RUSTFS_IMAGE = "rustfs/rustfs:latest"

const defaultPool: CreatePoolRequest = {
name: "pool-0",
servers: 2,
volumes_per_server: 2,
servers: 1,
volumes_per_server: 1,
storage_size: "10Gi",
storage_class: "",
}
Expand All @@ -48,9 +48,9 @@ spec:
name: rustfs-creds
pools:
- name: pool-0
servers: 2
servers: 1
persistence:
volumesPerServer: 2
volumesPerServer: 1
volumeClaimTemplate:
accessModes:
- ReadWriteOnce
Expand Down Expand Up @@ -474,9 +474,7 @@ export default function TenantCreatePage() {
<CardHeader className="flex flex-row items-center justify-between">
<div>
<CardTitle className="text-base">{t("Pools")}</CardTitle>
<CardDescription>
{t("At least one pool with 4+ volumes (e.g. 2 servers × 2 volumes).")}
</CardDescription>
<CardDescription>{t("RustFS validates storage layout when the tenant starts.")}</CardDescription>
</div>
<Button type="button" variant="outline" size="sm" onClick={addPool}>
{t("Add Pool")}
Expand Down
2 changes: 1 addition & 1 deletion console-web/i18n/locales/en-US.json
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@
"Optional": "Optional",
"Image": "Image",
"Credentials Secret": "Credentials Secret",
"At least one pool with 4+ volumes (e.g. 2 servers × 2 volumes).": "At least one pool with 4+ volumes (e.g. 2 servers × 2 volumes).",
"RustFS validates storage layout when the tenant starts.": "RustFS validates storage layout when the tenant starts.",
"Add Pool": "Add Pool",
"Pool": "Pool",
"Remove": "Remove",
Expand Down
2 changes: 1 addition & 1 deletion console-web/i18n/locales/zh-CN.json
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@
"Optional": "可选",
"Image": "镜像",
"Credentials Secret": "凭证 Secret",
"At least one pool with 4+ volumes (e.g. 2 servers × 2 volumes).": "至少一个池且总卷数不少于 4(例如 2 服务器 × 2 卷)。",
"RustFS validates storage layout when the tenant starts.": "RustFS 会在 Tenant 启动时校验存储布局。",
"Add Pool": "添加存储池",
"Pool": "存储池",
"Remove": "移除",
Expand Down
7 changes: 0 additions & 7 deletions deploy/rustfs-operator/crds/tenant-crd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1356,13 +1356,6 @@ spec:
- persistence
- servers
type: object
x-kubernetes-validations:
- messageExpression: '"pool " + self.name + " must have at least 4 total volumes (servers × volumesPerServer)"'
reason: FieldValueInvalid
rule: self.servers * self.persistence.volumesPerServer >= 4
- messageExpression: '"pool " + self.name + " with 3 servers must have at least 6 volumes in total"'
reason: FieldValueInvalid
rule: self.servers != 3 || self.servers * self.persistence.volumesPerServer >= 6
maxItems: 32
minItems: 1
type: array
Expand Down
7 changes: 0 additions & 7 deletions deploy/rustfs-operator/crds/tenant.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1356,13 +1356,6 @@ spec:
- persistence
- servers
type: object
x-kubernetes-validations:
- messageExpression: '"pool " + self.name + " must have at least 4 total volumes (servers × volumesPerServer)"'
reason: FieldValueInvalid
rule: self.servers * self.persistence.volumesPerServer >= 4
- messageExpression: '"pool " + self.name + " with 3 servers must have at least 6 volumes in total"'
reason: FieldValueInvalid
rule: self.servers != 3 || self.servers * self.persistence.volumesPerServer >= 6
maxItems: 32
minItems: 1
type: array
Expand Down
11 changes: 7 additions & 4 deletions docs/operator-user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ spec:
- name: dev-pool
servers: 1
persistence:
volumesPerServer: 4
volumesPerServer: 1
```

Apply and verify:
Expand Down Expand Up @@ -251,14 +251,15 @@ Key fields:
| `resources` | Container resource requests and limits for the pool. |
| `priorityClassName` | Pool-level priority class override. |

Validation rules:
Operator admission checks:

- `servers * volumesPerServer >= 4`.
- For `servers: 3`, total volumes must be at least `6`.
- `servers` and `persistence.volumesPerServer` must be greater than `0`.
- Pool names must be unique.
- Pool peer DNS labels must fit Kubernetes DNS label limits.
- Existing pool `servers` and `volumesPerServer` cannot be changed in place.

The operator does not validate whether a RustFS storage layout, erasure set size, or storage class parity is supported. RustFS performs those checks when the Tenant workload starts.

Example:

```yaml
Expand Down Expand Up @@ -346,6 +347,8 @@ The operator reserves these environment variables and manages them automatically
- `RUSTFS_CONSOLE_ENABLE`
- TLS-related RustFS variables when Tenant TLS is enabled.

For a single-pool single-node single-disk Tenant, `RUSTFS_VOLUMES` is rendered as the local data path, for example `/data/rustfs0`. Multi-pool tenants and other layouts render peer DNS URLs through the Tenant headless Service and are validated by RustFS at runtime.

`podDeletionPolicyWhenNodeIsDown` accepts:

- `DoNothing`: do not delete pods automatically.
Expand Down
11 changes: 7 additions & 4 deletions docs/operator-user-guide.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ spec:
- name: dev-pool
servers: 1
persistence:
volumesPerServer: 4
volumesPerServer: 1
```

应用并检查:
Expand Down Expand Up @@ -253,14 +253,15 @@ Tenant 名称必须兼容 DNS-1035,且长度不超过 55 个字符,因为 Op
| `resources` | Pool 容器资源 request 和 limit。 |
| `priorityClassName` | Pool 级 PriorityClass 覆盖。 |

校验规则
Operator admission 检查

- `servers * volumesPerServer >= 4`。
- 当 `servers: 3` 时,总卷数至少为 `6`。
- `servers` 和 `persistence.volumesPerServer` 必须大于 `0`。
- Pool 名称必须唯一。
- Pool peer DNS label 必须满足 Kubernetes DNS label 长度限制。
- 已存在 pool 的 `servers` 和 `volumesPerServer` 不能原地修改。

Operator 不校验 RustFS 存储布局、erasure set 大小或 storage class parity 是否被支持。这些检查由 Tenant workload 启动后的 RustFS 自行完成。

示例:

```yaml
Expand Down Expand Up @@ -348,6 +349,8 @@ Operator 会自动管理以下环境变量:
- `RUSTFS_CONSOLE_ENABLE`
- 启用 TLS 时的 RustFS TLS 相关变量

对于单 pool 的单节点单盘 Tenant,`RUSTFS_VOLUMES` 会渲染为本地数据路径,例如 `/data/rustfs0`。多 pool Tenant 和其他布局仍会通过 Tenant headless Service 渲染 peer DNS URL,并由 RustFS 在运行时校验。

`podDeletionPolicyWhenNodeIsDown` 支持以下值:

- `DoNothing`:不自动删除 Pod。
Expand Down
36 changes: 13 additions & 23 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This directory contains example Tenant configurations for the RustFS Kubernetes

| Example | Use Case | Complexity | Storage | Best For |
|---------|----------|------------|---------|----------|
| [minimal-dev-tenant.yaml](./minimal-dev-tenant.yaml) | Development/Learning | ⭐ Simple | 40Gi | **Start here** if new |
| [minimal-dev-tenant.yaml](./minimal-dev-tenant.yaml) | Development/Learning | ⭐ Simple | 10Gi | **Start here** if new |
| [simple-tenant.yaml](./simple-tenant.yaml) | Documentation Reference | ⭐⭐ Moderate | Configurable | Learning all options |
| [secret-credentials-tenant.yaml](./secret-credentials-tenant.yaml) | Secret-based Credentials | ⭐ Simple | Configurable | **Production credential security** |
| [provisioning-tenant.yaml](./provisioning-tenant.yaml) | Policy/User/Bucket Provisioning | ⭐⭐ Moderate | 40Gi | Tenant bootstrap automation |
Expand Down Expand Up @@ -46,8 +46,8 @@ This directory contains example Tenant configurations for the RustFS Kubernetes
**Smallest valid configuration** for learning, testing, and local development.

**Configuration:**
- 1 server × 4 volumes = 4 total volumes (minimum valid)
- 40Gi total storage (4 × 10Gi default)
- 1 server × 1 volume = 1 total volume (single-node single-disk)
- 10Gi total storage (1 × 10Gi default)
- Debug logging enabled

**Use case:** Local development, learning the operator, quick testing.
Expand All @@ -64,7 +64,7 @@ kubectl apply -f examples/minimal-dev-tenant.yaml
Single-pool tenant with **comprehensive comments** explaining all available options.

**Features demonstrated:**
- Basic pool configuration with validation
- Basic pool configuration and admission checks
- Persistence settings with volume claim templates
- Custom labels and annotations for PVCs
- Environment variable configuration
Expand Down Expand Up @@ -369,7 +369,6 @@ spec:
servers: <number> # Must be > 0
persistence:
volumesPerServer: <number> # Must be > 0
# Validation: servers * volumesPerServer >= 4

# Optional fields
env: [...]
Expand All @@ -378,31 +377,22 @@ spec:
# ... see Tenant CRD for all fields
```

## Validation Rules
## Admission Checks

### Pool Requirements

- **Minimum total volumes**: `servers * volumesPerServer >= 4` (RustFS erasure coding requirement)
- **Three-server pools**: `servers * volumesPerServer >= 6` (stricter than the general minimum)
- **Server count**: Must be > 0
- **Volumes per server**: Must be > 0
- **Pool name**: Must not be empty

These rules are enforced by the Tenant CRD (CEL) and the operator console API.
The operator intentionally does not validate RustFS storage topology, erasure set sizing, or parity compatibility. Those checks are left to RustFS when the Tenant starts.

### Valid Examples
### Layout Examples

✅ `servers: 4, volumesPerServer: 1` → 4 total volumes
✅ `servers: 2, volumesPerServer: 2` → 4 total volumes
✅ `servers: 3, volumesPerServer: 2` → 6 total volumes (minimum for 3 servers)
✅ `servers: 4, volumesPerServer: 4` → 16 total volumes

### Invalid Examples

❌ `servers: 2, volumesPerServer: 1` → 2 total volumes (< 4)
❌ `servers: 1, volumesPerServer: 1` → 1 total volume (< 4)
❌ `servers: 3, volumesPerServer: 1` → 3 total volumes (< 6 for 3 servers)
❌ `servers: 0, volumesPerServer: 4` → Server count must be > 0
- `servers: 1, volumesPerServer: 1` → single-node single-disk style
- `servers: 2, volumesPerServer: 1` → passed through to RustFS
- `servers: 3, volumesPerServer: 1` → passed through to RustFS
- `servers: 4, volumesPerServer: 4` → distributed style

## Common Configurations

Expand All @@ -414,7 +404,7 @@ spec:
- name: dev
servers: 1
persistence:
volumesPerServer: 4 # 1 * 4 = 4 (minimum valid)
volumesPerServer: 1 # single-node single-disk
```

### Production (High Availability)
Expand Down Expand Up @@ -511,9 +501,9 @@ kubectl logs -n rustfs-system -l app=rustfs-operator
### Validation Errors

Common issues:
- Pool validation: Ensure `servers * volumesPerServer >= 4`
- Empty pool name
- Zero servers or volumes
- RustFS rejecting an unsupported storage layout after the Tenant starts

### Pods Not Starting

Expand Down
20 changes: 10 additions & 10 deletions examples/minimal-dev-tenant.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
#
# Resources created:
# - 1 StatefulSet with 1 replica
# - 4 PVCs (1 server × 4 volumes)
# - 1 PVC (1 server × 1 volume)
# - 3 Services (IO at 9000, Console at 9001, Headless)
# - RBAC resources (Role, ServiceAccount, RoleBinding)
#
# Total storage: 40Gi (4 volumes × 10Gi default)
# Total storage: 10Gi (1 volume × 10Gi default)

apiVersion: rustfs.com/v1alpha1
kind: Tenant
Expand All @@ -22,14 +22,14 @@ spec:
# Container image (optional - uses operator default if not specified)
image: rustfs/rustfs:latest

# Minimal pool configuration
# servers * volumesPerServer must be >= 4 (RustFS erasure coding requirement)
# This uses: 1 server × 4 volumes = 4 total volumes (minimum valid)
# Minimal pool configuration.
# A single-node single-disk Tenant is expressed as 1 server × 1 volume.
# Other storage layouts are passed through to RustFS for validation at startup.
pools:
- name: dev-pool
servers: 1 # Single server for development
persistence:
volumesPerServer: 4 # Minimum 4 volumes required
volumesPerServer: 1 # Single PVC; RustFS uses no erasure parity

# Optional: Reduce storage size for development
# volumeClaimTemplate:
Expand Down Expand Up @@ -61,7 +61,7 @@ spec:
# kubectl wait --for=condition=ready pod -l rustfs.tenant=dev-minimal --timeout=300s

# 4. Access S3 API (port 9000):
# kubectl port-forward svc/rustfs 9000:9000
# kubectl port-forward svc/dev-minimal-io 9000:9000
# # Test: aws --endpoint-url http://localhost:9000 s3 ls

# 5. Access Console UI (port 9001):
Expand All @@ -74,10 +74,10 @@ spec:
# -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="RUSTFS_VOLUMES")].value}'
#
# Output will be:
# http://dev-minimal-dev-pool-0.dev-minimal-hl.default.svc.cluster.local:9000/data/rustfs{0...3}
# /data/rustfs0
#
# This tells RustFS to use 4 volumes at paths:
# /data/rustfs0, /data/rustfs1, /data/rustfs2, /data/rustfs3
# This tells RustFS to use one local volume at:
# /data/rustfs0

# 7. Test with MinIO client:
# mc alias set devlocal http://localhost:9000 rustfsadmin rustfsadmin
Expand Down
10 changes: 5 additions & 5 deletions examples/simple-tenant.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,14 @@ spec:
podManagementPolicy: Parallel

# Pool configuration - at least one pool is required
# Each pool must have: servers * volumesPerServer >= 4
# RustFS validates whether the requested storage layout is supported at startup.
pools:
- name: primary
servers: 2

# Persistence configuration for storage (REQUIRED)
# Note: servers * volumesPerServer must be >= 4 for RustFS distributed storage
persistence:
# Number of volumes per server (REQUIRED, must satisfy servers * volumesPerServer >= 4)
# Number of volumes per server (REQUIRED)
volumesPerServer: 2

# Optional: Path where volumes will be mounted in the container (defaults to /data)
Expand Down Expand Up @@ -99,9 +98,10 @@ spec:
# Optional: Priority class - overrides tenant-level priority
# priorityClassName: high-priority

# Multi-Node Communication (Automatic)
# Volume Configuration (Automatic)
# The operator automatically generates the RUSTFS_VOLUMES environment variable
# for multi-node, multi-disk distributed storage communication.
# for the selected layout. Single-pool single-node single-disk tenants use a
# local path, while multi-pool and distributed layouts use peer DNS.
#
# For this configuration, the generated RUSTFS_VOLUMES will be:
# http://example-tenant-primary-{0...1}.example-tenant-hl.rustfs-system.svc.cluster.local:9000/data/rustfs{0...1}
Expand Down
2 changes: 1 addition & 1 deletion examples/spot-instance-tenant.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,7 @@ spec:
# Best Practices:

# 1. Maintain minimum on-demand capacity:
# - At least servers × volumesPerServer >= 4 on on-demand
# - Keep enough on-demand RustFS capacity for the layout you choose
# - Ensures cluster survives even if all spot instances interrupted

# 2. Use topology spread for spot pools:
Expand Down
2 changes: 1 addition & 1 deletion examples/tenant-4nodes.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# RustFS Tenant - 4-node distributed deployment (Kind multi-node + local storage)
# Architecture: 4 servers x 2 volumes = 8 drives (servers * volumesPerServer >= 4)
# Architecture: 4 servers x 2 volumes = 8 drives
# Requires: kind-rustfs-cluster (3 workers), StorageClass local-storage, 12 PVs
apiVersion: rustfs.com/v1alpha1
kind: Tenant
Expand Down
Loading
Loading