backup-operator

A Kubernetes-native backup operator in Go for PostgreSQL, MySQL, MariaDB, MongoDB, and Redis, with public-key encryption (age), multi-destination fan-out (SFTP + S3-compatible), semantic dump analysis, and Prometheus-driven alerting.

The contract: label a Secret, get a backup. No CRDs to install, no extra resources to learn. The operator watches Secrets in its namespace, materialises a CronJob per labelled source, and Kubernetes does the running.

Built mostly with AI. This project is an experiment: how far can you take a complete, production-shaped system — Kubernetes operator, three binaries, encryption, multi-destination fan-out, dashboard UI, restore verification, Helm chart, alerts, docs — with different AI coding assistants doing most of the typing? The architecture decisions, code review, and direction are human; the implementation, refactors, and large parts of the documentation (this file included) were written collaboratively with several AI agents and then audited. Treat the design choices in CLAUDE.md §18 as the human-curated record of what was actually decided.

Why this exists

K8up, Stash, and Velero solve adjacent problems but none of them satisfy all three of:

Discovery via labelled Secrets, not CRDs. Labelling a Secret is the entire user contract. No CRD to install, no API surface to learn, no version skew to worry about.
Semantic dump analysis. Alerts fire on dump content — table disappeared, row-count collapsed, schema fingerprint changed — not just on job exit codes. A backup that "succeeds" with an empty dump is a silent disaster everywhere else; here it pages you.
Multi-destination fan-out as first-class. One dump streams to N storage backends in parallel, mixed protocols (SFTP + S3 + …). Failure of one destination doesn't fail the run.

If you don't need all three, you have simpler choices.

How it works

   user labels a Secret               ┌──────────────────────────────────────────┐
            │                         │ Kubernetes API                           │
            ▼                         │                                          │
   ┌──────────────────┐  watch        │   Source Secret  ──┐                     │
   │ Operator pod     ├───────────────┤                     │ OwnerReference     │
   │ (backup-operator)│  reconcile    │                     ▼                    │
   └──────────────────┘               │   batch/v1.CronJob ──tick──▶ Job pod     │
                                      │                                  │       │
                                      └──────────────────────────────────┼───────┘
                                                                         │
                              dump → gzip → age encrypt → temp file ◀────┘
                                                  │
                                       fan-out, parallel uploads
                                                  ▼
                                ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
                                │   AWS S3     │  │  MinIO/R2    │  │ Hetzner SFTP │
                                └──────────────┘  └──────────────┘  └──────────────┘
                                                  │
                                  before next run: read previous meta.json,
                                  diff stats, write new meta + analyzer report
                                  (alerts fire from Prometheus rules)

                   ┌──────────────────────────────────────────────┐
                   │ Operator's machine (offline)                 │
                   │   age private key  ──▶  backup-restore CLI  │
                   └──────────────────────────────────────────────┘

Three binaries, one image:

Binary	Where it runs	Job
`backup-operator`	Operator Deployment	Reconciles `Source` Secret → managed `CronJob`. Optionally hosts the read-only Dashboard UI.
`backup-worker`	CronJob-spawned Job pod	One-shot: dump → encrypt → fan-out → retention. Exits 0 / 1.
`backup-restore`	Operator's laptop	Lists, downloads, and decrypts artifacts. The only place the age private key ever lives.

Quick Start

# 1. Generate an age key pair OFFLINE on your machine.
age-keygen -o ~/age.key
# Two lines: the public recipient (age1qx...) and the private identity
# (AGE-SECRET-KEY-1...). Keep this file safe — it's the only way to
# decrypt your backups.

# 2. Install the operator with the public key as a Helm value.
#    From OCI registry (recommended):
helm install backup-operator oci://ghcr.io/behrangalavi/charts/backup-operator \
  -n backup --create-namespace \
  --set agePublicKeys="age1qx...your-recipient-here"

#    Or from local chart (development):
# helm install backup-operator ./charts/backup-operator \
#   -n backup --create-namespace \
#   --set agePublicKeys="age1qx...your-recipient-here"

# 3. Label a database Secret as a backup source.
kubectl -n backup apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: prod-users-db
  labels:
    backup.mogenius.io/role: source
    backup.mogenius.io/db-type: postgres
  annotations:
    backup.mogenius.io/name: "prod-users"
    backup.mogenius.io/schedule: "0 2 * * *"
type: Opaque
stringData:
  host: postgres.production.svc.cluster.local
  port: "5432"
  database: users
  username: backup
  password: super-secret
EOF

# 4. Label a destination Secret (S3 example).
kubectl -n backup apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: prod-s3
  labels:
    backup.mogenius.io/role: destination
    backup.mogenius.io/storage-type: s3
  annotations:
    backup.mogenius.io/name: "prod-s3"
    backup.mogenius.io/path-prefix: "backups/prod"
type: Opaque
stringData:
  bucket: my-backups
  access-key-id: AKIA...
  secret-access-key: ...
  region: eu-central-1
EOF

# 5. Confirm the CronJob was reconciled.
kubectl -n backup get cronjobs
# NAME                       SCHEDULE      ...
# backup-prod-users-db       0 2 * * *

# 6. Trigger a manual run instead of waiting for the schedule.
kubectl -n backup create job --from=cronjob/backup-prod-users-db manual-$(date +%s)

# 7. Restore (run from your laptop, with the offline private key).
backup-restore --storage-secret prod-s3 -n backup --target prod-users \
  --age-key ~/age.key --decompress | psql -h localhost prod_clone

Helm Installation & Distribution

The chart is published as an OCI artifact to GitHub Container Registry on every tagged release.

Install from OCI registry

helm install backup-operator oci://ghcr.io/behrangalavi/charts/backup-operator \
  -n backup --create-namespace \
  --set agePublicKeys="age1qx...your-recipient"

Upgrade

helm upgrade backup-operator oci://ghcr.io/behrangalavi/charts/backup-operator \
  -n backup --reuse-values

Install from source

git clone https://github.com/behrangalavi/backup-operator.git
helm install backup-operator ./backup-operator/charts/backup-operator \
  -n backup --create-namespace \
  --set agePublicKeys="age1qx...your-recipient"

Key Helm values

Value	Default	Description
`agePublicKeys`	(required)	Newline-separated age public keys for encryption
`config.defaultSchedule`	`0 2 * * *`	Default cron schedule for new sources
`config.runTimeoutSeconds`	`3600`	Max seconds per backup run
`config.defaultRetentionDays`	`30`	Days to keep backups (0 = forever)
`config.defaultMinKeep`	`3`	Minimum backups to keep regardless of age
`ui.enabled`	`false`	Enable the management UI on port 8081
`workerResources.limits.cpu`	`2000m`	CPU limit for worker pods
`workerResources.limits.memory`	`2Gi`	Memory limit for worker pods
`networkPolicy.enabled`	`false`	Restrict operator egress to known ports
`image.digest`	(empty)	Pin image by SHA256 digest for supply-chain security

See charts/backup-operator/values.yaml for the full list.

Defining a Backup Target (Source)

A Source is any Secret with backup.mogenius.io/role=source in the operator's watch namespace. The operator parses the Secret's labels and annotations, materialises a batch/v1.CronJob, and mounts an OwnerReference so deleting the Secret cascades to the CronJob.

Required labels

Label	Value
`backup.mogenius.io/role`	`source`
`backup.mogenius.io/db-type`	`postgres` \| `mysql` \| `mariadb` \| `mongo` \| `redis`

Required `data` keys

Key	Required	Notes
`host`	yes	Reachable hostname from the worker pod
`port`	no	Defaults: 5432 (pg), 3306 (mysql/mariadb), 27017 (mongo), 6379 (redis)
`database`	yes for pg/mysql/mariadb; optional for mongo/redis	Mongo: omit to back up all non-system databases. Redis: optional DB index `0`–`15` — narrows stats only; the RDB dump is always full-instance.
`username`	yes for all except `redis`	Redis pre-6 uses password-only AUTH; ACL usernames came in 6.0 and are optional
`password`	yes

Annotations

Annotation	Default	Effect
`backup.mogenius.io/name`	Secret name	Logical target name. Used in metrics, object paths, CronJob name.
`backup.mogenius.io/schedule`	`0 2 * * *` (chart default)	Cron expression. Standard Linux/Vixie syntax.
`backup.mogenius.io/analyzer-enabled`	`true`	Set to `false` if the user lacks `pg_stat_*` access. Disables stats collection and dump analysis for this target.
`backup.mogenius.io/destinations`	unset	CSV allow-list of destination names. Empty = fan out to all destinations in the namespace.
`backup.mogenius.io/retention-days`	`30` (chart default)	Delete dumps older than N days. `0` = keep forever.
`backup.mogenius.io/min-keep`	`3` (chart default)	Safety floor: never delete below this many newest dumps.
`backup.mogenius.io/extra-<key>`	none	Surfaced into `dumper.Config.Extra[key]` for db-specific options (e.g. `extra-sslmode=require`, `extra-authSource=admin`). For MySQL/MariaDB, `extra-max-allowed-packet` overrides the mysqldump client packet ceiling (default `1G`); raise it if a wide row aborts the dump with `Error 2026: TLS/SSL error: unexpected eof`.

A typo on a feature-flag annotation (analyzer-enabled: tru) silently falls back to the default — backups must keep running even if a flag is misspelled.

Examples

PostgreSQL:

apiVersion: v1
kind: Secret
metadata:
  name: orders-db
  labels:
    backup.mogenius.io/role: source
    backup.mogenius.io/db-type: postgres
  annotations:
    backup.mogenius.io/name: "orders"
    backup.mogenius.io/schedule: "*/30 * * * *"
    backup.mogenius.io/extra-sslmode: "require"
type: Opaque
stringData:
  host: postgres.orders.svc.cluster.local
  database: orders
  username: backup
  password: ...

MySQL:

apiVersion: v1
kind: Secret
metadata:
  name: legacy-mysql
  labels:
    backup.mogenius.io/role: source
    backup.mogenius.io/db-type: mysql
  annotations:
    backup.mogenius.io/name: "legacy"
    backup.mogenius.io/schedule: "0 3 * * *"
type: Opaque
stringData:
  host: mysql.legacy.svc.cluster.local
  database: app
  username: backup
  password: ...

MongoDB:

apiVersion: v1
kind: Secret
metadata:
  name: events-mongo
  labels:
    backup.mogenius.io/role: source
    backup.mogenius.io/db-type: mongo
  annotations:
    backup.mogenius.io/name: "events"
    backup.mogenius.io/schedule: "0 4 * * *"
    backup.mogenius.io/extra-authSource: "admin"
type: Opaque
stringData:
  host: mongo.events.svc.cluster.local
  username: backup
  password: ...

MariaDB (uses the MySQL wire protocol — same mysqldump tool):

apiVersion: v1
kind: Secret
metadata:
  name: cms-mariadb
  labels:
    backup.mogenius.io/role: source
    backup.mogenius.io/db-type: mariadb
  annotations:
    backup.mogenius.io/name: "cms"
type: Opaque
stringData:
  host: mariadb.cms.svc.cluster.local
  database: cms
  username: backup
  password: ...

Redis (RDB snapshot via redis-cli --rdb; full-instance, all DB indexes):

apiVersion: v1
kind: Secret
metadata:
  name: sessions-redis
  labels:
    backup.mogenius.io/role: source
    backup.mogenius.io/db-type: redis
  annotations:
    backup.mogenius.io/name: "sessions"
type: Opaque
stringData:
  host: redis.sessions.svc.cluster.local
  # username is optional — only needed for Redis 6+ ACL users
  password: ...

Defining a Destination

A Destination is any Secret with backup.mogenius.io/role=destination. Destinations are discovered at run time by each worker — there is no managed object for them.

Required label

Label	Value
`backup.mogenius.io/role`	`destination`
`backup.mogenius.io/storage-type`	`s3` \| `sftp` \| `hetzner-sftp`

Annotations

Annotation	Effect
`backup.mogenius.io/name`	Logical destination name. Matched against source's `destinations` allow-list. Defaults to Secret name.
`backup.mogenius.io/path-prefix`	Prepended to every object path. Useful for separating clusters/environments inside a shared bucket.

S3-compatible (`storage-type: s3`)

Works with AWS S3, MinIO, Hetzner Object Storage, Cloudflare R2, Backblaze B2, Wasabi, and anything else speaking the S3 API.

Key	Required	Notes
`bucket`	yes	Must already exist; the operator does not create buckets.
`access-key-id`	yes
`secret-access-key`	yes
`region`	no	Defaults to `us-east-1`; non-AWS providers usually ignore this.
`endpoint`	no	Required for non-AWS (e.g. `https://s3.eu-central-1.amazonaws.com` for AWS implicit, `https://gateway.eu1.storjshare.io` for Storj, etc.).
`path-style`	no	`"true"` for MinIO and others that require path-style addressing.

SFTP (`storage-type: sftp` or `hetzner-sftp`)

Key	Required	Notes
`host`	yes
`port`	no	Defaults to 22; Hetzner Storage Box uses 23.
`username`	yes
`ssh-private-key`	yes	PEM-encoded.
`known-hosts`	recommended	Output of `ssh-keyscan host`. Use `[host]:port` for non-22 ports. Without it the worker logs a loud `INSECURE` warning and uses `InsecureIgnoreHostKey`.

Examples

AWS S3:

apiVersion: v1
kind: Secret
metadata:
  name: aws-prod
  labels:
    backup.mogenius.io/role: destination
    backup.mogenius.io/storage-type: s3
  annotations:
    backup.mogenius.io/name: "aws-prod"
    backup.mogenius.io/path-prefix: "cluster-prod"
type: Opaque
stringData:
  bucket: my-backups
  access-key-id: AKIA...
  secret-access-key: ...
  region: eu-central-1

MinIO (in-cluster):

apiVersion: v1
kind: Secret
metadata:
  name: minio
  labels:
    backup.mogenius.io/role: destination
    backup.mogenius.io/storage-type: s3
  annotations:
    backup.mogenius.io/name: "minio"
type: Opaque
stringData:
  bucket: backups
  access-key-id: minioadmin
  secret-access-key: minioadmin
  endpoint: http://minio.backup.svc.cluster.local:9000
  path-style: "true"

Hetzner Storage Box (SFTP, port 23):

apiVersion: v1
kind: Secret
metadata:
  name: hetzner-sb
  labels:
    backup.mogenius.io/role: destination
    backup.mogenius.io/storage-type: hetzner-sftp
  annotations:
    backup.mogenius.io/name: "hetzner"
    backup.mogenius.io/path-prefix: "/cluster-prod"
type: Opaque
stringData:
  host: u123456.your-storagebox.de
  port: "23"
  username: u123456
  ssh-private-key: |-
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----
  known-hosts: |-
    [u123456.your-storagebox.de]:23 ssh-ed25519 AAAA...

The Dashboard UI

The operator ships a full management UI — a single-page application (SPA) with CRUD operations, live updates, and a settings wizard. No build step, no external dependencies.

Enable it

helm upgrade backup-operator oci://ghcr.io/behrangalavi/charts/backup-operator \
  -n backup --reuse-values --set ui.enabled=true

This adds a second container port (default 8081) and a Service port. The chart never creates an Ingress — bring your own.

Access locally

kubectl -n backup port-forward svc/backup-operator 8081:8081
# Browser: http://localhost:8081

What you get

Dashboard (#/): overview with stats cards (source count, healthy/failed, running jobs), target table with status badges, manual trigger button per target.
Sources (#/sources): card grid of all backup sources. Create, edit, and delete database backup sources via forms. Supports PostgreSQL, MySQL, MariaDB, MongoDB, and Redis with all configuration options.
Destinations (#/destinations): manage storage destinations (SFTP, S3). Create, edit, and delete with full field support. Sensitive fields (passwords, SSH keys) are masked in API responses.
Jobs (#/jobs): running and recent backup jobs with status and timing. For currently-running jobs, the Duration column shows a live progress bar driven by the median of past successful runs (last 10) for that target. The bar caps at 99 % until the run actually completes; if the run is overdue, the bar turns orange with a "länger als üblich" hint. Estimate-free fallback when no past runs are available yet.
Target detail (#/target/<name>): full run history table — timestamps, sizes, SHA256 checksums, schema status, anomaly counts, and download buttons per run. Failed runs surface phase + full error message inline.
Alerts (#/alerts): currently-firing backup alerts with severity counters and a sidebar pill counter. Pulls from Prometheus when configured (alerts.prometheusURL), otherwise re-evaluates the same conditions locally — see Surfacing alerts in the operator UI.
Settings (#/settings): configuration wizard (see Settings Wizard below).
Live updates: Server-Sent Events (SSE) push changes to all connected browsers in real time — no polling, no page refresh. Events are routed by page so editing a destination no longer re-renders the Audit log; bursts are coalesced into a single render via a 200 ms debounce.
Languages: EN / DE / FR via a sidebar language picker; choice is stored in localStorage and falls back to navigator.language. Add a language by dropping <code>.json into src/ui/static/i18n/ and registering the code in app.js — no build step.
Downloads: .age (encrypted dump, pass-through) and .json (analyzer metadata).

REST API

Method	Endpoint	Description
`GET`	`/api/targets`	List all backup sources with latest run status
`GET`	`/api/targets/{name}/runs`	Run history for one target
`GET`	`/api/sources/{name}`	Get source configuration
`POST`	`/api/sources`	Create a new source Secret
`PUT`	`/api/sources/{name}`	Update source configuration
`DELETE`	`/api/sources/{name}`	Delete source (verifies role label)
`GET`	`/api/destinations`	List all destinations
`POST`	`/api/destinations`	Create a new destination Secret
`GET`	`/api/destinations/{name}`	Get destination configuration
`PUT`	`/api/destinations/{name}`	Update destination
`DELETE`	`/api/destinations/{name}`	Delete destination (verifies role label)
`POST`	`/api/trigger/{target}`	Trigger a manual backup run
`GET`	`/api/jobs`	List running/recent jobs
`GET`	`/api/settings`	Get current operator settings
`PUT`	`/api/settings`	Update operator settings
`GET`	`/api/settings/export`	Download settings as `values.yaml`
`GET`	`/api/events`	SSE stream for live updates
`GET`	`/api/alerts`	Currently-firing backup alerts (Prometheus or local fallback)
`GET`	`/api/alerts/status`	Connectivity check for Prometheus + Alertmanager
`POST`	`/api/alerts/test`	Send a self-resolving test alert via Alertmanager v2 API

Security model

Role-verified CRUD. All Secret operations (GET, UPDATE, DELETE) verify the target Secret carries the expected backup.mogenius.io/role label before proceeding. Non-backup Secrets cannot be accessed or deleted through the API.
No built-in auth. Cluster-internal use is the assumed default. To expose externally, put oauth2-proxy, an Ingress with basic-auth annotation, or your platform's SSO in front of the Service (see CLAUDE.md §3.1 for examples).
Sensitive data masked. Passwords, SSH keys, and access keys are returned as *** in API responses. They are only written, never read back.
Pass-through downloads. The operator streams encrypted bytes from the destination to the client without decrypting. The age private key never enters the cluster.

Settings Wizard

The Settings Wizard (#/settings) provides a guided 4-step form to configure the operator at runtime — no helm upgrade needed.

Steps

Step	What you configure
1. Schedule & Timeout	Default cron schedule, run timeout
2. Retention Policy	Retention days, minimum keep, temp directory, temp dir size
3. Worker Resources	CPU/Memory limits and requests for backup worker pods
4. Review & Apply	Summary of all settings, save button

How it works

Settings are stored in a Kubernetes ConfigMap ({release}-settings), created automatically when ui.enabled=true. The wizard reads and writes this ConfigMap via the API.

Helm values.yaml → ConfigMap (install-time defaults)
                         ↕
                    UI Settings Wizard (runtime overrides)
                         ↓
                    Export values.yaml → Git → helm upgrade (GitOps)

Export for GitOps

Click "Export values.yaml" to download the current settings as a Helm-compatible values file. Commit it to your repo and apply with:

helm upgrade backup-operator oci://ghcr.io/behrangalavi/charts/backup-operator \
  -n backup -f values.yaml

This gives you the best of both worlds: interactive UI for quick tuning and declarative GitOps for controlled rollouts.

Restore

The backup-restore CLI runs on your machine, not in the cluster. It reads the destination Secret via your kubeconfig, downloads the chosen artifact, and decrypts with the offline private key.

Build the binary

just build-restore
# produces dist/native/backup-restore

List available dumps

backup-restore --storage-secret aws-prod -n backup --target prod-users --list

# 20260428T020005Z  prod-users/2026/04/28/dump-20260428T020005Z.sql.gz.age
# 20260427T020003Z  prod-users/2026/04/27/dump-20260427T020003Z.sql.gz.age
# ...

Restore the latest

# To stdout, decompressed, ready to pipe into psql/mysql/mongorestore:
backup-restore --storage-secret aws-prod -n backup --target prod-users \
  --age-key ~/age.key --decompress | psql -h localhost new_users_db

Restore a specific timestamp

backup-restore --storage-secret aws-prod -n backup --target prod-users \
  --age-key ~/age.key --timestamp 20260428T020005Z -o dump.sql.gz
gunzip dump.sql.gz   # or pipe `--decompress` directly

Flags

Flag	Required	Default
`--storage-secret`	yes	—
`--target`	yes	—
`--namespace` (`-n`)	no	`default`
`--age-key`	yes (for download)	—
`--timestamp`	no	latest
`--list`	no	false
`--decompress`	no	false
`-o`	no	`-` (stdout)

Restore Verification

A backup that hasn't been restored is not a backup.

The operator can prove on a configurable cadence that an encrypted dump can actually be decrypted, parsed, and (optionally) restored — without ever sending the age private key into the cluster. The worker generates a one-shot X25519 keypair in process memory at the moment it decides to verify, encrypts the dump for both the long-lived disaster-recovery recipient and the ephemeral one (age supports multi-recipient natively, ~200 bytes header overhead per recipient), runs the verifier in-process, then the pod terminates and the ephemeral private half is gone. The DR key is unaffected and remains the only path that can decrypt the artifact for a real recovery.

Modes

Set on a source via backup.mogenius.io/restore-verification-mode. Off by default. A typo on the annotation falls back to off (parsed via parseRestoreVerificationMode in internal/secrets/parser.go).

Mode	What it does	RBAC needed	Cost
`off`	No verification. (default)	none	—
`stream-validate`	In-process: decrypt with the ephemeral identity → gunzip → engine-aware parser. SQL engines (postgres / mysql / mariadb): re-runs `dumper.RowCounter` against the plaintext stream, header sanity-check against the engine banner, then total-rows comparison vs pre-dump stats with the same 99 % tolerance the dump-time verifier uses. Mongo: asserts the BSON archive magic `0x8199e26d` and drains the stream so a corrupt gzip layer fails loudly. Redis: asserts the `REDIS` magic + 4 ASCII version digits, drains the body. Catches bit rot, broken encryption, corrupt gzip, truncated dumps, "looks like a dump but isn't" garbage.	none	seconds; no extra pods
`schema-only`	Spawns an ephemeral DB pod, restores DDL only. SQL engines stream-filter the `COPY ... FROM stdin` body and `INSERT INTO` lines on the fly so the engine literally never sees the data — proves schema restores cleanly without paying data-restore cost on a 50 GiB DB. Mongo / Redis: `schema-only` is a label, not a meaningfully cheaper restore — the BSON archive and RDB binary aren't decomposable that way. Pick `stream-validate` instead on those engines if cost matters.	`pods: create/get/list/watch/delete`, `pods/status: get`, `pods/log: get` in worker namespace	a small DB pod for ~30 s
`sample`	Spawns an ephemeral DB pod, restores schema + a sample of data. Catches data-encoding issues a schema-only run would miss. From the engine's perspective `sample` is currently equivalent to `full`; pre-filtering hooks are reserved for a future iteration.	same as above	small pod, modest disk
`full`	Spawns an ephemeral DB pod, restores the entire dump, runs smoke queries (per-table `SELECT count(*)` for SQL engines; ping + auth roundtrip for Redis whose RDB-restore is deferred to a follow-up iteration). Highest fidelity.	same as above	full-size pod and emptyDir; node needs the headroom

Annotations

Annotation	Default	Effect
`backup.mogenius.io/restore-verification-mode`	`off`	One of the modes above. Unknown values fall back to `off`.
`backup.mogenius.io/restore-verification-interval`	`168h` (weekly)	Minimum gap between verifier runs (Go duration: `30m`, `48h`, `7d` …). State-driven: the worker reads `latestMeta.restoreVerification.completedAt` and skips when `now - completedAt < interval`. Cron drift doesn't matter; manual runs (`kubectl create job --from=cronjob/...`) verify whenever overdue. The very first run after enabling verification falls through to a "first verification" path that runs immediately so operators see signal without waiting one full interval.
`backup.mogenius.io/verification-image`	per-DB-type default	Container image for the verifier pod. Pin to the source DB's exact major version when restore semantics depend on it (charset defaults, function signatures, dump-format compatibility). Phase-2 modes only (`schema-only` / `sample` / `full`) — `stream-validate` ignores it. Per-engine defaults: `postgres:16-alpine`, `mysql:8.0`, `mariadb:11`, `mongo:7`, `redis:7-alpine`.
`backup.mogenius.io/verification-volume-size`	`1Gi` (schema-only), `5Gi` (sample), `50Gi` (full)	`emptyDir.sizeLimit` for the verifier pod's data volume. Accepts `K`/`M`/`G`/`T` (decimal) and `Ki`/`Mi`/`Gi`/`Ti` (binary) suffixes. Override when one source's restore needs more headroom — at scale, the node's ephemeral storage is a real budget. Phase-2 modes only.

Enabling Phase 2 in the chart

Phase-2 modes spawn ephemeral DB pods and need a wider RBAC grant on the worker ServiceAccount. Off by default; flip on with:

# values.yaml
restoreVerification:
  enableEphemeralPodSpawn: true

The chart then grants the worker SA pods: create/get/list/watch/delete, pods/status: get, and pods/log: get in its own namespace. Without this flag, attempting to set a Phase-2 mode produces a Verdict=Skipped result with the RBAC error captured in meta.json.

The spawned pod is restricted-PSA compliant out of the box: runAsNonRoot=true, RunAsUser=999, readOnlyRootFilesystem=true, capabilities.drop=ALL, seccompProfile=RuntimeDefault. It carries an OwnerReference → worker pod, so the moment the worker exits, K8s GC cascades the verifier pod away regardless of completion state.

When does verification actually run?

ShouldVerify (in verifier/verifier.go) skips a run for any of:

mode is off or absent
No prior run for this target — verifier needs a preStats baseline that the regular DumpVerification has already established. The very first backup of a new source establishes that baseline; verification starts from the second run.
Interval not elapsed since the last completedAt.

It runs (and emits restore_verification_passed + _last_timestamp) when:

Mode is set and the interval has elapsed, OR
Mode is set but no RestoreVerification block exists on the latest meta yet (the "first verification" path fires immediately).

Alerts

Two rules ship with the chart:

Alert	Condition	Severity
`BackupRestoreVerificationFailed`	`max by (target, mode) (backup_operator_restore_verification_passed) == 0` for 5 m	critical
`BackupRestoreVerificationStale`	`time() - max by (target, mode) (backup_operator_restore_verification_last_timestamp_seconds) > 86400 * 14` for 1 h	warning

A failed or skipped verification does not fail the backup run itself — verification is observability, not a gate. The dump still uploads, the artifact remains decryptable with the DR key, and downstream integrations are unaffected. The alerts exist so an operator can investigate before the next disaster-recovery drill.

What the UI shows

Target Detail (#/target/<name>): every run carries a verification badge — match / mismatch / skipped / not configured — plus the verifier mode, completed-at, and ephemeral-recipient fingerprint. The fingerprint is per-run; seeing the same one twice in a row would itself be a bug, since the keypair regenerates on every verifier-run.
Source form / Settings → "Restore Verification": four fields mirroring the annotations above (mode, interval, verifier image, volume size). Phase-2-only fields are disabled in the form when mode is off or stream-validate.

Documentation Portal

The operator can serve CLAUDE.md, README.md, and a generated tech-stack page (built from go.mod) on a separate port. Off by default. The portal is read-only — it has no Kubernetes client, no Secret access, no ability to mutate state. That's what justifies exposing it more loosely than the management UI.

Enable it

# values.yaml
docs:
  enabled: true
  port: 8083  # default

Access locally

kubectl -n backup port-forward svc/backup-operator 8083:8083
# Browser: http://localhost:8083

What you get

/ — README (this file), rendered with goldmark.
/claude — CLAUDE.md (operator reference + architectural decisions).
/tech-stack — direct dependencies from go.mod, with version + purpose.
In-page search — Ctrl+K opens a dropdown with snippet highlighting across all pages. Client-side only; no server-side index, so the portal works on read-only mounts.

Why a separate port

The management UI mutates Secrets, ConfigMaps, and Jobs; it must be SSO-gated in production. The docs portal renders Markdown files. Splitting them onto separate ports lets cluster admins write two distinct ingress rules — public docs, gated UI — without code changes.

Alerting & Monitoring

Metrics

The operator pod exposes Prometheus metrics on :8080/metrics. Run-level signals come from the operator's MetricsRefresher controller, which periodically reads the latest *.meta.json sidecar from each destination and writes the resulting state into the gauges below. This is why the run-state metrics are gauges, not counters — worker pods are too short-lived for Prometheus to scrape, so the operator reconstructs the state from storage instead.

Metric	Type	Labels	Meaning
`backup_operator_dump_size_bytes`	Gauge	`target`	Encrypted size of the most recent successful dump
`backup_operator_dump_size_change_ratio`	Gauge	`target`	current/previous encrypted size; `<0.5` = suspicious shrinkage
`backup_operator_table_count`	Gauge	`target`	Tables/collections in the most recent successful run
`backup_operator_table_row_count`	Gauge	`target`, `table`	Per-table row count (estimate) at the most recent run
`backup_operator_schema_changed`	Gauge	`target`	1 if schema hash differs from previous run, 0 otherwise
`backup_operator_charset_changed`	Gauge	`target`	1 if database character set or collation differs from previous run; warns about silent multibyte truncation at restore time
`backup_operator_schema_last_change_timestamp_seconds`	Gauge	`target`	Unix ts of the most recent run where the schema fingerprint actually changed; carried forward across unchanged runs
`backup_operator_last_run_anomalies`	Gauge	`target`	Analyzer anomaly count in the most recent run
`backup_operator_last_run_status`	Gauge	`target`	1 = most recent run wrote a usable artifact, 0 = failure
`backup_operator_last_success_timestamp_seconds`	Gauge	`target`, `destination`	Unix ts of last successful upload to that destination
`backup_operator_destination_failed`	Gauge	`target`, `destination`	1 if the destination is unreadable / last upload failed
`backup_operator_storage_scrub_passed`	Gauge	`target`, `destination`	1 if the most recent scrub matched the recorded SHA256, 0 if mismatch. Only present when `STORAGE_SCRUB_ENABLED=true`.
`backup_operator_storage_scrub_last_check_timestamp_seconds`	Gauge	`target`, `destination`	Unix ts of the most recent scrub attempt
`backup_operator_storage_scrub_failed_total`	Counter	`target`, `destination`	Cumulative scrub failures (mismatch or unreachable). Operator-side, scraped normally.
`backup_operator_restore_verification_passed`	Gauge	`target`, `mode`	1 if the most recent restore-verifier run produced verdict `match`, 0 for `mismatch`/`skipped`. Absent until at least one verifier has run.
`backup_operator_restore_verification_last_timestamp_seconds`	Gauge	`target`, `mode`	Unix ts of the most recent restore-verifier completion. Drives the stale alert.
`backup_operator_retention_deleted_total`	Counter	`target`, `destination`, `kind`	Worker-only — see caveat below
`backup_operator_retention_failed_total`	Counter	`target`, `destination`	Worker-only — see caveat below
`backup_operator_dump_duration_seconds`	Histogram	`target`, `db_type`	Worker-only — see caveat below
`backup_operator_upload_duration_seconds`	Histogram	`target`, `destination`, `storage_type`	Worker-only — see caveat below
`backup_operator_run_duration_seconds`	Histogram	`target`, `db_type`	Worker-only — see caveat below
`backup_operator_restore_verification_duration_seconds`	Histogram	`target`, `mode`	Worker-only — see caveat below

Caveat — "worker-only" metrics: the histograms and retention_* counters are observed inside the worker pod, but worker pods finish in seconds and Prometheus's scrape interval (≥15s) cannot catch them. They live in the codebase for future use (e.g. an aggregator that pushes to Pushgateway or rebuilds them from meta.json). For timing alerts today, rely on Job duration via kube-state-metrics.

ServiceMonitor

Enabled by default when the Prometheus Operator CRDs are installed. The chart's templates render the ServiceMonitor only if monitoring.coreos.com/v1 is present.

# values.yaml
serviceMonitor:
  enabled: true
  interval: 30s
  scrapeTimeout: 10s

# Top-level convenience: kube-prometheus-stack's Prometheus selects
# ServiceMonitors and PrometheusRules via `release: <name>`. The chart
# applies this label to BOTH out of the box. Set to "" to opt out and
# use your own labels via {serviceMonitor,prometheusRule}.labels instead.
prometheusReleaseLabel: "kube-prometheus-stack"

Without this label, kube-prometheus-stack's Prometheus renders the CRDs but never loads them — the most common reason backup alerts silently don't fire. If your stack uses a different release name, override accordingly.

If you don't run Prometheus Operator, set serviceMonitor.enabled=false and configure your scrape job manually pointing at :8080/metrics.

Default Alert Rules

The chart ships a PrometheusRule with eleven default alerts. They're rendered only when monitoring.coreos.com/v1 is present, and they live under prometheusRule.rules in values.yaml so you can override or add your own at install time.

Alert	Expression (simplified)	For	Severity
`BackupOverdue`	`time() - max by (target) (last_success_timestamp_seconds) > 86400 * 1.5`	10m	warning
`BackupDestinationFailing`	`max by (target, destination) (destination_failed) == 1`	15m	warning
`BackupDumpSizeCollapsed`	`dump_size_change_ratio < 0.5`	5m	critical
`BackupSchemaChanged`	`schema_changed == 1`	1m	info
`BackupCharsetChanged`	`charset_changed == 1`	1m	warning
`BackupStorageCorrupted`	`max by (target, destination) (storage_scrub_passed) == 0`	1m	critical
`BackupAnomaliesAppearing`	`last_run_anomalies > 0`	5m	warning
`BackupLastRunFailed`	`last_run_status == 0`	5m	warning
`BackupSucceeded`	`time() - max by (target) (last_success_timestamp_seconds) < 120`	—	info
`BackupRestoreVerificationFailed`	`max by (target, mode) (restore_verification_passed) == 0`	5m	critical
`BackupRestoreVerificationStale`	`time() - max by (target, mode) (restore_verification_last_timestamp_seconds) > 86400 * 14`	1h	warning

BackupSucceeded is a heartbeat-style positive signal — every successful run produces one firing + one resolved notification (Alertmanager has send_resolved: true). With a frequent cron this is intentionally noisy; silence it if you only want failure alerts.

The semantic alerts (BackupDumpSizeCollapsed, BackupSchemaChanged, BackupAnomaliesAppearing) are this project's main differentiator. They alert on what's actually in the dump, not on whether the job exited cleanly. A backup that succeeds with empty tables silently in another tool will page you here.

Wiring to Alertmanager / Slack / PagerDuty

The chart deliberately ships no notifier. Routing, deduplication, silencing, and rendering belong in Alertmanager. Configure your Alertmanager once for all your alerts (cluster-wide), and the rules above will route there automatically via the Prometheus Operator's standard pipeline.

Minimal Alertmanager route example for these alerts:

route:
  receiver: 'default'
  routes:
    - matchers: [ alertname=~"Backup.*", severity="critical" ]
      receiver: 'pagerduty'
    - matchers: [ alertname=~"Backup.*" ]
      receiver: 'slack-backups'

receivers:
  - name: 'default'
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: ${PAGERDUTY_KEY}
  - name: 'slack-backups'
    slack_configs:
      - api_url: ${SLACK_WEBHOOK}
        channel: '#alerts-backups'
        title: '{{ .CommonAnnotations.summary }}'
        text: |-
          target: {{ .CommonLabels.target }}
          {{- if .CommonLabels.destination }}
          destination: {{ .CommonLabels.destination }}
          {{- end }}

Customising the alerts

Override prometheusRule.rules at install time to drop, modify, or add rules:

helm upgrade backup-operator ./charts/backup-operator -n backup --reuse-values \
  --values custom-rules.yaml

# custom-rules.yaml
prometheusRule:
  rules:
    - alert: BackupOverdue
      expr: time() - max by (target) (backup_operator_last_success_timestamp_seconds) > 7200  # 2h instead of 36h
      for: 5m
      labels: { severity: critical, team: data }
      annotations:
        summary: "Backup target {{ $labels.target }} overdue >2h"
    # ... add or omit other rules ...

Disabling alerts entirely

helm upgrade ... --set prometheusRule.enabled=false

Surfacing alerts in the operator UI

When the UI is enabled (ui.enabled=true), an Alerts tab in the sidebar shows currently-firing alerts with severity counters (critical/warning/info) and a sortable list. Two providers feed it, picked automatically:

Prometheus mode (preferred) — queries <prometheusURL>/api/v1/alerts filtered by alertname=~"^Backup.*" so the UI mirrors what Alertmanager will route. Honours each rule's for: duration.
Local heuristic (fallback) — when prometheusURL is empty or unreachable, the operator re-evaluates the same six conditions on its own gathered metric registry. Useful before kube-prometheus-stack is wired up; does not honour for: debounce, so it fires immediately on threshold crossing.

The UI marks each alert with a source badge (prometheus vs local) so you can see which view you're looking at.

# values.yaml — defaults assume kube-prometheus-stack lives in the `alert` namespace
alerts:
  prometheusURL: "http://prometheus-operated.alert.svc.cluster.local:9090"
  alertmanagerURL: "http://alertmanager-operated.alert.svc.cluster.local:9093"

alertmanagerURL is currently used for the "Open in Alertmanager" link plus a connectivity-status indicator and a one-click "Send test alert" button. The operator never receives notifications from Alertmanager — wiring stays one-way through Prometheus.

Set both to "" to disable the integration entirely; the UI then falls back to local-heuristic mode silently.

Encryption Model

Backups are encrypted with age — modern, audited, public-key-only encryption.

How it works

Operator's machine (offline):
  age-keygen -o age.key       →   ~/age.key
                                  ├── public:  age1qx...   (RECIPIENT)
                                  └── private: AGE-SECRET-KEY-1...  (NEVER LEAVES YOUR LAPTOP)

Cluster (online):
  Helm install --set agePublicKeys="age1qx..."
   └── creates Secret backup-operator-age with key AGE_PUBLIC_KEYS
        └── mounted into every worker pod via secretKeyRef

Worker pod runtime:
  pg_dump | gzip | age encrypt -r <recipient>  →  dump.sql.gz.age

Restore (offline):
  age -d -i ~/age.key dump.sql.gz.age | gunzip | psql ...

Properties enforced by the code

The operator refuses to start without AGE_PUBLIC_KEYS. There is no plaintext-backup code path.
The age recipient list is newline-separated → key rotation works by adding the new public key to the list before retiring the old; both can decrypt during the transition.
The restore CLI accepts the same multi-key format → matches age-keygen -o's output.
Storage backends never see plaintext bytes — they receive *.sql.gz.age ciphertext only.
The dashboard UI streams encrypted bytes pass-through; the operator never decrypts.

Properties you must enforce operationally

Keep the private key offline. Putting it in the cluster collapses the entire security model.
Back up the private key separately. Paper, hardware token, password manager, anything but the cluster. Losing it means losing every backup it can decrypt.
Rotate by adding, then retiring. Add a new recipient to agePublicKeys, run a few backups (so they're encrypted to both keys), then remove the old recipient. New recipients can decrypt going forward; old recipients still work for older artifacts.
For multi-region recovery, distribute multiple public keys to the cluster and keep each region's private key in that region's safe.

CI/CD

Two GitHub Actions workflows automate testing and releasing. See .github/workflows/ for the full YAML.

CI (`ci.yaml`)

Runs on every pull request to main:

Job	Steps
`test`	`go build ./...` → `go test ./...` → `go vet ./...`
`helm-lint`	`helm lint charts/backup-operator --set agePublicKeys="age1test"`

Release (`release.yaml`)

Triggered automatically on every push to main via Semantic Release. The version is determined from commit messages — no manual tagging required.

Run go test to gate the release.
Analyze commits since the last release to determine the next version.
Create a GitHub Release with auto-generated release notes.
Build a multi-arch Docker image (linux/amd64 + linux/arm64).
Push the image to ghcr.io/behrangalavi/backup-operator:<version> and :latest.
Package the Helm chart with the matching version.
Push the chart to oci://ghcr.io/behrangalavi/charts/backup-operator.

Commit conventions

Releases are driven by Conventional Commits:

Prefix	Effect	Example
`fix:`	Patch release (1.0.x)	`fix: correct retention day calculation`
`feat:`	Minor release (1.x.0)	`feat: add S3 destination support`
`feat!:` or `BREAKING CHANGE:`	Major release (x.0.0)	`feat!: rename source annotations`
`docs:`, `ci:`, `chore:`	No release	`docs: update README`

Merging a PR with fix: or feat: commits triggers an automatic release. No manual git tag needed.

After the workflow completes, users can install with:

helm install backup-operator oci://ghcr.io/behrangalavi/charts/backup-operator \
  -n backup --create-namespace \
  --set agePublicKeys="age1qx...your-recipient"

Local Development

Prerequisites

Go 1.26+ (for native binaries)
just for the task runner
Docker Desktop with Kubernetes enabled
kubectl, helm
Optionally: age CLI (brew install age)

Docker Desktop K8s caveat

Modern Docker Desktop versions implement Kubernetes via kind under the hood (docker desktop kubernetes status reports Mode: kind). The kind node has its own containerd, separate from the Docker daemon's image store.

To make local builds visible to the cluster:

Open Docker Desktop → Settings → General.
Check "Use containerd for pulling and storing images".
Click Apply & Restart (the cluster gets reset — workloads will be wiped).

After this, docker info shows Storage Driver: overlayfs (containerd snapshotter). docker build outputs end with an unpacking to docker.io/library/... line confirming the image landed in the shared store.

Use imagePullPolicy: IfNotPresent, not Never. Docker Desktop's desktop-containerd-registry-mirror bridges pulls from the kind cluster to the daemon's containerd — but only when kubelet actually attempts a pull. With Never, kubelet never tries, and the mirror cannot help.

One-command test stack

# Copy the env template; defaults match the test stack.
cp .env.example .env

# Build the operator/worker image into the (now-shared) containerd store.
just build-image

# Generate an age key pair offline + apply the public key as a Secret.
just gen-age-key

# Apply the test stack: namespace, worker SA/RBAC, in-cluster Postgres
# (with a seed table), in-cluster MinIO (with bucket init), source +
# destination Secrets.
just test-up

# Install the operator via Helm.
helm install backup-operator ./charts/backup-operator -n backup \
  --set image.repository=backup-operator \
  --set image.tag=dev \
  --set image.pullPolicy=IfNotPresent \
  --set agePublicKeys="$(grep 'public key:' ~/age-backup-test.key | cut -d' ' -f4)" \
  --set serviceMonitor.enabled=false \
  --set prometheusRule.enabled=false \
  --set ui.enabled=true

# Trigger a manual run instead of waiting 5 minutes.
just test-trigger

# Browse the dashboard.
kubectl -n backup port-forward svc/backup-operator 8081:8081
# → http://localhost:8081

# Tear it all down.
just test-down
helm uninstall backup-operator -n backup

Iterating on code

When you change the operator/worker code, you have to rebuild and make sure the cluster pulls the new image. The kind cluster doesn't notice rebuilt-with-same-tag images. Two options:

Option A — unique tag per build (recommended):

TAG="dev-$(date +%s)"
docker build -t backup-operator:$TAG .
helm upgrade backup-operator ./charts/backup-operator -n backup --reuse-values \
  --set image.tag=$TAG

Option B — same tag, force pod recreation:

just build-image
kubectl -n backup delete pod -l app.kubernetes.io/name=backup-operator
# Pod recreates and re-resolves the image from the daemon's containerd
# (which now has the new content under the same `dev` tag).

Run the operator on your laptop instead of as a Pod

The operator is a normal controller-runtime app — ctrl.GetConfigOrDie() falls back to your kubeconfig when there's no in-cluster token. You can run it on your machine while watching a real cluster:

just build
just run    # reads .env for required envs

Worker pods still run in the cluster; only the operator-pod is replaced by your local process. Useful for fast reconciler iteration without touching the cluster's image cache.

Tests, lint, vet

just check        # lint + unit tests
just test-unit
just golangci-lint

Troubleshooting

`ErrImageNeverPull` on Docker Desktop

The kind cluster's containerd doesn't have the locally built image. See Docker Desktop K8s caveat — enable the containerd image store toggle and use imagePullPolicy: IfNotPresent.

Operator pod logs `events is forbidden`

Cosmetic — controller-runtime tries to record leader-election events but the chart's RBAC doesn't include events (intentional, to keep the role minimal). Backups run normally; only the leader-election event is suppressed.

Worker pod `ErrImagePull` for `postgres:17-alpine` etc.

Transient Docker Hub network issues. Delete the pod (kubectl delete pod ...) to clear the kubelet backoff and retry. If persistent, check the cluster's network egress.

`BackupDumpSizeCollapsed` firing on first run

False positive. The first run has no previous to compare against; dump_size_change_ratio defaults to 0. The chart's default expression checks < 0.5 — a fresh install will trigger it once. After the second run completes, the metric reflects real change ratios. Workaround: silence in Alertmanager for an hour after deployment, or filter dump_size_change_ratio < 0.5 and dump_size_change_ratio > 0 if you accept the slight loss of sensitivity.

`INSECURE` warnings in worker logs (SFTP destinations)

The destination Secret is missing known-hosts. Add the output of ssh-keyscan -p <port> <host> to the Secret's data.known-hosts field. Without it the worker accepts any host key — a downgrade in security but not a backup failure.

`analyzer-enabled: false` not detected

Annotation typo (e.g. analyzer-enable). The parser is forgiving and falls back to the default rather than rejecting the Secret, so look closely at the annotation name. Use kubectl get secret -o yaml to inspect.

Name		Name	Last commit message	Last commit date
Latest commit History 315 Commits
.github/workflows		.github/workflows
charts/backup-operator		charts/backup-operator
src		src
.env.example		.env.example
.gitignore		.gitignore
.releaserc.json		.releaserc.json
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
Justfile		Justfile
README.md		README.md
SECURITY.md		SECURITY.md
renovate.json		renovate.json

Folders and files

Latest commit

History

Repository files navigation

backup-operator

Table of Contents

Why this exists

How it works

Quick Start

Helm Installation & Distribution

Install from OCI registry

Upgrade

Install from source

Key Helm values

Defining a Backup Target (Source)

Required labels

Required data keys

Annotations

Examples

Defining a Destination

Required label

Annotations

S3-compatible (storage-type: s3)

SFTP (storage-type: sftp or hetzner-sftp)

Examples

The Dashboard UI

Enable it

Access locally

What you get

REST API

Security model

Settings Wizard

Steps

How it works

Export for GitOps

Restore

Build the binary

List available dumps

Restore the latest

Restore a specific timestamp

Flags

Restore Verification

Modes

Annotations

Enabling Phase 2 in the chart

When does verification actually run?

Alerts

What the UI shows

Documentation Portal

Enable it

Access locally

What you get

Why a separate port

Alerting & Monitoring

Metrics

ServiceMonitor

Default Alert Rules

Wiring to Alertmanager / Slack / PagerDuty

Customising the alerts

Disabling alerts entirely

Surfacing alerts in the operator UI

Encryption Model

How it works

Properties enforced by the code

Properties you must enforce operationally

CI/CD

CI (ci.yaml)

Release (release.yaml)

Commit conventions

Local Development

Prerequisites

Docker Desktop K8s caveat

One-command test stack

Iterating on code

Run the operator on your laptop instead of as a Pod

Tests, lint, vet

Troubleshooting

ErrImageNeverPull on Docker Desktop

Operator pod logs events is forbidden

Worker pod ErrImagePull for postgres:17-alpine etc.

Required `data` keys

S3-compatible (`storage-type: s3`)

SFTP (`storage-type: sftp` or `hetzner-sftp`)

CI (`ci.yaml`)

Release (`release.yaml`)

`ErrImageNeverPull` on Docker Desktop

Operator pod logs `events is forbidden`

Worker pod `ErrImagePull` for `postgres:17-alpine` etc.

`BackupDumpSizeCollapsed` firing on first run

`INSECURE` warnings in worker logs (SFTP destinations)

`analyzer-enabled: false` not detected