Skip to content

Security: ImranR98/Infra

Security

docs/security.md

Security

  • Secrets never touch git (/config/, compose.private.yaml, targets/*/compose/.env are gitignored); .secret-style files are 0600 host files. Authelia SSO (forward-auth + basic-auth, 2FA); CrowdSec (srv0: Helm chart with agent/LAPI/AppSec + per-service postoverflow whitelists; vps0: single container, bouncer key auto-registered from BOUNCER_KEY_TRAEFIK); geoblock allowlist (CA/CN/CU) using the public get.geojs.io API (fails closed); per-component NetworkPolicies plus baselines in the namespaces component (its kube-system policy explicitly allows 80/443/8000/8443 to Traefik).
  • Docker socket via wollomatic/socket-proxy: dockerproxy (read-only, Traefik/monitoring) and dockerproxy_priv (read-write, watchtower; only on pc/bigpc — cap_drop: ALL, read_only: true, mem_limit: 512M, user 65534:$DOCKER_GID).
  • Runtime security — Falco (srv0 + bigpc) consumes kernel events with the modern eBPF driver and evaluates the default syscall rules; events are JSON in Loki and metrics drive Grafana alerts. It runs with capabilities (BPF, SYS_RESOURCE, PERFMON, SYS_PTRACE) plus pod-level spc_t, reads /proc, /etc and /sys/kernel read-only, and uses the K3s containerd socket; the k8s-metacollector is disabled, so the sensor has no cluster-wide RBAC. Detection only — no Talon/auto-remediation, and no k8saudit plugin yet.
  • Known tradeoff — K3s HelmChart valuesContent (DB passwords, JWKS, OIDC secrets) is readable by anyone with get on helmcharts.helm.cattle.io; fine for single-user, but audit before granting namespace access. Where a chart exposes a secret ref it is used: authelia (path:), crowdsec (externalSecret), grafana (admin.existingSecret), headlamp (oidc.externalSecret), immich/nextcloud/plik (secretKeyRef in plain YAML). Where a chart has no secret-ref support but takes plain YAML, the component uses spec.valuesSecrets: values live in a namespaced Secret (e.g. targets/srv0/k3s-apps/templates/frigate.yaml, key values.yaml) listed in the CR, projected as a later -f values file so the merge is plain Helm deep-merge and renders identically. Done for frigate's env passwords (its env key only accepts plain strings — chart limitation). What can't be avoided: authelia's JWKS PEM has to be embedded as value: (the chart otherwise generates a fresh RANDOM key when CryptographicKey isn't inline). freshrss stays a root container (the official image hardcodes apache on :80 and its entrypoint runs as root — de-rooting needs a custom apache config); nextcloud stays root too (the official entrypoint writes /etc/apache2 as root even with APACHE_PORT set — uid 33 crashes on remoteip.conf removal; the PVC is already www-data-owned, so this is purely an entrypoint limitation).
  • SELinux (Fedora nodes) — Kubernetes assigns per-pod MCS categories, and files keep their creator's categories forever. Any pods that share a hostPath tree (file-pipelines/dscpln), plus the backup/restore pods, need pod-level seLinuxOptions.level: s0 — the container-level setting is ignored; anything that writes there (dscpln's git-tracked Excel ledger) also runs as MY_UID so new files stay imranr-owned and the ledger repo stays committable; dscpln commits every workbook write as D$CPLN <dscpln@localhost> with signing disabled; syncthing adds pod-level type: spc_t because its trees can carry foreign-labeled files (e.g. moved in from /tmp). privileged: true sidesteps enforcement, but newly written files still get labelled. For its filesystem collector to read /host/proc/1/mountinfo, node-exporter needs pod-level seLinuxOptions.type: spc_t, root, and CAP_SYS_PTRACE (the ptrace LSM hook rejects container_t; because the denial is dontaudited it surfaces only as EACCES). It runs on the pod network (not hostNetwork) so clustered Alloy can scrape it from either node; the allow-node-exporter-metrics-ingress NetworkPolicy admits alloy-metrics on :9100. Python/Node io_uring denials are audit spam with an epoll fallback — fix with PYTHON_IO_URING=0 / UV_USE_IO_URING=0, not SELinux changes. setroubleshootd pegging CPU means a denial backlog; fix the denials, don't mask.
  • NFS app data — the nfs-apps export uses no_root_squash (required for CSI subdir creation and Postgres chowns) and is reachable only from the LAN CIDR; the backup pod and restore pod run as root (PVC_POD_UID=0). The Immich Postgres image is forced onto NFS with IGNORE_DATABASE_FSTYPE=true. These are deliberate full-NFS tradeoffs; see docs/storage.md.
  • Static node labels — hostpath-main/external-exposed are set by k3s-server.sh/k3s-join.sh --hostpath-main; NFD/node-facts are gone. validate.sh warns if srv0 is missing them.

Accepted security tradeoffs

Resolved audit findings and deliberate choices — don't re-flag any of these without reading the referenced reasoning.

  • qBittorrent's adminadmin password is fine (targets/srv0/k3s-apps/templates/qbittorrent.yaml) — the UI is already gated by forwardauth-authelia + geoblock + crowdsec, so the password is convenience, not a boundary. Never propose changing it.
  • NetworkPolicies are enforced — K3s embeds the kube-router network-policy controller and enables it by default; nothing here passes --disable-network-policy (the scripts only set flannel-backend: wireguard-native). Flannel carrying the data path does not disable policy enforcement. Check kubectl -n kube-system get pods | grep -i router before disputing this.
  • SSH hardening lives outside this repo — sshd/fail2ban are configured manually per machine; don't suggest harden-ssh-style tooling. srv0's tunneled SSH is not publicly reachable: it is only reachable at 10.99.0.2:22 from vps0, so remote logins must pass vps0's pubkey-only sshd as a jump host (its effective config is hardened out-of-band via a sshd_config.d drop-in).
  • Unpinned supply-chain fetches are deliberate — the dracut clone in preboot.sh, the "latest" etcdctl download in update-node-ip, the releases/latest system-upgrade manifests, the TOFU get.k3s.io installer fetch (TLS only, no local hash), and get.docker.com in prereqs. Pinning each would force manual version bumps; don't propose it.
  • The s01-whitelist subnet regex on vps0 stays (targets/vps0/compose/files/crowdsec/postoverflows/s01-whitelist/internal.yaml) — uptime-kuma needs the exemption, its container IP is not fixed, and nothing better exists.
  • Authelia intentionally sits outside geoblock — the operator travels beyond the CA/CN/CU allowlist. Crowdsec guards the srv0 auth route; geoblock would lock the operator out.
  • Public by design: seerr (geoblock only — meant to be shareable, no SSO) and owncast's RTMP ingest on :1935 (protected only by a strong stream key set in the Owncast UI; Traefik can't gate it).
  • Hosts that intentionally skip SSO and/or geoblock — on srv0, gpt (Open WebUI), fmd, and rss (FreshRSS) are public behind geoblock + crowdsec only (they have their own auth, no Authelia); tv (Jellyfin) uses crowdsec + jellyfin-ratelimit + the app's own auth, no Authelia forward-auth and no geoblock. On vps0, plausible (crowdsec only) and ytdl (metube; crowdsec + Authelia one_factor) skip geoblock. Don't propose adding SSO or geoblock to any of them.
  • Home Assistant reaches the ConBee II through the generic-device-plugin, not privileged (targets/srv0/k3s-base/templates/generic-device-plugin.yaml + k3s-apps/templates/homeassistant.yaml). The device cgroup forbids unprivileged opens of /dev/ttyACM0, so HA asks for the infra.local/conbee resource. The plugin DaemonSet (squat/generic-device-plugin) scans each node's /dev — grants track the hardware rather than a node label — and publishes stable IDs derived from the device path; a node without the device advertises zero. HA also requires pod-level seLinuxOptions.type: spc_t (container_t is denied the mounted /run/dbus/system_bus_socket — Bluetooth integration via host bluez — and the device) and capabilities.add: [NET_ADMIN, NET_RAW] (habluetooth talks to the host adapter over direct HCI sockets; with hostNetwork off and auto-discovery unused, these are scoped to the pod netns). The device cgroup grant covers only the ConBee II. Don't propose bringing back privileged.
  • Jellyfin gets /dev/dri through the generic-device-plugin, not privileged (targets/srv0/k3s-apps/templates/jellyfin.yaml) — it requests infra.local/dri, runs as $MY_UID under container_t, and needs no spc_t (renderD128 is 0666 and container_t can open it). The device cgroup grant is scoped to the render node. frigate and immich use the same grant (container_t + s0, pinned to srv0 — frigate retains CAP_PERFMON but its dashboard GPU-stats graph fails because SELinux denies container_t perf_event_open). The device plugin itself is unprivileged (spc_t, no capabilities); the Task Runner server runs as root with spc_t and DAC_OVERRIDE/FOWNER/CHOWN plus SETUID/SETGID (it needs the app-data UIDs and the MY_UID-owned hostPath backup repo; user-data tasks drop to MY_UID via run_as); the removed legacy pvc-backup/restore pods were non-privileged (spc_t + DAC_OVERRIDE/FOWNER); ollama uses the infra.local/amd grant (kfd + dri on bigpc) with container_t + s0, and its NetworkPolicy admits only the open-webui and opencode pods on 11434; mosquitto runs as uid 1883. On TCP 2049, nfs-server ingress is allowlisted to the node CIDR (every share is nfsvers=4.2 and CSI mounts originate on nodes); the CIDR is hardcoded as 192.168.8.0/24 in nfs-server.yaml, so change it for a different LAN.
  • One public edge key covers all public hostnames — edge-public-tls lists the 24 public names exactly (not a wildcard), but they share one key, so a key compromise has wildcard-like blast radius. The private cert (files/traefik) is separate and never attached to the public-exposure listener. Don't propose per-host certificates without accepting the operational cost this replaced.
  • nfs-server must stay privileged (targets/srv0/k3s-base/templates/nfs-server.yaml) — it hosts a kernel NFS server (nfsd/rpc.mountd) inside the container, which privileged is genuinely required for. It's an internal base component; don't suggest de-privileging.
  • The AmneziaWG pod runs root + NET_ADMIN under spc_t (targets/srv0/k3s-base/templates/vpn.yaml) — creating/removing the host-netns ra-amneziawg0 interface and the iptables NAT/FORWARD rules needs it, and the official amneziavpn/amneziawg-go image runs the userspace implementation (no host kernel module, /dev/net/tun mounted) by choice. The DNS pod is unprivileged (UID 65532 + NET_BIND_SERVICE) and binds 10.100.0.1:53 only, so it is reachable solely through the tunnel. Only files/ra-amneziawg-start.sh touches host state (the keygen init container writes only the PVC, and the conn-notify sidecar only reads the UAPI socket and posts to ntfy).
  • Falco is a host-level sensor with elevated access (targets/srv0/k3s-base/templates/falco.yaml) — modern eBPF needs CAP_BPF/PERFMON/SYS_PTRACE/SYS_RESOURCE and SELinux spc_t (container_t is denied perf_event_open/BPF, as with node-exporter); it also mounts /proc, /etc, /sys/kernel read-only and /run/k3s/containerd for container metadata. That access is the point of the sensor; don't propose removing it without replacing the runtime-detection capability. The default rules are noisy on this stack (Task Runner, backup pods, opencode, kubectl exec -it, Home Assistant integration installs); tune with rule-level disables/exceptions rather than disabling whole tags. Event-rate alerts use a 30-minute warning recurrence gate, and the daemonset readiness alert is gated on Ready nodes, so bigpc being powered off does not page. The Contact K8S API Server From Container output is overridden to drop command=%proc.cmdline: command lines can carry bearer tokens (logtfy's API polling exposed a service-account token), and Loki retains events for 90 days.
  • vps0's one_factor Authelia rules are intentional for ytdl/ikom/sale-site (family and guests); the admin catch-all is the only two_factor rule (TOTP is Authelia's default second factor, so no default_second_factor_policy is needed).
  • srv0's PROXY-protocol trustedIPs are exactly the hub address (targets/srv0/k3s-base/templates/traefik.yaml): 10.99.0.1/32. vps0's Traefik connects to 10.99.0.2:8443 over the public-exposure tunnel; the hub container SNATs the proxy network's traffic to 10.99.0.1, so the flow carrying the PROXY v2 header has a single stable trusted source and no pod can forge it.
  • vps0's Cloudflare proxy setup is deliberate — plausible.imranr.dev is proxied with Bot Fight Mode on the imranr.dev zone; the zone SSL mode is full (not strict) because Traefik's TLS-ALPN renewal cannot complete behind the proxy, and Bot Fight Mode cannot be skipped on the Free plan — hence the obtainium_apps stats fetch bypasses Cloudflare (extra_hosts → host gateway) and the apps source accepts plain HTTP/unverified HTTPS. Third-level hostnames (apps.obtainium.imranr.dev) are not covered by Universal SSL and would need the paid Advanced Certificate Manager to be proxied, so they stay DNS-only.
  • CrowdSec's clientTrustedIPs is a bypass-whitelist for clients, not an XFF/proxy knob (the plugin README: "List of client IPs to trust, they will bypass any check from the bouncer or cache"). Proxy trust is forwardedHeadersTrustedIPs: 127.0.0.1/32 on srv0, and 127.0.0.1/32 plus the Cloudflare edge ranges on vps0 (required for CF-proxied hostnames); docker-network callers are exempted at the crowdsec layer by the s01-whitelist postoverflow, not clientTrustedIPs. Don't reintroduce it to "fix" internal traffic.
  • Agents must never read config/ or anything in it — those secrets are private by design; an audit verifies gitignore coverage and git history, never file contents.

See docs/decisions.md for the accepted duplication list (do not consolidate).

There aren't any published security advisories