Issues to flesh out and open / Draft list of issues to detail and formalize

[simonwacker]

• mention that cloud-user needs sudo privileges and add test command
• monitor SSHGuard with monit service
• keep certbot files in volume
• when NGINX response fails put into answer which one did so. At the moment this can be seen through the HTTP Header X-Proxy-Layer which either has the value outer or inner.
• when the backend returns for example a 404 not found response, NGINX catches it and returns its own 404 with a custom message. Thereby the original response with its information is lost, which is unfortunate, in particular for requests to /api/resources/... because it returns detailed information about why (which part of) the resource was not found.
• regularly clean-up left-over images of past deployments: just keep the last n that are not older than x months.

The issues below are general DevOps issues that may also belong to the metabase/database/api infrastructure sections in metabase issues to flesh out, the database issues to flesh out, or the api issues to flesh out.

• improve hand-made content-addressable storage used in the database-project's database.mk backup/restore and in the machine-project's maintenance.mk backup/prune-backups
• as first post-deployment step, run Smoke Tests (in particular, availability tests), infrastructure tests (can the application connect to external services), security tests (TLS handshake, open ports, CORS and security headers), critical user journey tests (authentication, frontend loading), performance & resource usage tests (latency and response times, resource consumption) --- some of that is covered by Monit
• docker ps shows port mappings. It does neither show all internally open ports nor NGINX upstreams. These would be necessary to get a full data-flow picture.
• protect "Basic Auth" used for logs and metrics against brute force attacks. Or better: Only make logs and metrics available through an SSH tunnel to reduce the attack surface.
• fix MX records of our domains to be able to receive emails and to send emails to/from ...@buildingenvelopedata.org and ...@solarbuildingenvelopes.com
• consider using traefik instead of NGINX. traefik works seamlessly with Docker
• improve GNU Make self documentation: https://www.khoi.io/posts/self-documented-makefile/
• integrate "GraphQL Inspector GitHub App" as Git action to track GraphQL schema changes: https://the-guild.dev/graphql/inspector/docs/products/action
• migration fails (at least during deployment) when the PostgreSQL version changed and the container with the previous version is still running and a migration attempt is made with that version
• the pruning done by make down with the filter does not remove anonymous volumes (make prune does so)
• README: mention that during initial deployment "make build" on server or "./forge.mk" somewhere else can both be used
• Makefiles: tell in the error message how to recover when the targets dotenv or symlink fail
• vector: consumes 50% memory when the docker compose service logs is not available. Reduce! In general: Optimize it for production usage as detailed in https://vector.dev/docs/setup/going-to-prod/
• document .env variable dependencies like HTTPS_PORT in machine and database
• README: mention deploy.mk history/restore/state
• docker: use hardened images: https://hub.docker.com/hardened-images/catalog/dhi/nginx
• in NGINX security policy settings do neither generally allow JavaScript eval nor inline for /graphql/ nor /openapi/docs/. Instead use a nonce or a hash value
• postgres: backup database with https://github.com/pgbackrest/pgbackrest
• store "deployment_host" in .env for development to by used by forge.mk
• change SSH port
• check nginx config: sudo nginx -t -c ./path
• security: fierewall pfSense: https://www.pfsense.org/ --- see also https://docs.netgate.com/pfsense/en/latest/
• wait for health with streamed logs: Stream logs during --wait docker/compose#9122 (comment)
• nginx upstream: add active health checks
• mimic cron for testing: env -i HOME="$HOME" USER="$USER" PATH="/usr/bin:/bin" SHELL="/bin/sh" setsid ./maintenance.mk backup </dev/null >/tmp/cron_debug.log 2>&1
• consider using crontab dashboard (web UI): https://crontab.guru/dashboard.html ... https://cronitor.io/guides/cron-jobs ... https://crontab.guru/
  already setup for database
  /etc/cronitor/cronitor.json
  sudo cronitor configure --dash-username YOUR_USERNAME --dash-password YOUR_PASSWORD
  sudo cronitor configure --users cloud,root
  sudo cronitor configure --allowed-ips 172.0.0.1
  sudo cronitor update
  sudo cronitor dash --port 10200
• consider using NGINX Proxy Manager and/or NGINX UI
• track the NGINX virtual host traffic status with https://github.com/vozlt/nginx-module-vts
• consider using Biome for formatting and linting
• More automation e.g. for installing ansible and apt packages with make.
• Clean up old Docker images from local storage with retention controls: Cleanup https://github.com/iloveitaly/docker-image-cleanup#docker-compose Unregistry https://github.com/psviderski/unregistry
• Map certbot log files to stdout and thereby to docker logs

---

Similar lists exist for the metabase and the database projects.