Skip to content

feat: add Loki log aggregation with Alloy journal forwarding#390

Merged
stackptr merged 8 commits intomainfrom
feat/loki-log-aggregation
Mar 20, 2026
Merged

feat: add Loki log aggregation with Alloy journal forwarding#390
stackptr merged 8 commits intomainfrom
feat/loki-log-aggregation

Conversation

@stackptr
Copy link
Copy Markdown
Owner

@stackptr stackptr commented Mar 20, 2026

Summary

  • Removes unused Cockpit service from glyph
  • Adds Grafana Loki on glyph for centralized log storage (filesystem backend, TSDB index, 30-day retention)
  • Deploys Grafana Alloy on all NixOS hosts (glyph, spore, zeta) to forward systemd journal entries to Loki over Tailscale
  • Provisions Loki as a datasource in Grafana
  • Adds a Log Explorer dashboard with host filtering, text search, log volume histogram, and error/warning counters

Deploy order

  1. glyph first — Loki must be running before Alloy agents on remote hosts can push
  2. spore and zeta in any order after glyph

Test plan

  • nix-flake build nixosConfigurations.glyph.config.system.build.toplevel
  • nix-flake build nixosConfigurations.spore.config.system.build.toplevel
  • nix-flake build nixosConfigurations.zeta.config.system.build.toplevel
  • Deploy to glyph, verify Loki is listening on port 3100
  • Deploy to spore/zeta, verify Alloy service is running and shipping logs
  • Open Grafana → Log Explorer dashboard, query {host="glyph"} to confirm logs are ingested

🤖 Generated with Claude Code

stackptr and others added 8 commits March 20, 2026 10:13
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Configures Loki on glyph with filesystem storage, TSDB index,
and 30-day retention. Listens on port 3100 for log ingestion
from Alloy agents on all hosts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Configures Grafana Alloy to read systemd journal entries and
forward them to the local Loki instance with host=glyph label.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Configures Grafana Alloy on spore to forward systemd journal
entries to Loki on glyph over Tailscale.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Configures Grafana Alloy on zeta to forward systemd journal
entries to Loki on glyph over Tailscale.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds Loki on glyph as a datasource so logs from all hosts
are queryable via the Grafana Explore view.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Provisioned dashboard with host selector, text search, log volume
histogram, error/warning counters, and a full log stream panel.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
`{_}` is a destructuring pattern that rejects extra specialArgs
like `inputs`. Use `_:` which accepts any argument set without
destructuring, satisfying both statix and the module system.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@stackptr stackptr merged commit 6dcd84e into main Mar 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant