A quiet journal about running small infrastructure well.
Kubernetes kubectl drain politely evicts pods — but for StatefulSets that hold open gRPC streams, politeness isn't enough. The trick was pre-stopping the stream readers with a preStop hook that calls into the controller's drain endpoint, then letting the standard eviction run. Zero dropped connections across 18 drains.
Running Loki in filesystem mode (no S3, no BoltDB index store) sounds like a toy, but on three nodes with ~40 GB of logs/week it has been fine for a year. Compaction runs hourly, the index-period is 24h, and the only operational task is pruning the cache directory once a month.
Needed to scrape a service that only exposes metrics over a private UNIX socket. Twenty lines of Go: open the socket, pipe response headers and body through, publish on :9100. The nice part was wiring it into systemd with socket activation so the proxy only runs when something actually scrapes.
Every quarter I pick one random Postgres database and restore it end-to-end into a scratch VM from the latest WAL-G backup. The last drill caught a missing pg_dumpall --globals-only step — roles and grants weren't being captured. Fixed and documented.
Edge devices run NixOS (declarative, immutable, perfect for far-away kiosks). The main cluster runs Debian with Ansible. The glue is a shared Prometheus setup and the same systemd-journald remote push config. Having two provisioning styles in one head sounds annoying — in practice they just stay out of each other's way.