ncm/operate/README.md

Operate

This module manages NixOS configurations for cluster nodes using Colmena.

Overview

The operate module:

• Defines NixOS system configurations for each cluster node
• Manages secrets with agenix-rekey (automated re-encryption on host key changes)
• Configures Nomad cluster (server + client mode)
• Sets up Tailscale VPN
• Manages firewall rules and networking
• Handles system-level services and packages

Structure

operate/
├── hosts/          # Per-host configurations
│   ├── ncm-0/     # First cluster node
│   ├── ncm-1/     # Second cluster node
│   └── ncm-2/     # Third cluster node
├── operate/       # Shared NixOS modules
└── templates/     # Templates for new hosts (used by ncm-prepare)

Host Configuration

Each host directory contains:

• default.nix - Main system configuration (includes hostPubkey for secret encryption)
• base-system.nix - Common system settings
• hardware-configuration.nix - Hardware-specific config
• nomad.nix - Nomad cluster configuration

Secrets are managed centrally:

• ../../secrets/master/ - Source secrets (encrypted with master key)
• ../../secrets/rekeyed/ - Per-host encrypted secrets (auto-generated by agenix-rekey)

Prerequisites

1. Infrastructure deployed via the deploy/ module
2. SSH access to all nodes
3. Age key for secret management (~/.ssh/id_rsa_agenix)

Quick Start

1. Generate Host Configurations

After deploying infrastructure, run from the project root:

ncm-prepare

This creates host directories and configures:

• SSH authorized keys
• Hostnames and networking
• Deployment targets
• Encrypted secrets (Tailscale, API tokens)

2. Deploy Configurations

Deploy to all nodes:

colmena apply --impure

Deploy to specific nodes:

colmena apply --on ncm-0 --impure
colmena apply --on ncm-1,ncm-2 --impure

3. Verify Deployment

Check node status:

colmena exec --on @all -- systemctl status nomad
colmena exec --on @all -- tailscale status

Common Tasks

Update System Configurations

1. Edit host configuration files in operate/hosts/
2. Apply changes:

   colmena apply --impure
   

Add a New Host

1. Provision infrastructure in deploy/
2. Run ncm-prepare to generate configuration
3. Customize operate/hosts/<hostname>/ as needed
4. Deploy with colmena apply

Manage Secrets

Secrets are managed centrally with agenix-rekey. To edit a secret:

# Edit the master secret
cd secrets/master/
agenix -e <secret-name> -i ~/.ssh/id_rsa_agenix

# Rekey for all hosts (auto-encrypts for each host's public key)
nix run .#agenix-rekey.x86_64-linux.rekey

# Add rekeyed secrets to git
git add secrets/rekeyed/

When host SSH keys change (e.g., after instance recreation), just run the rekey command to automatically re-encrypt secrets for the new keys.

Rebuild Single Node

colmena apply --on ncm-0 --impure --build-on-target

View Configuration Diff

colmena build --show-trace

Configuration Details

Nomad Cluster

Each node runs Nomad in both server and client mode, forming a highly-available cluster. Configuration in nomad.nix includes:

• Server mode (consensus protocol)
• Client mode (workload execution)
• Container networking (CNI plugins)
• Storage plugins (CSI)

Networking

• Public interface: Direct internet access
• Private VLAN: Inter-node communication (enp0s5)
• Tailscale: Secure VPN overlay (tailscale0)
• Container networking: Podman bridge (podman0)

Firewall trusts: tailscale0, enp0s5, podman0

Services

Common services configured:

• SSH (port 22)
• Nomad (ports 4646, 4647, 4648)
• HTTP/HTTPS (ports 80, 443)
• Tailscale

Troubleshooting

Connection Issues

Check SSH connectivity:

ssh root@<node-ip>

Verify deployment target in operate/hosts/<hostname>/default.nix:

deployment.targetHost = "x.x.x.x";

Build Failures

Build locally first:

colmena build

View detailed errors:

colmena apply --show-trace

Secret Decryption Failures

If secrets fail to decrypt on a host:

1. Check the host's public key is set correctly in operate/hosts/<hostname>/default.nix:

   age.rekey.hostPubkey = "age1...";
   

2. Verify the master key can decrypt the source secret:

   cd secrets/master/
   agenix -d <secret-name> -i ~/.ssh/id_rsa_agenix
   

3. Re-run agenix-rekey to regenerate host-specific secrets:

   nix run .#agenix-rekey.x86_64-linux.rekey --force
   git add secrets/rekeyed/
   colmena apply --on <hostname> --impure