ncm/README.md

NCM

NCM is a NixOS-based infrastructure management toolbox for building, deploying, and operating cloud infrastructure with declarative configuration.

Overview

NCM provides an end-to-end workflow for managing cloud infrastructure:

• Build custom NixOS images
• Provision cloud resources
• Configure and manage nodes
• Deploy containerized workloads

Architecture

This project is organized around four independent modules:

Directory	Purpose	Technologies
build/	Build blessed NixOS images for deployment	NixOS, nixos-generators
deploy/	Provision cloud infrastructure	OpenTofu/Terraform, Linode
operate/	Configure and manage deployed nodes	NixOS, Colmena, Tailscale
jobs/	Deploy workloads to the cluster	Nomad, Terraform

Workflow

1. Build Images

Build custom NixOS images for your target platform:

cd build/
nix build .#linode    # Build Linode image
nix build .#qemu      # Build QEMU VM for testing

See build/README.md for details on building and uploading images.

2. Deploy Infrastructure

Provision cloud resources with OpenTofu:

cd deploy/
tofu init
tofu plan
tofu apply

See deploy/README.md for configuration details.

3. Prepare Node Configurations

After infrastructure is provisioned, prepare NixOS configurations:

ncm-prepare  # Generates host configs in operate/hosts/

This script:

• Creates per-host configuration directories
• Sets up SSH keys and secrets
• Configures networking and Tailscale
• Updates the flake.nix with new hosts

4. Apply Node Configurations

Deploy NixOS configurations to all cluster nodes:

colmena apply --impure

See operate/README.md for details on node configuration.

5. Deploy Workloads

Deploy applications and services to the Nomad cluster:

cd jobs/
tofu init
tofu plan
tofu apply

See jobs/README.md for available job specifications.

Quick Start

1. Prerequisites: Ensure you have Nix with flakes enabled
2. Enter development environment: nix develop (or use direnv)
3. View available commands: The shell will show available helper scripts
4. Configure environment: Copy .env.sample and fill in required values
5. Follow the workflow outlined above

Blue/Green Deployments

NCM supports zero-downtime cluster updates through blue/green deployments. Add new nodes alongside existing ones, migrate workloads, then remove old nodes.

See BLUE_GREEN_DEPLOYMENT.md for the complete guide.

Quick example:

# 1. Add new nodes to deploy/compute_providers/linode/nodes.tf
# 2. Deploy infrastructure
cd deploy/ && tofu apply

# 3. Configure new nodes only
ncm-add-nodes

# 4. Deploy NixOS to new nodes
colmena apply --on ncm-3,ncm-4,ncm-5 --impure

# 5. Drain old nodes
nomad node drain -enable -yes ncm-0

# 6. Remove old nodes from nodes.tf and destroy
cd deploy/ && tofu apply

Environment Variables

Required environment variables are documented in .env.sample. Use the ess tool (env-sample-sync) to keep them synchronized.

Documentation

Each module has detailed documentation:

• build/README.md - Image building and uploading
• deploy/README.md - Infrastructure provisioning
• operate/README.md - Node configuration and management
• jobs/README.md - Workload deployment

Technologies

• NixOS: Declarative Linux distribution
• OpenTofu: Infrastructure as Code
• Colmena: NixOS deployment tool
• Nomad: Workload orchestration
• Tailscale: Zero-config VPN
• agenix-rekey: Automated age-based secrets management