ncm/deploy/README.md

Deploy

This module provisions cloud infrastructure using OpenTofu (Terraform).

Overview

The deploy module:

• Provisions compute instances on cloud providers (currently Linode)
• Configures networking (public/private IPs, VLANs)
• Sets up initial SSH access
• Integrates with Tailscale VPN
• Stores state remotely in S3-compatible object storage

Prerequisites

1. Cloud Provider Account: Currently supports Linode
2. Object Storage: S3-compatible bucket for Terraform state (e.g., Linode Object Storage)
3. Tailscale Account: For VPN connectivity
4. SSH Keys: For accessing provisioned nodes

Configuration

Environment Variables

Set the following variables (see .env in this directory):

TF_VAR_compute_provider_access_token    # Linode API token
TF_VAR_admin_ssh_public_key             # Your SSH public key
TF_VAR_tailscale_auth_key               # Tailscale auth key
TF_VAR_compute_provider_node_type       # e.g., "g6-standard-2"
TF_VAR_compute_provider_region          # e.g., "us-east"

Backend Configuration

The Terraform state is stored remotely in Linode Object Storage. Configure credentials:

export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"

Usage

Initial Deployment

# Initialize Terraform
tofu init

# Review planned changes
tofu plan

# Apply infrastructure changes
tofu apply

View Outputs

# View all outputs
tofu output

# View cluster details as JSON
tofu output -json cluster_node_details

Modify Infrastructure

Edit variables.tf or module configuration, then:

tofu plan
tofu apply

Destroy Infrastructure

tofu destroy

Warning: This will permanently delete all provisioned resources.

Node Management

Node Definitions

Nodes are explicitly defined in compute_providers/linode/nodes.tf. Each node has:

• Unique name: Stable identifier (e.g., ncm-0, ncm-3)
• Generation label: Version/generation tag for tracking (e.g., blue, green)
• Base image: Can differ per node, enabling gradual image rollouts
• Private IP: Deterministically assigned based on node name

Example:

locals {
  nodes = {
    "ncm-0" = {
      generation = "blue"
      base_image = "private/11111111"
    }
    "ncm-3" = {
      generation = "green"
      base_image = "private/12345678"  # Different image
    }
  }
}

Adding Nodes

To add new nodes without affecting existing ones:

1. Edit compute_providers/linode/nodes.tf
2. Add new node definitions
3. Run tofu plan to verify only additions
4. Run tofu apply

Removing Nodes

To remove nodes:

1. Comment out or delete node definitions in nodes.tf
2. Run tofu plan to verify only deletions
3. Run tofu apply

Important: Never modify existing node definitions. Always add new entries or remove old entries.

Blue/Green Deployments

The explicit node definition approach enables blue/green deployments:

• Add new "green" generation nodes
• Migrate workloads from "blue" to "green"
• Remove "blue" nodes

See ../BLUE_GREEN_DEPLOYMENT.md for the complete workflow.

Architecture

Modules

• compute_providers/linode: Linode-specific provisioning logic
  • Creates compute instances with stable identities
  • Configures VPCs for private networking
  • Sets up initial user access
  • Tags nodes by generation for tracking

Outputs

The module outputs cluster details including:

• Node labels (hostnames)
• Public IPv4 addresses
• Private IPv4 addresses
• SSH connection information

These outputs are consumed by:

1. The ncm-prepare script (generates operate/hosts configurations)
2. The jobs module (for Nomad connectivity)

Integration

After deploying infrastructure:

1. Run ncm-prepare from the project root to generate host configurations
2. Deploy NixOS configurations with Colmena (see operate/README.md)
3. Deploy workloads with Nomad (see jobs/README.md)

Troubleshooting

State Lock Issues

If apply/plan fails due to state lock:

tofu force-unlock <lock-id>

Provider Authentication

Verify credentials:

linode-cli regions list  # Test Linode access