feat: support blue/green deploys

2026-03-22 17:00:54 -07:00 · 2026-03-22 17:00:54 -07:00 · fe8dd040b3
commit fe8dd040b3
parent 6f5e982ee7
8 changed files with 815 additions and 30 deletions
--- a/BLUE_GREEN_DEPLOYMENT.md
+++ b/BLUE_GREEN_DEPLOYMENT.md
@ -0,0 +1,336 @@
+# Blue/Green Deployment Guide
+
+This guide explains how to perform zero-downtime cluster updates using blue/green deployments.
+
+## Overview
+
+The NCM cluster now supports incremental node additions and removals, allowing you to:
+- Add new nodes without touching existing ones
+- Deploy new base images to new nodes
+- Gradually migrate workloads from old to new nodes
+- Remove old nodes once migration is complete
+
+## Node Identity
+
+Each node is defined in `deploy/compute_providers/linode/nodes.tf` with:
+- **Unique name**: Stable identifier (e.g., `ncm-0`, `ncm-3`)
+- **Generation label**: Version tag (e.g., `blue`, `green`, `v1`, `v2`)
+- **Base image**: Can differ between nodes, allowing gradual image rollouts
+- **Node type**: Optional override for different instance sizes
+
+## Workflow
+
+### Phase 1: Add New Nodes (Green)
+
+#### 1. Build New Base Image (Optional)
+
+If you're deploying a new OS image:
+
+```bash
+cd build/
+# Make your changes to systems/default/configuration.nix
+nix build .#linode
+linode-cli image-upload --label "nixos-green-$(date +%Y%m%d)" \
+  --description "NixOS Green Generation" \
+  --region "us-east" result/nixos.img.gz
+```
+
+Note the new image ID (e.g., `private/12345678`).
+
+#### 2. Define New Nodes
+
+Edit `deploy/compute_providers/linode/nodes.tf`:
+
+```hcl
+locals {
+  nodes = {
+    # Existing blue generation (DON'T MODIFY THESE)
+    "ncm-0" = {
+      generation = "blue"
+      base_image = "private/11111111"  # old image
+    }
+    "ncm-1" = {
+      generation = "blue"
+      base_image = "private/11111111"
+    }
+    "ncm-2" = {
+      generation = "blue"
+      base_image = "private/11111111"
+    }
+
+    # NEW: Green generation nodes
+    "ncm-3" = {
+      generation = "green"
+      base_image = "private/12345678"  # new image (or reuse old)
+    }
+    "ncm-4" = {
+      generation = "green"
+      base_image = "private/12345678"
+    }
+    "ncm-5" = {
+      generation = "green"
+      base_image = "private/12345678"
+    }
+  }
+}
+```
+
+#### 3. Deploy New Infrastructure
+
+```bash
+cd deploy/
+tofu plan  # Review - should show 3 new nodes, 0 changes to existing
+tofu apply
+```
+
+**Critical**: Verify the plan shows **only additions**, no modifications to existing nodes.
+
+#### 4. Prepare New Node Configurations
+
+Update the `ncm-prepare` script to handle only new nodes. For now, manually prepare:
+
+```bash
+# Get new node details
+cd deploy/
+tofu output -json cluster_node_details | jq '.value[] | select(.generation == "green")'
+
+# For each new node, create config directory
+for node in ncm-3 ncm-4 ncm-5; do
+  cp -r operate/templates/* operate/hosts/$node/
+  # Follow the ncm-prepare steps manually for each node
+done
+
+# Or run the full ncm-prepare (will regenerate all nodes)
+ncm-prepare
+```
+
+#### 5. Deploy NixOS Configurations to New Nodes
+
+```bash
+# Deploy only to new nodes
+colmena apply --on ncm-3,ncm-4,ncm-5 --impure
+
+# Verify Nomad cluster sees new nodes
+nomad node status
+# Should show 6 nodes (3 blue + 3 green)
+```
+
+### Phase 2: Migrate Workloads
+
+#### 6. Verify New Nodes
+
+Check that new nodes are healthy:
+
+```bash
+nomad node status
+# All 6 nodes should show "ready"
+
+# Check node metadata
+nomad node status -verbose ncm-3 | grep "Attributes"
+```
+
+#### 7. Drain Old Nodes
+
+Gradually drain workloads from blue nodes to green:
+
+```bash
+# Mark blue nodes as ineligible for new workloads
+nomad node eligibility -disable ncm-0
+nomad node eligibility -disable ncm-1
+nomad node eligibility -disable ncm-2
+
+# Drain one node at a time
+nomad node drain -enable -yes ncm-0
+
+# Wait for allocations to migrate
+nomad node status ncm-0
+# Should show 0 running allocations
+
+# Repeat for other blue nodes
+nomad node drain -enable -yes ncm-1
+nomad node drain -enable -yes ncm-2
+```
+
+#### 8. Verify Workload Migration
+
+Ensure all jobs are running on green nodes:
+
+```bash
+nomad job status
+# Check each job
+
+nomad job status caddy
+# Allocations should all be on ncm-3, ncm-4, ncm-5
+```
+
+### Phase 3: Remove Old Nodes (Blue)
+
+#### 9. Remove Old Nodes from Terraform
+
+Edit `deploy/compute_providers/linode/nodes.tf` and **comment out or delete** blue nodes:
+
+```hcl
+locals {
+  nodes = {
+    # OLD: Blue generation nodes (REMOVED)
+    # "ncm-0" = { ... }
+    # "ncm-1" = { ... }
+    # "ncm-2" = { ... }
+
+    # Current: Green generation nodes
+    "ncm-3" = {
+      generation = "green"
+      base_image = "private/12345678"
+    }
+    "ncm-4" = {
+      generation = "green"
+      base_image = "private/12345678"
+    }
+    "ncm-5" = {
+      generation = "green"
+      base_image = "private/12345678"
+    }
+  }
+}
+```
+
+#### 10. Destroy Old Infrastructure
+
+```bash
+cd deploy/
+tofu plan  # Review - should show 3 deletions, 0 changes to green nodes
+tofu apply
+```
+
+**Critical**: Verify the plan shows **only deletions** of blue nodes.
+
+#### 11. Clean Up Operate Configs
+
+```bash
+# Archive old configs (don't delete in case you need to reference)
+mkdir -p operate/hosts/_archive
+mv operate/hosts/ncm-{0,1,2} operate/hosts/_archive/
+
+# Update flake.nix to remove old nodes
+nix-editor flake.nix "outputs.colmena.ncm-0" -d
+nix-editor flake.nix "outputs.colmena.ncm-1" -d
+nix-editor flake.nix "outputs.colmena.ncm-2" -d
+```
+
+## Tips
+
+### Gradual Rollout
+
+Instead of deploying all 3 green nodes at once, add them one at a time:
+
+```bash
+# Add ncm-3 only
+# ... deploy and verify ...
+
+# Add ncm-4
+# ... deploy and verify ...
+
+# Add ncm-5
+# ... deploy and verify ...
+```
+
+### Canary Deployments
+
+Add a single new-generation node first:
+
+```bash
+# Add only ncm-3 with new image
+# Wait and observe
+# If stable, add ncm-4 and ncm-5
+```
+
+### Node Naming
+
+You can use any naming scheme:
+
+```hcl
+"ncm-blue-1" = { generation = "blue", ... }
+"ncm-green-1" = { generation = "green", ... }
+```
+
+Or version numbers:
+
+```hcl
+"ncm-v1-0" = { generation = "v1", ... }
+"ncm-v2-0" = { generation = "v2", ... }
+```
+
+### Rollback
+
+If green nodes have issues, drain them and keep blue nodes:
+
+```bash
+nomad node drain -enable -yes ncm-3
+nomad node drain -enable -yes ncm-4
+nomad node drain -enable -yes ncm-5
+
+nomad node eligibility -enable ncm-0
+nomad node eligibility -enable ncm-1
+nomad node eligibility -enable ncm-2
+```
+
+Then remove green nodes from Terraform.
+
+### Testing Before Production
+
+1. Add a single green node
+2. Deploy a test workload to it specifically
+3. Verify functionality
+4. If good, continue with full rollout
+5. If bad, remove green node and investigate
+
+## Safety Checklist
+
+Before each `tofu apply`:
+
+- [ ] Run `tofu plan` and **carefully review** the output
+- [ ] Verify **no changes** to nodes you want to keep
+- [ ] Verify **only additions** when adding nodes
+- [ ] Verify **only deletions** when removing nodes
+- [ ] Ensure you have backups of critical data
+- [ ] Test connectivity to new nodes before draining old ones
+- [ ] Verify all workloads are healthy after migration
+
+## Troubleshooting
+
+### New Nodes Not Joining Nomad Cluster
+
+Check Nomad logs on the new node:
+
+```bash
+ssh root@<new-node-ip> journalctl -u nomad -f
+```
+
+Verify Tailscale connectivity:
+
+```bash
+colmena exec --on ncm-3 -- tailscale status
+```
+
+### Workloads Not Migrating
+
+Check node constraints in job specs:
+
+```bash
+nomad job inspect caddy | jq '.Job.TaskGroups[].Constraints'
+```
+
+Check node eligibility:
+
+```bash
+nomad node status -verbose ncm-3 | grep Eligibility
+```
+
+### Terraform Wants to Recreate Nodes
+
+This usually means you modified an existing node definition. Instead:
+- Never modify existing node entries
+- Always add new entries for new nodes
+- Remove old entries when decommissioning
+
+If this happens, revert your changes and start over.
--- a/MIGRATION_GUIDE.md
+++ b/MIGRATION_GUIDE.md
@ -0,0 +1,244 @@
+# Migration Guide: count → for_each
+
+This guide explains how to migrate your existing NCM cluster from the count-based infrastructure to the new for_each-based approach that enables blue/green deployments.
+
+## What Changed?
+
+### Before (count-based)
+```hcl
+# deploy/variables.tf
+variable "node_count" {
+  default = 3
+}
+
+# deploy/compute_providers/linode/main.tf
+resource "linode_instance" "ncm" {
+  count = var.node_count
+  label = "ncm-${count.index}"  # ncm-0, ncm-1, ncm-2
+  ...
+}
+```
+
+**Problem**: Adding or removing nodes changes the count.index, causing Terraform to recreate resources.
+
+### After (for_each-based)
+
+```hcl
+# deploy/compute_providers/linode/nodes.tf
+locals {
+  nodes = {
+    "ncm-0" = { generation = "blue", base_image = "..." }
+    "ncm-1" = { generation = "blue", base_image = "..." }
+    "ncm-2" = { generation = "blue", base_image = "..." }
+  }
+}
+
+# deploy/compute_providers/linode/main.tf
+resource "linode_instance" "ncm" {
+  for_each = local.nodes
+  label    = each.key  # ncm-0, ncm-1, ncm-2
+  ...
+}
+```
+
+**Benefit**: Each node has a stable identity. Adding `ncm-3` doesn't affect `ncm-0`, `ncm-1`, `ncm-2`.
+
+## Migration Options
+
+### Option 1: Fresh Start (Recommended for Development)
+
+If you can afford downtime or are in development:
+
+1. **Destroy existing infrastructure**:
+   ```bash
+   cd deploy/
+   tofu destroy
+   ```
+
+2. **Update to new code** (already done in this repo)
+
+3. **Define your nodes** in `deploy/compute_providers/linode/nodes.tf`
+
+4. **Deploy fresh**:
+   ```bash
+   tofu apply
+   ncm-add-nodes
+   colmena apply --impure
+   ```
+
+### Option 2: Blue/Green Migration (Zero Downtime)
+
+If you need zero downtime:
+
+1. **Keep existing nodes running** with old Terraform state
+
+2. **Add new green nodes** to `nodes.tf`:
+   ```hcl
+   locals {
+     nodes = {
+       # Don't add old nodes yet - they're managed by old state
+       # "ncm-0" = { ... }  # Skip for now
+       # "ncm-1" = { ... }  # Skip for now
+       # "ncm-2" = { ... }  # Skip for now
+
+       # Add new nodes
+       "ncm-3" = {
+         generation = "green"
+         base_image = var.compute_provider_instance_base_image
+       }
+       "ncm-4" = {
+         generation = "green"
+         base_image = var.compute_provider_instance_base_image
+       }
+       "ncm-5" = {
+         generation = "green"
+         base_image = var.compute_provider_instance_base_image
+       }
+     }
+   }
+   ```
+
+3. **Deploy new nodes**:
+   ```bash
+   cd deploy/
+   tofu apply  # Creates ncm-3, ncm-4, ncm-5
+   ```
+
+4. **Configure new nodes**:
+   ```bash
+   ncm-add-nodes
+   colmena apply --on ncm-3,ncm-4,ncm-5 --impure
+   ```
+
+5. **Migrate workloads**:
+   ```bash
+   nomad node drain -enable -yes ncm-0
+   nomad node drain -enable -yes ncm-1
+   nomad node drain -enable -yes ncm-2
+   ```
+
+6. **Clean up old infrastructure**:
+
+   First, remove old nodes from old Terraform state:
+   ```bash
+   cd deploy/
+   tofu state rm 'module.compute_provider.linode_instance.ncm[0]'
+   tofu state rm 'module.compute_provider.linode_instance.ncm[1]'
+   tofu state rm 'module.compute_provider.linode_instance.ncm[2]'
+   # Also remove associated resources (configs, disks, etc.)
+   tofu state list | grep '\[0\]' | xargs -n1 tofu state rm
+   tofu state list | grep '\[1\]' | xargs -n1 tofu state rm
+   tofu state list | grep '\[2\]' | xargs -n1 tofu state rm
+   ```
+
+   Then manually destroy old nodes via Linode console or CLI.
+
+   Finally, remove old host configs:
+   ```bash
+   mv operate/hosts/ncm-{0,1,2} operate/hosts/_archive/
+   nix-editor flake.nix "outputs.colmena.ncm-0" -d
+   nix-editor flake.nix "outputs.colmena.ncm-1" -d
+   nix-editor flake.nix "outputs.colmena.ncm-2" -d
+   ```
+
+### Option 3: Import Existing Resources (Advanced)
+
+If you want to keep the exact same nodes (ncm-0, ncm-1, ncm-2) but migrate them to the new approach:
+
+**Warning**: This is complex and risky. Only attempt if you're comfortable with Terraform state manipulation.
+
+1. **Define your existing nodes** in `nodes.tf`:
+   ```hcl
+   locals {
+     nodes = {
+       "ncm-0" = {
+         generation = "current"
+         base_image = "private/xxxxx"  # Your current image
+       }
+       "ncm-1" = {
+         generation = "current"
+         base_image = "private/xxxxx"
+       }
+       "ncm-2" = {
+         generation = "current"
+         base_image = "private/xxxxx"
+       }
+     }
+   }
+   ```
+
+2. **Get existing resource IDs**:
+   ```bash
+   cd deploy/
+   tofu state show 'module.compute_provider.linode_instance.ncm[0]' | grep "^id"
+   # Note the Linode ID for each resource
+   ```
+
+3. **Remove old resources from state** (without destroying):
+   ```bash
+   tofu state rm 'module.compute_provider.linode_instance.ncm'
+   tofu state rm 'module.compute_provider.linode_instance_config.config'
+   tofu state rm 'module.compute_provider.linode_instance_disk.boot'
+   tofu state rm 'module.compute_provider.linode_instance_disk.swap'
+   ```
+
+4. **Import resources with new keys**:
+   ```bash
+   tofu import 'module.compute_provider.linode_instance.ncm["ncm-0"]' <linode-id-0>
+   tofu import 'module.compute_provider.linode_instance.ncm["ncm-1"]' <linode-id-1>
+   tofu import 'module.compute_provider.linode_instance.ncm["ncm-2"]' <linode-id-2>
+   # Repeat for configs, disks, etc.
+   ```
+
+5. **Verify**:
+   ```bash
+   tofu plan  # Should show no changes
+   ```
+
+**This is error-prone. We recommend Option 1 or 2 instead.**
+
+## Verification
+
+After migration, verify:
+
+```bash
+# Check Terraform sees all nodes correctly
+cd deploy/
+tofu state list | grep linode_instance.ncm
+
+# Check nodes are accessible
+colmena exec --on @all -- hostname
+
+# Check Nomad cluster
+nomad node status
+
+# Check workloads
+nomad job status
+```
+
+## Rollback
+
+If something goes wrong during migration:
+
+### If you used Option 1 (Fresh Start)
+- Restore from backup
+- Or re-deploy from old commit
+
+### If you used Option 2 (Blue/Green)
+- Keep old nodes running
+- Remove new nodes: comment out in `nodes.tf`, run `tofu apply`
+- Continue using old cluster
+
+### If you used Option 3 (Import)
+- Restore Terraform state from backup:
+  ```bash
+  cp terraform.tfstate.backup terraform.tfstate
+  ```
+- Revert code changes
+
+## Need Help?
+
+- Check `tofu plan` output carefully before `apply`
+- Keep backups of Terraform state
+- Test in a dev environment first
+- See [BLUE_GREEN_DEPLOYMENT.md](BLUE_GREEN_DEPLOYMENT.md) for operational guidance
--- a/README.md
+++ b/README.md
@ -93,6 +93,32 @@ See [jobs/README.md](jobs/README.md) for available job specifications.
 4. **Configure environment**: Copy `.env.sample` and fill in required values
 5. **Follow the workflow** outlined above

+## Blue/Green Deployments
+
+NCM supports zero-downtime cluster updates through blue/green deployments. Add new nodes alongside existing ones, migrate workloads, then remove old nodes.
+
+See [BLUE_GREEN_DEPLOYMENT.md](BLUE_GREEN_DEPLOYMENT.md) for the complete guide.
+
+Quick example:
+
+```bash
+# 1. Add new nodes to deploy/compute_providers/linode/nodes.tf
+# 2. Deploy infrastructure
+cd deploy/ && tofu apply
+
+# 3. Configure new nodes only
+ncm-add-nodes
+
+# 4. Deploy NixOS to new nodes
+colmena apply --on ncm-3,ncm-4,ncm-5 --impure
+
+# 5. Drain old nodes
+nomad node drain -enable -yes ncm-0
+
+# 6. Remove old nodes from nodes.tf and destroy
+cd deploy/ && tofu apply
+```
+
 ## Environment Variables

 Required environment variables are documented in `.env.sample`. Use the `ess` tool (env-sample-sync) to keep them synchronized.
--- a/deploy/README.md
+++ b/deploy/README.md
@ -83,14 +83,72 @@ tofu destroy

 **Warning**: This will permanently delete all provisioned resources.

+## Node Management
+
+### Node Definitions
+
+Nodes are explicitly defined in `compute_providers/linode/nodes.tf`. Each node has:
+
+- **Unique name**: Stable identifier (e.g., `ncm-0`, `ncm-3`)
+- **Generation label**: Version/generation tag for tracking (e.g., `blue`, `green`)
+- **Base image**: Can differ per node, enabling gradual image rollouts
+- **Private IP**: Deterministically assigned based on node name
+
+Example:
+
+```hcl
+locals {
+  nodes = {
+    "ncm-0" = {
+      generation = "blue"
+      base_image = "private/11111111"
+    }
+    "ncm-3" = {
+      generation = "green"
+      base_image = "private/12345678"  # Different image
+    }
+  }
+}
+```
+
+### Adding Nodes
+
+To add new nodes without affecting existing ones:
+
+1. Edit `compute_providers/linode/nodes.tf`
+2. Add new node definitions
+3. Run `tofu plan` to verify only additions
+4. Run `tofu apply`
+
+### Removing Nodes
+
+To remove nodes:
+
+1. Comment out or delete node definitions in `nodes.tf`
+2. Run `tofu plan` to verify only deletions
+3. Run `tofu apply`
+
+**Important**: Never modify existing node definitions. Always add new entries or remove old entries.
+
+### Blue/Green Deployments
+
+The explicit node definition approach enables blue/green deployments:
+
+- Add new "green" generation nodes
+- Migrate workloads from "blue" to "green"
+- Remove "blue" nodes
+
+See [../BLUE_GREEN_DEPLOYMENT.md](../BLUE_GREEN_DEPLOYMENT.md) for the complete workflow.
+
 ## Architecture

 ### Modules

 - **compute_providers/linode**: Linode-specific provisioning logic
-  - Creates compute instances
-  - Configures VLANs for private networking
+  - Creates compute instances with stable identities
+  - Configures VPCs for private networking
  - Sets up initial user access
+  - Tags nodes by generation for tracking

 ### Outputs

--- a/deploy/compute_providers/linode/main.tf
+++ b/deploy/compute_providers/linode/main.tf
@ -20,28 +20,22 @@ resource "linode_sshkey" "admin" {
 # Instances
 #####################################################################################################################

-locals {
-  # These are the private IP addresses that are assigned to each node
-  # We start at the buttom of the cidr range + 2 because 0 is not routable and 1 is reserved; hence "i + 2" below
-  node_private_ips = {
-    value = [for i in range(var.node_count) : "${cidrhost(var.internal_cidr_block, i + 2)}"]
-  }
-}
-
 resource "linode_instance" "ncm" {
-  count  = var.node_count
-  label  = "ncm-${count.index}"
-  tags   = ["ncm"]
+  for_each = local.nodes
+
+  label  = each.key
+  tags   = ["ncm", "generation:${each.value.generation}"]
  region = var.compute_provider_region
-  type   = var.compute_provider_node_type
+  type   = lookup(each.value, "node_type", var.compute_provider_node_type)
 }

 resource "linode_instance_config" "config" {
-  count        = var.node_count
-  label        = "nvm-${count.index}"
+  for_each = local.nodes
+
+  label        = "${each.key}-config"
  kernel       = "linode/grub2"
  booted       = true
-  linode_id    = linode_instance.ncm[count.index].id
+  linode_id    = linode_instance.ncm[each.key].id
  memory_limit = 0
  root_device  = "/dev/sda"
  run_level    = "default"
@ -55,19 +49,18 @@ resource "linode_instance_config" "config" {
    purpose   = "vpc"
    subnet_id = linode_vpc_subnet.ncm.id
    ipv4 {
-      vpc = local.node_private_ips.value[count.index]
+      vpc = local.node_private_ips[each.key]
    }
  }

-
  device {
    device_name = "sda"
-    disk_id     = linode_instance_disk.boot[count.index].id
+    disk_id     = linode_instance_disk.boot[each.key].id
  }

  device {
    device_name = "sdb"
-    disk_id     = linode_instance_disk.swap[count.index].id
+    disk_id     = linode_instance_disk.swap[each.key].id
  }
 }

@ -84,19 +77,21 @@ resource "linode_vpc_subnet" "ncm" {
 }

 resource "linode_instance_disk" "boot" {
-  count           = var.node_count
+  for_each = local.nodes
+
  label           = "boot"
-  linode_id       = linode_instance.ncm[count.index].id
-  size            = linode_instance.ncm[count.index].specs.0.disk - 512
-  image           = var.compute_provider_instance_base_image
+  linode_id       = linode_instance.ncm[each.key].id
+  size            = linode_instance.ncm[each.key].specs.0.disk - 512
+  image           = each.value.base_image
  authorized_keys = [linode_sshkey.admin.ssh_key]
 }

 # Create a swap disk
 resource "linode_instance_disk" "swap" {
-  count      = var.node_count
+  for_each = local.nodes
+
  label      = "swap"
-  linode_id  = linode_instance.ncm[count.index].id
+  linode_id  = linode_instance.ncm[each.key].id
  size       = 512
  filesystem = "swap"
 }
@ -146,6 +141,6 @@ resource "linode_firewall" "ncm" {
  inbound_policy  = "DROP"
  outbound_policy = "ACCEPT"

-  linodes = linode_instance.ncm[*].id
+  linodes = [for instance in linode_instance.ncm : instance.id]
 }

--- a/deploy/compute_providers/linode/nodes.tf
+++ b/deploy/compute_providers/linode/nodes.tf
@ -0,0 +1,52 @@
+# Node Definitions
+# Each node has a stable identity. Add new nodes here for blue/green deployments.
+# Nodes can have different base images, allowing gradual rollout of new images.
+
+locals {
+  # Define your nodes here. Each node needs:
+  # - name: unique identifier (used in hostname and as Terraform resource key)
+  # - generation: version/generation label (e.g., "blue", "green", "v1", "v2")
+  # - base_image: override default base image (optional)
+  # - node_type: override default node type (optional)
+  nodes = {
+    # Existing nodes (generation: blue)
+    "ncm-0" = {
+      generation = "blue"
+      base_image = var.compute_provider_instance_base_image
+    }
+    "ncm-1" = {
+      generation = "blue"
+      base_image = var.compute_provider_instance_base_image
+    }
+    "ncm-2" = {
+      generation = "blue"
+      base_image = var.compute_provider_instance_base_image
+    }
+
+    # Example: Add new green generation nodes
+    # Uncomment and customize when ready to add new nodes:
+    # "ncm-3" = {
+    #   generation = "green"
+    #   base_image = var.compute_provider_instance_base_image  # or use a new image
+    # }
+    # "ncm-4" = {
+    #   generation = "green"
+    #   base_image = var.compute_provider_instance_base_image
+    # }
+    # "ncm-5" = {
+    #   generation = "green"
+    #   base_image = var.compute_provider_instance_base_image
+    # }
+  }
+
+  # Assign stable private IPs based on node name
+  # Uses hash of node name to deterministically assign IPs
+  node_private_ips = {
+    for name, config in local.nodes :
+    name => cidrhost(
+      var.internal_cidr_block,
+      # Start at .10 and use last digit of hash for stable but distributed IPs
+      10 + (sum([for c in split("", name) : index(split("", "0123456789abcdefghijklmnopqrstuvwxyz-"), c)]) % 240)
+    )
+  }
+}
--- a/deploy/compute_providers/linode/outputs.tf
+++ b/deploy/compute_providers/linode/outputs.tf
@ -1,12 +1,13 @@
 output "node_details" {
  value = [
-    for idx, instance in linode_instance.ncm :
+    for name, instance in linode_instance.ncm :
    {
      id           = instance.id
      label        = instance.label
      ipv4         = instance.ip_address
      ipv6         = instance.ipv6
-      private_ipv4 = linode_instance_config.config[idx].interface[1].ipv4[0].vpc
+      private_ipv4 = local.node_private_ips[name]
+      generation   = local.nodes[name].generation
    }
  ]
 }
--- a/flake.nix
+++ b/flake.nix
@ -144,6 +144,79 @@
                  git add ./operate/hosts
                '';
              };
+              ncm-add-nodes = {
+                description = "Add new nodes without touching existing ones (for blue/green deployments)";
+                exec = ''
+                  #!/run/current-system/sw/bin/bash
+                  set -e
+
+                  hosts_dir="./operate/hosts"
+                  mkdir -p "$hosts_dir"
+
+                  echo "Checking for new nodes..."
+                  new_count=0
+
+                  tofu -chdir="./deploy" output -json | jq -r '.cluster_node_details.value[] | "\(.label)|\(.ipv4)|\(.private_ipv4)|\(.generation)"' | while IFS='|' read -r hostName host private_ip generation; do
+                      host_dir="./operate/hosts/$hostName"
+
+                      # Skip if host directory already exists
+                      if [ -d "$host_dir" ]; then
+                          echo "  ⏭  Skipping $hostName (already exists)"
+                          continue
+                      fi
+
+                      echo "  ✨ Adding new node: $hostName (generation: $generation)"
+                      new_count=$((new_count + 1))
+
+                      echo "    - Getting SSH key from $host..."
+                      ssh_public_key=$(ssh-keyscan "$host" 2>/dev/null | grep ssh-ed25519 | cut -d " " -f 2,3)
+
+                      if [ -z "$ssh_public_key" ]; then
+                          echo "    ⚠  Warning: Could not get SSH key from $host"
+                          echo "    ℹ  Make sure the node is accessible via SSH"
+                          continue
+                      fi
+
+                      echo "    - Creating config directory..."
+                      mkdir -p "$host_dir/secrets"
+                      cp -rf operate/templates/* "$host_dir/"
+
+                      echo "    - Configuring NixOS settings..."
+                      nix-editor "$host_dir/default.nix" users.users.sfos.openssh.authorizedKeys.keys -i -v "[\"$TF_VAR_admin_ssh_public_key\"]"
+                      nix-editor "$host_dir/default.nix" users.users.root.openssh.authorizedKeys.keys -i -v "[\"$TF_VAR_admin_ssh_public_key\"]"
+                      nix-editor "$host_dir/default.nix" networking.hostName -i -v "\"$hostName\""
+                      nix-editor "$host_dir/default.nix" deployment.targetHost -i -v "\"$host\""
+
+                      echo "    - Configuring secrets..."
+                      nix-editor "$host_dir/secrets/secrets.nix" tailscale_auth_key.publicKeys -i -v "[\"$TF_VAR_admin_ssh_public_key\" \"$ssh_public_key\"]"
+                      nix-editor "$host_dir/secrets/secrets.nix" linode_cli_pat.publicKeys -i -v "[\"$TF_VAR_admin_ssh_public_key\" \"$ssh_public_key\"]"
+
+                      echo "    - Updating flake.nix..."
+                      nix-editor flake.nix "outputs.colmena.$hostName" -i -v "$host_dir"
+
+                      echo "    - Encrypting secrets..."
+                      cd "$host_dir/secrets"
+                      echo "$TF_VAR_tailscale_auth_key" | agenix -e tailscale_auth_key -i ~/.ssh/id_rsa_agenix 2>/dev/null
+                      echo "$TF_VAR_nodes_linode_cli_pat" | agenix -e linode_cli_pat -i ~/.ssh/id_rsa_agenix 2>/dev/null
+                      cd ../../../..
+
+                      echo "    ✅ Node $hostName configured successfully"
+                  done
+
+                  echo ""
+                  if [ "$new_count" -eq 0 ]; then
+                      echo "✨ No new nodes to add. All nodes are already configured."
+                  else
+                      echo "✅ Added $new_count new node(s)"
+                      echo ""
+                      echo "Next steps:"
+                      echo "  1. Review the new configs in operate/hosts/"
+                      echo "  2. Deploy with: colmena apply --on <node-names> --impure"
+                      echo "  3. Verify: nomad node status"
+                      git add ./operate/hosts
+                  fi
+                '';
+              };
            };
          }
        ];