ncm/MIGRATION_GUIDE.md

Migration Guide: count → for_each

This guide explains how to migrate your existing NCM cluster from the count-based infrastructure to the new for_each-based approach that enables blue/green deployments.

What Changed?

Before (count-based)

# deploy/variables.tf
variable "node_count" {
  default = 3
}

# deploy/compute_providers/linode/main.tf
resource "linode_instance" "ncm" {
  count = var.node_count
  label = "ncm-${count.index}"  # ncm-0, ncm-1, ncm-2
  ...
}

Problem: Adding or removing nodes changes the count.index, causing Terraform to recreate resources.

After (for_each-based)

# deploy/compute_providers/linode/nodes.tf
locals {
  nodes = {
    "ncm-0" = { generation = "blue", base_image = "..." }
    "ncm-1" = { generation = "blue", base_image = "..." }
    "ncm-2" = { generation = "blue", base_image = "..." }
  }
}

# deploy/compute_providers/linode/main.tf
resource "linode_instance" "ncm" {
  for_each = local.nodes
  label    = each.key  # ncm-0, ncm-1, ncm-2
  ...
}

Benefit: Each node has a stable identity. Adding ncm-3 doesn't affect ncm-0, ncm-1, ncm-2.

Migration Options

Option 1: Fresh Start (Recommended for Development)

If you can afford downtime or are in development:

1. Destroy existing infrastructure:

   cd deploy/
   tofu destroy
   

2. Update to new code (already done in this repo)

3. Define your nodes in deploy/compute_providers/linode/nodes.tf

4. Deploy fresh:

   tofu apply
   ncm-add-nodes
   colmena apply --impure
   

Option 2: Blue/Green Migration (Zero Downtime)

If you need zero downtime:

1. Keep existing nodes running with old Terraform state

2. Add new green nodes to nodes.tf:

   locals {
     nodes = {
       # Don't add old nodes yet - they're managed by old state
       # "ncm-0" = { ... }  # Skip for now
       # "ncm-1" = { ... }  # Skip for now
       # "ncm-2" = { ... }  # Skip for now
   
       # Add new nodes
       "ncm-3" = {
         generation = "green"
         base_image = var.compute_provider_instance_base_image
       }
       "ncm-4" = {
         generation = "green"
         base_image = var.compute_provider_instance_base_image
       }
       "ncm-5" = {
         generation = "green"
         base_image = var.compute_provider_instance_base_image
       }
     }
   }
   

3. Deploy new nodes:

   cd deploy/
   tofu apply  # Creates ncm-3, ncm-4, ncm-5
   

4. Configure new nodes:

   ncm-add-nodes
   colmena apply --on ncm-3,ncm-4,ncm-5 --impure
   

5. Migrate workloads:

   nomad node drain -enable -yes ncm-0
   nomad node drain -enable -yes ncm-1
   nomad node drain -enable -yes ncm-2
   

6. Clean up old infrastructure:

   First, remove old nodes from old Terraform state:

   cd deploy/
   tofu state rm 'module.compute_provider.linode_instance.ncm[0]'
   tofu state rm 'module.compute_provider.linode_instance.ncm[1]'
   tofu state rm 'module.compute_provider.linode_instance.ncm[2]'
   # Also remove associated resources (configs, disks, etc.)
   tofu state list | grep '\[0\]' | xargs -n1 tofu state rm
   tofu state list | grep '\[1\]' | xargs -n1 tofu state rm
   tofu state list | grep '\[2\]' | xargs -n1 tofu state rm
   

   Then manually destroy old nodes via Linode console or CLI.

   Finally, remove old host configs:

   mv operate/hosts/ncm-{0,1,2} operate/hosts/_archive/
   nix-editor flake.nix "outputs.colmena.ncm-0" -d
   nix-editor flake.nix "outputs.colmena.ncm-1" -d
   nix-editor flake.nix "outputs.colmena.ncm-2" -d
   

Option 3: Import Existing Resources (Advanced)

If you want to keep the exact same nodes (ncm-0, ncm-1, ncm-2) but migrate them to the new approach:

Warning: This is complex and risky. Only attempt if you're comfortable with Terraform state manipulation.

1. Define your existing nodes in nodes.tf:

   locals {
     nodes = {
       "ncm-0" = {
         generation = "current"
         base_image = "private/xxxxx"  # Your current image
       }
       "ncm-1" = {
         generation = "current"
         base_image = "private/xxxxx"
       }
       "ncm-2" = {
         generation = "current"
         base_image = "private/xxxxx"
       }
     }
   }
   

2. Get existing resource IDs:

   cd deploy/
   tofu state show 'module.compute_provider.linode_instance.ncm[0]' | grep "^id"
   # Note the Linode ID for each resource
   

3. Remove old resources from state (without destroying):

   tofu state rm 'module.compute_provider.linode_instance.ncm'
   tofu state rm 'module.compute_provider.linode_instance_config.config'
   tofu state rm 'module.compute_provider.linode_instance_disk.boot'
   tofu state rm 'module.compute_provider.linode_instance_disk.swap'
   

4. Import resources with new keys:

   tofu import 'module.compute_provider.linode_instance.ncm["ncm-0"]' <linode-id-0>
   tofu import 'module.compute_provider.linode_instance.ncm["ncm-1"]' <linode-id-1>
   tofu import 'module.compute_provider.linode_instance.ncm["ncm-2"]' <linode-id-2>
   # Repeat for configs, disks, etc.
   

5. Verify:

   tofu plan  # Should show no changes
   

This is error-prone. We recommend Option 1 or 2 instead.

Verification

After migration, verify:

# Check Terraform sees all nodes correctly
cd deploy/
tofu state list | grep linode_instance.ncm

# Check nodes are accessible
colmena exec --on @all -- hostname

# Check Nomad cluster
nomad node status

# Check workloads
nomad job status

Rollback

If something goes wrong during migration:

If you used Option 1 (Fresh Start)

• Restore from backup
• Or re-deploy from old commit

If you used Option 2 (Blue/Green)

• Keep old nodes running
• Remove new nodes: comment out in nodes.tf, run tofu apply
• Continue using old cluster

If you used Option 3 (Import)

• Restore Terraform state from backup:

  cp terraform.tfstate.backup terraform.tfstate
  

• Revert code changes

Need Help?

• Check tofu plan output carefully before apply
• Keep backups of Terraform state
• Test in a dev environment first
• See BLUE_GREEN_DEPLOYMENT.md for operational guidance