I'm always looking for the easy way. Now certbot already makes retrieving TLS certificates from [Let's Encrypt](https://letsencrypt.org/) easy. But it's getting those certificates "into production" that tends to be less easy. This is the easy way to get Let's Encrypt TLS certificates into production with Hashicorp's Nomad.
This is an overview of what we'll be doing.
- Using Nomad's [docker driver](https://developer.hashicorp.com/nomad/docs/drivers/docker) to run docker container jobs.
- Using a [`cerbot` docker image](https://hub.docker.com/r/inetsoftware/certbot-dns-hetzner) that performs a [DNS-01 Challenge](https://letsencrypt.org/docs/challenge-types/) using Hetzner's DNS API.
- Using Nomad [host volumes](https://developer.hashicorp.com/nomad/docs/configuration/client#host_volume-block) to share certificates between nomad tasks.
- Using Nomad [lifecycle block](https://developer.hashicorp.com/nomad/docs/job-specification/lifecycle) to initialize certificates before Nginx startup.
- Reviewing how to set up Nomad [host volumes](https://developer.hashicorp.com/nomad/docs/configuration/client#host_volume-block)
- Setting up Nomad jobs that Nginx can reverse proxy to. However, the example nginx configuration provides an example that uses Nomad service discovery
- Auto-configuring DNS records to point at the Nomad client running our nginx job
If anything mentioned here is not _exactly_ the same as your stack, you'll still be able to take a lot away from this general approach. Stick around. For example, let's say your not using Hetzner, but AWS Route 53 for DNS -- there are certbot images that can create Route 53 records.
If you're here, you probably familiar with each of these, at least in theory if not in experience.
# Let's Go
Enough setup, let's get to it.
## What do with with Hetzner
Head over to Hetzner DNS and retrieve an API token: https://dns.hetzner.com/settings/api-token
Save your token somewhere; you'll need it later.
## What to do with Nomad
First of all, while I'm not going to get into the details of setting up host volumes, I will provide my cluster's client configuration for host volumes:
**Create host volumes in `/opt/nomad/config/client.hcl`**:
```
client {
enabled = true
host_volume "letsencrypt" {
path = "/var/www/certbot"
read_only = false
}
host_volume "certbot" {
path = "/etc/certbot"
read_only = false
}
}
```
> What does this config mean? We're making `/var/www/certbot` and `/etc/certbot` _from our nomad clients_ -- that is, the nodes that run nomad jobs -- available to be mounted inside of our nomad jobs. Doing so has implication that you should be aware of: **every node that your nginx job runs on will need to fetch new TLS certificates from Let's Encrypt**. If you're deploying your nginx job to dozens of nodes, or auto-scaling up and down frequently, this can result in you *hitting Let's Encrypt [API limits](https://letsencrypt.org/docs/rate-limits/)*. So tread lightly. Ideally, instead of "host volumes", you can use CSI volumes, where the `certbot` jobs claims a read/write CSI volume on startup, and `nginx` jobs make multi-node read claims to the same volume. That way, a single set of certs is shared across all nginx nodes.
After adding this client configuration, restart your client nodes.
**Create a combined Certbot / Ninx job**
[Full Example job.nomad.hcl](/files/job.nomad.hcl)
```hcl
variable "hetzner_dns_access_token" {
type = string
}
job "nginx" {
datacenters = ["dc1"]
namespace = "default"
spread {
attribute = "${node.unique.name}"
weight = 100
}
update {
stagger = "30s"
max_parallel = 1
}
group "work" {
# My cluster node names are cluster-0, cluster-1, etc.
# I run this job on a single node at a time, and "pin" it to the same node using the below constraint
# This way, once certificates are generated for my domains, they'll be reused until they expire since
# the job always runs on the same host where they were generated.
count = 1
constraint {
attribute = "${node.unique.name}"
operator = "set_contains_any"
# CHANGE THIS TO THE NAME OF THE NODE THAT WILL RUN NGINX
value = "cluster-0"
}
# Claim our host volumes, so they can be mounted inside our certbot/nginx jobs
volume "certbot" {
type = "host"
read_only = false
source = "certbot"
}
volume "letsencrypt" {
type = "host"
read_only = false
source = "letsencrypt"
}
network {
port "http" {
static = 80
}
port "https" {
static = 443
}
}
# This is the task that fetches certificates for all of our domains and places them on our host volume
# Doing so makes the certificates available in our `nginx` job
# Note: This task's lifecycle is "prestart", meaning it must complete before the `nginx` task starts
task "certbot-all-domains" {
driver = "docker"
user = "root"
config {
image = "inetsoftware/certbot-dns-hetzner"
# We use a custom entrypoint so we can script this tasks's behavior
entrypoint = ["${NOMAD_TASK_DIR}/run.sh"]
}
template {
data = <<EOF
#!/bin/sh
# Note, we're not dealing with bash here. We're dealing with 'ash', as this docker image is BusyBox linux
# Hence, we can't use normal bash arrays. Separate each domain by SPACES, NOTE COMMAS
DOMAINS="example.com foo.com bar.com"
# Set this to an empty string "" for production mode
# I use nomad for service discovery, but many nomad users prefer "consul" for this
provider = "nomad"
meta {
meta = "General purpose load balancer"
}
# Take this check with a grain of salt ant tune yours for yourself
check {
type = "tcp"
port = "http"
interval = "10s"
timeout = "2s"
}
}
}
}
}
```
While you'll obviously want to change all the `example.com` values, there are three other things you'll want to explicitly set:
You'll need to change the following line to be the name of one of your Nomad client nodes
`value = "cluster-0"`
When you're ready for production, you'll want to change `STAGING="--staging"` to `STAGING=""` in the `run.sh` script.
And if it's not obvious, you'll also want to change the `DOMAINS` variable in the `run.sh` script:
`DOMAINS="example.com foo.com bar.com"` to your domains for which Hetzner controls the zone.
**Deploy the job**
`nomad run -var="hetzner_dns_access_token=<YOUR TOKEN HERE>" job.nomad.hcl`
# Bonus for Terraform users
If you use the [Terraform Nomad provider](https://registry.terraform.io/providers/hashicorp/nomad/latest/docs), which I recommend you do, you can drop this into your terraform config to deploy the above job with Terraform:
That's all here is to it. Certbot performs the DNS-01 challenge using Hetzner's DNS API, saves the issued certificates on a host volume shared with the nginx job, and nginx's configuration points at the same host volume to fetch domain certificates.
Note that if you run this job on multiple hosts, or auto-scale this job across multiple hosts, you should be aware of [Let's Encrypt rate limits](https://letsencrypt.org/docs/rate-limits/). If you end up requesting too many of certificates for the same domain, LE can block your domain for long periods of time, so be careful.
Also note that the above job does not automate the creation of records pointing at your nomad client running the nginx job. This is another reasin why I've pinned this job to a specific client node. I have existing DNS records pointing at that node.
