From c84455f4d126b26b1d8a95456c4d8454fbf75184 Mon Sep 17 00:00:00 2001 From: Adriano Caloiaro Date: Mon, 20 Jan 2025 15:03:56 -0700 Subject: [PATCH] post: new fly.io postgres cluster post --- .../posts/fly-dot-io-postgres-failover-fix.md | 73 +++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 content/posts/fly-dot-io-postgres-failover-fix.md diff --git a/content/posts/fly-dot-io-postgres-failover-fix.md b/content/posts/fly-dot-io-postgres-failover-fix.md new file mode 100644 index 0000000..aa1764c --- /dev/null +++ b/content/posts/fly-dot-io-postgres-failover-fix.md @@ -0,0 +1,73 @@ +--- +title: Fly.io Postgres failover fix +cover: https://fly.io/static/images/brand/logo-landscape-inverted.svg +date: "2024-11-28T00:00:00Z" +comments: + host: indieweb.social + username: adriano + id: 113862958142410590 +tags: + - note-to-self +--- + +# Fly.io Postgres failover fix (`flyctl pg failover`) + +This is a note to myself, meant to be succinct and helpful. I'm sharing it publicly to save others time. + +Most of the time Fly.io works as I expect it to, but occasionally there are edge cases that lack documentation, public announcements, or both. + +It's possible that at some point Fly.io announced a breaking change and I missed it, but the behavior I observed deserves more than an announcement or silently released documentation. + +Today, I wanted to perform a manual leader failover to one of my cluster's followers, but it quickly failed in a way that wasn't immediately clear. + +## The problem + +Performing `flyctl pg failover` can fail for opaque reasons because the underlying error is eaten. + +Example +```bash +$ flyctl pg failover -a --debug +Performing a failover +Connecting to fdaa:9:1d32:a7b:94:1aeb:7b94:2... complete +Stopping current leader... 328725ec309d85 +Starting new leader +Promoting new leader... e784126feee248 +Connecting to fdaa:9:1d32:a7b:94:1aeb:7b94:2... complete +WARNING: unable to connect to remote host "fdaa:9:1d32:a7b:e:e63c:7516:2" via SSH +WARNING: unable to connect to remote host "fdaa:9:1d32:a7b:2b5:4c5e:f3b8:2" via SSH +... +``` + +Now this isn't entirely opaque -- we can see that _something_ is having a problem connecting to two cluster nodes via SSH. But what is trying to connect via SSH? My local machine, or one of the replicas in the Postgres cluster? + + +If you aren't aware of how Fly Postgres clustering works, under the hood it's simply [repmgr](https://github.com/EnterpriseDB/repmgr), which under _its_ hood uses [passwordless SSH sessions](https://www.repmgr.org/docs/4.4/performing-switchover.html) to orchestrate changes with cluster members. + +`fly pg failover` executes `repmgr standby switchover --siblings-follow` (amongst other things) on one of the cluster's follower nodes, which takes over the `primary` role in the cluster. + +So the errors above -- those are coming from a _follower_ cluster node attempting connections to other cluster nodes. Failover must fail because without SSH sessions to other cluster nodes, `repmgr` cannot orchestrate any changes. + +> Aside: `fly pg failover` should absolutely accept repmgr's `--dry-run` switch for performing dry runs. Currently, it does not. Typically `fly pg failover` stops the leader machine to make way for the new leader. A `--dry-run` switch should prevent any such service disruption, just as repmgr's dry run behavior prevents service disruption. + +Unfortunately, `repmgr` can eat errors from `ssh` and won't show you exactly _why_ some connections to remote hosts aren't possible. + +By using SSH directly (`ssh postgres@fdaa:9:1d32:a7b:e:e63c:7516:2`) I quickly spotted the underlying error. + +> postgres@fdaa:9:1d32:a7b:e:e63c:7516:2: Permission denied (publickey). + +At this point, the problem was clearly a SSH public key problem, but I wasn't aware of the below command that'll quickly fix your cluster members up with working ssh keys. + +## The fix + +To fix this issue, re-distribute SSH public keys throughout the cluster with + +> `fly pg renew-certs -a ` + + +``` +Your SSH certificate(s) have been renewed are set to expire in 36525 day(s) +Run fly deploy --app --image docker-hub-mirror.fly.io/flyio/postgres-flex:15.8@sha256:5016ffb34e66eca43d4f9ef2f898c166257bd28bd5095c41d049a5e3be15caf5 to apply the changes! +``` + +Don't forget to re-deploy your app after renewing certificates. +