post: new fly.io postgres cluster post

content/posts/fly-dot-io-postgres-failover-fix.md

@@ -0,0 +1,73 @@
+---
+title: Fly.io Postgres failover fix 
+cover: https://fly.io/static/images/brand/logo-landscape-inverted.svg
+date: "2024-11-28T00:00:00Z"
+comments:
+  host: indieweb.social
+  username: adriano
+  id: 113862958142410590
+tags:
+  - note-to-self 
+---
+
+# Fly.io Postgres failover fix (`flyctl pg failover`)
+
+This is a note to myself, meant to be succinct and helpful. I'm sharing it publicly to save others time. 
+
+Most of the time Fly.io works as I expect it to, but occasionally there are edge cases that lack documentation, public announcements, or both.
+
+It's possible that at some point Fly.io announced a breaking change and I missed it, but the behavior I observed deserves more than an announcement or silently released documentation.
+
+Today, I wanted to perform a manual leader failover to one of my cluster's followers, but it quickly failed in a way that wasn't immediately clear.
+
+## The problem
+
+Performing `flyctl pg failover` can fail for opaque reasons because the underlying error is eaten. 
+
+Example
+```bash
+$ flyctl pg failover -a <APP> --debug
+Performing a failover
+Connecting to fdaa:9:1d32:a7b:94:1aeb:7b94:2... complete
+Stopping current leader...  328725ec309d85
+Starting new leader
+Promoting new leader...  e784126feee248
+Connecting to fdaa:9:1d32:a7b:94:1aeb:7b94:2... complete
+WARNING: unable to connect to remote host "fdaa:9:1d32:a7b:e:e63c:7516:2" via SSH
+WARNING: unable to connect to remote host "fdaa:9:1d32:a7b:2b5:4c5e:f3b8:2" via SSH
+...
+```
+
+Now this isn't entirely opaque -- we can see that _something_ is having a problem connecting to two cluster nodes via SSH. But what is trying to connect via SSH? My local machine, or one of the replicas in the Postgres cluster?
+
+
+If you aren't aware of how Fly Postgres clustering works, under the hood it's simply [repmgr](https://github.com/EnterpriseDB/repmgr), which under _its_ hood uses [passwordless SSH sessions](https://www.repmgr.org/docs/4.4/performing-switchover.html) to orchestrate changes with cluster members.
+
+`fly pg failover` executes `repmgr standby switchover --siblings-follow` (amongst other things) on one of the cluster's follower nodes, which takes over the `primary` role in the cluster.
+
+So the errors above -- those are coming from a _follower_ cluster node attempting connections to other cluster nodes. Failover must fail because without SSH sessions to other cluster nodes, `repmgr` cannot orchestrate any changes. 
+
+> Aside: `fly pg failover` should absolutely accept repmgr's `--dry-run` switch for performing dry runs. Currently, it does not. Typically `fly pg failover` stops the leader machine to make way for the new leader. A `--dry-run` switch should prevent any such service disruption, just as repmgr's dry run behavior prevents service disruption.
+
+Unfortunately, `repmgr` can eat errors from `ssh` and won't show you exactly _why_ some connections to remote hosts aren't possible.
+
+By using SSH directly (`ssh postgres@fdaa:9:1d32:a7b:e:e63c:7516:2`) I quickly spotted the underlying error.
+
+> postgres@fdaa:9:1d32:a7b:e:e63c:7516:2: Permission denied (publickey).
+
+At this point, the problem was clearly a SSH public key problem, but I wasn't aware of the below command that'll quickly fix your cluster members up with working ssh keys.
+
+## The fix
+
+To fix this issue, re-distribute SSH public keys throughout the cluster with
+
+> `fly pg renew-certs -a <APP>`
+
+
+```
+Your SSH certificate(s) have been renewed are set to expire in 36525 day(s)
+Run fly deploy --app <APP> --image docker-hub-mirror.fly.io/flyio/postgres-flex:15.8@sha256:5016ffb34e66eca43d4f9ef2f898c166257bd28bd5095c41d049a5e3be15caf5 to apply the changes!
+```
+
+Don't forget to re-deploy your app after renewing certificates.
+