SlashDB_Testnet

Highly available PostgreSQL deployment slashdb-testnet0* nodes are in patroni cluster

Maintenance

Access boxes

Get token:

vault login -method=oidc -address=https://vault-a.ts.chorus1.net

Setup token:

$ cd chorus-infrastructure # This is the github repo with IaC.
$ ./tools/ssh/gen-cert.py --root # This is the command to get a root key.

SSH to the appropriate box:

ssh root@slashdb-testnet01

Patroni cluster operations

We are free to do maintenance on replica's, but failover first before starting any maintenance on Leader

Check current master and replica

/venv/bin/patronictl -c /etc/postgresql/patroni.yml list

root@slashdb-testnet02:~# /venv/bin/patronictl -c /etc/postgresql/patroni.yml list
+ Cluster: slashdb_testnet (7540664946901117766) ----------+----+-----------+------------------+
| Member            | Host           | Role    | State     | TL | Lag in MB | Tags             |
+-------------------+----------------+---------+-----------+----+-----------+------------------+
| slashdb-testnet01 | 100.72.19.24   | Replica | streaming | 37 |         0 | nofailover: true |
| slashdb-testnet02 | 100.84.152.120 | Leader  | running   | 37 |           |                  |
| slashdb-testnet03 | 100.95.73.120  | Replica | streaming | 37 |         0 | nofailover: true |
+-------------------+----------------+---------+-----------+----+-----------+------------------+

Failover procedure(example to failover from slashdb-testnet02 to slashdb-testnet03)

Login as root to both servers slashdb-testnet03 and slashdb-testnet02 Check current status of the cluster from any host and ensure Lag in MB is 0

/venv/bin/patronictl -c /etc/postgresql/patroni.yml list

you will see the Tag 'nofailover: true' for replica's

root@slashdb-testnet02:~# /venv/bin/patronictl -c /etc/postgresql/patroni.yml list
+ Cluster: slashdb_testnet (7540664946901117766) ----------+----+-----------+------------------+
| Member            | Host           | Role    | State     | TL | Lag in MB | Tags             |
+-------------------+----------------+---------+-----------+----+-----------+------------------+
| slashdb-testnet01 | 100.72.19.24   | Replica | streaming | 37 |         0 | nofailover: true |
| slashdb-testnet02 | 100.84.152.120 | Leader  | running   | 37 |           |                  |
| slashdb-testnet03 | 100.95.73.120  | Replica | streaming | 37 |         0 | nofailover: true |
+-------------------+----------------+---------+-----------+----+-----------+------------------+

Update config on slashdb-testnet03 to be able to to take on the "Leader" role.

2.1 Edit /etc/postgresql/patroni.yml and change the nofailover tag

nofailover: True 
to 
nofailover: false

2.2 In our example currently slashdb-testnet02 is Leader, so reload the config for slashdb-testnet03 from slashdb-testnet02, the leader.

patronictl -c /etc/postgresql/patroni.yml reload slashdb_testnet slashdb-testnet02

Now you will see the Tag 'nofailover: true' is blank for slashdb-testnet03 too

   root@slashdb-testnet02:~# /venv/bin/patronictl -c /etc/postgresql/patroni.yml list
   + Cluster: slashdb_testnet (7540664946901117766) ----------+----+-----------+------------------+
   | Member            | Host           | Role    | State     | TL | Lag in MB | Tags             |
   +-------------------+----------------+---------+-----------+----+-----------+------------------+
   | slashdb-testnet01 | 100.72.19.24   | Replica | streaming | 37 |         0 | nofailover: true |
   | slashdb-testnet02 | 100.84.152.120 | Leader  | running   | 37 |           |                  |
   | slashdb-testnet03 | 100.95.73.120  | Replica | streaming | 37 |         0 |                  |
   +-------------------+----------------+---------+-----------+----+-----------+------------------+

Ask the leader to evict the leadership role.

run on current Leader slashdb-testnet02:

   patronictl -c /etc/postgresql/patroni.yml switchover

wait 10-15 seconds and check that slashdb-testnet02 is new leader now

you will see the Role column changed and

   root@slashdb-testnet02:~# /venv/bin/patronictl -c /etc/postgresql/patroni.yml list
   + Cluster: slashdb_testnet (7540664946901117766) ----------+----+-----------+------------------+
   | Member            | Host           | Role    | State     | TL | Lag in MB | Tags             |
   +-------------------+----------------+---------+-----------+----+-----------+------------------+
   | slashdb-testnet01 | 100.72.19.24   | Replica | streaming | 37 |         0 | nofailover: true |
   | slashdb-testnet02 | 100.84.152.120 | Replica | running   | 37 |         0 |                  |
   | slashdb-testnet03 | 100.95.73.120  | Leader  | streaming | 37 |           |                  |
   +-------------------+----------------+---------+-----------+----+-----------+------------------+

Do the step 2 to make slashdb-testnet02 a replica. Ensure the output looks similar to how it was, ie 2 hosts with 'nofailover: true'

   root@slashdb-testnet01:~# /venv/bin/patronictl -c /etc/postgresql/patroni.yml list
   + Cluster: slashdb_testnet (7540664946901117766) ----------+----+-----------+------------------+
   | Member            | Host           | Role    | State     | TL | Lag in MB | Tags             |
   +-------------------+----------------+---------+-----------+----+-----------+------------------+
   | slashdb-testnet01 | 100.72.19.24   | Replica | streaming | 37 |         0 | nofailover: true |
   | slashdb-testnet02 | 100.84.152.120 | Replica | streaming | 37 |         0 | nofailover: true |
   | slashdb-testnet03 | 100.95.73.120  | Leader  | running   | 37 |           |                  |
   +-------------------+----------------+---------+-----------+----+-----------+------------------+

Update the IP of the new leader

5.1. Find the current Patroni leader IP

/venv/bin/patronictl -c /etc/postgresql/patroni.yml list

5.2. Update pgbouncer configmap in github(Example PR 28861)

vi kubernetes/components/evm/pgbouncer/hoodi/configmap.yaml
Change: * = host=<old_ip>
To:     * = host=<new_leader_ip>

5.3. Apply to cluster

kubectl --context ts-k8s-operator-prod01.atlas-pierce.ts.net \
  -n eth-hoodi-chorusone apply \
  -f kubernetes/components/evm/pgbouncer/hoodi/configmap.yaml

5.4. Restart pgbouncer across all hoodi namespaces

for ns in eth-hoodi-chorusone eth-hoodi-firi eth-hoodi-renzo eth-hoodi-hextrust; do
  kubectl --context ts-k8s-operator-prod01.atlas-pierce.ts.net \
    -n $ns rollout restart statefulset/pgbouncer-hoodi
done

5.5. Confirm pgbouncer pods are up

for ns in eth-hoodi-chorusone eth-hoodi-firi eth-hoodi-renzo eth-hoodi-hextrust; do
  echo "=== $ns ==="
  kubectl --context ts-k8s-operator-prod01.atlas-pierce.ts.net \
    -n $ns get pods -l app.kubernetes.io/name=pgbouncer
done

Connected services:

pgbouncer
web3signer