✅ Post-baseline actions

This document lists the tasks to complete after a server reaches the Ansible baseline: installing approved applications, validating connectivity (for validators), handling signing servers (special process), and adding the server to monitoring dashboards.

1 — Install Attestant-approved applications

When a host is added to inventories/<pod>/hosts it is typically assigned roles and associated applications. Deploy them with the site playbook:

dev/prd-ansible-playbook -i inventories/<pod>/hosts site.yml --limit <host>.attestant.io

When clients are fully synced and services are healthy, proceed to final checks.

2 — Validating servers: open Dirk access on signers

If the new validating server needs to communicate with signers (Dirk), update Dirk on each signer in the pod so it allows the validator's IP/host through the signer firewall and configuration.

Run the playbook (example):

# Run from ops repo; replace wrapper if required
ansible-playbook -i inventories/<pod>/hosts dirk.yml --limit <signer>.attestant.io

After updating Dirk on each signer, validate connectivity from the validating server:

# From the validating server
nc -vz <signer-host> <dirk-port>
# or use the vouch endpoint test (replace with your service endpoints):
curl -fS http://localhost:port/healthz

Add any successful checks to the ticket and mark the validator for "Final Checks" once sync and connectivity are green.

3 — Signing servers (special process)

⚠️ Signing servers are highly sensitive. Only senior operators with explicit permission should perform replacements or key operations.

All signing-server operations (creating keys, wallets, restoring, rotation) live in the separate Signing Servers document:

  • Signing Servers - Creating Keys & Wallets: https://docs.google.com/document/d/1WPAP6zahegCen7U0HqqJjYFsS7KIEK3GJEpvQbpArXc/edit

If you are not the designated signing-server owner, escalate to the owner and follow the separate runbook.

4 — Add server to Grafana / Prometheus

To ensure observability, update Prometheus scrape configs and dashboards:

  1. Update Prometheus job config or service discovery to include the new host.
  2. Reload Prometheus or push the updated configuration via your automation.
  3. Confirm metrics are present in Grafana and add the host to relevant dashboards.

Reference: Update Prometheus configuration in ops-tools (internal doc): https://docs.google.com/document/d/1eMChMtfbXatSPtIrmF4BwMZTL7UV0-lN6IMATVzsqtc/edit#heading=h.6b7ijhhlcbsw

Example (validate Prometheus target up):

curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.discoveredLabels.instance|contains("<host>"))'

5 — Final checks & mark complete

Checklist before marking server as fully deployed:

  • All assigned applications deployed via site.yml and services are running
  • Consensus & execution clients fully synced (for validators)
  • Dirk connectivity validated for signer<->validator flows (if applicable)
  • Monitoring metrics present and dashboard updated
  • Backups/rotate keys and any post-deploy automated tasks scheduled

When done, mark the server as Final Checks or Ready in the Server Info spreadsheet and add links to logs/PRs/tickets in the change record.

Resources

  • Signing servers runbook (sensitive): https://docs.google.com/document/d/1WPAP6zahegCen7U0HqqJjYFsS7KIEK3GJEpvQbpArXc/edit
  • Prometheus/Grafana update doc: https://docs.google.com/document/d/1eMChMtfbXatSPtIrmF4BwMZTL7UV0-lN6IMATVzsqtc/edit

 


 

↩️ Back to the Server Commissioning main page