🗑️ 01 — Decommissioning servers (runbook)

When a server reaches end-of-life (hardware failure, end-of-support, or replaced by newer hardware) follow this runbook to remove it from our fleet safely and consistently.

Quick checklist

1 — Remove server from Ansible inventory & files

From the ansible repository root, search for references to the host (replace <host> with the hostname):

# fast search using git (preferred when repo is available locally)
git -C ~/ops grep -n "<host>" || true

# fallback recursive grep (exclude .git)
grep -RIn --exclude-dir=.git "<host>" . || true

Common places to update:

• inventories/<pod>/hosts — remove host entry from groups
• inventories/<pod>/group_vars/all/main.yml — remove from pod-wide lists (e.g. dirk_permissions)
• inventories/<pod>/host_vars/<host>/ — remove host variables
• inventories/<pod>/host_files/<host>/ — remove certificate files

After removing references, run the search again to confirm nothing remains.

Delete host-specific directories (careful, irreversible):

rm -rf inventories/<pod>/host_files/<host>.attestant.io
rm -rf inventories/<pod>/host_vars/<host>.attestant.io

Commit your Ansible inventory changes following the normal git workflow (create branch, commit, PR):

git checkout -b decomm/<host>
git add -A
git commit -m "remove <host> from <pod> inventory and host files"
git push --set-upstream origin decomm/<host>
# open PR and add reviewers

2 — Disable monitoring & remove from DNS

Before deleting DNS, remove the server from monitoring and alerting so you don't generate noise.

Prometheus example (validate target removal):

curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.discoveredLabels.instance|contains("<host>"))'

Remove DNS records (A and PTR) via Route53 / provider portal. Verify DNS removal:

dig +short <host>.attestant.io @your-resolver
dig +short -x <ip> @your-resolver

3 — Securely erase disks

Warning: these steps are destructive and irreversible. Confirm you are operating on the correct device(s) before running any format/erase commands.

Prefer provider-side secure-erase tooling where available (most providers support a secure wipe during repurpose). If you must wipe disks yourself, prefer NVMe crypto-erase using nvme-cli for SSDs and the vendor-recommended secure-erase for HDD.

Install nvme-cli (Debian/Ubuntu):

sudo apt update && sudo apt install -y nvme-cli
nvme list

Check controller capabilities (look for fna field; 0x4 indicates crypto erase supported):

nvme id-ctrl /dev/nvme0 | grep fna
# example output: fna : 0x4

If the drive supports crypto-erase, run a crypto-erase (quick):

sudo nvme format /dev/nvme0 -n 0xffffffff --ses=2

If the drive does not support crypto-erase, use the vendor's recommended method (example uses --ses=1 as an alternative and --force if needed):

sudo nvme format /dev/nvme0 -n 0xffffffff --ses=1 --force

Notes on NVMe secure-erase:

• Crypto-erase typically replaces the media encryption key in the controller, instantly rendering data unrecoverable.
• The operation may make the running OS unstable; a reboot or provider re-image is expected.
• Confirm success with nvme list or nvme id-ctrl /dev/nvme0 showing usage reset.

For spinning disks (HDD) use vendor-appropriate secure erase (e.g. hdparm --security-erase) or the provider's secure-wipe feature. Example (careful):

# Example only — confirm drive and vendor docs before using
sudo hdparm --user-master u --security-set-pass p /dev/sdX
sudo hdparm --user-master u --security-erase p /dev/sdX

If in doubt, ask the data centre to perform the wipe or use their portal to securely erase before returning hardware.

4 — Reinstall OS / hand back to provider

After secure erase, use the data centre dashboard to reinstall a vanilla operating system image or follow the provider's return workflow.

Steps:

1. Use DC console to re-image to the vendor/vanilla image.
2. Verify that the server boots to the installer or base image.
3. Confirm disks show zeroed/empty usage (e.g., lsblk, nvme list outputs show 0.00 B usage for NVMe where applicable).

5 — Finalize records

• Update the Server Info spreadsheet: mark server as Decommissioned and record the date and ticket/PR reference.
• Close any monitoring alerts and remove escalation policies tied to the host.
• Ensure the Ansible PR is merged (or the changes are otherwise recorded) and that the runbook ticket contains links to the PR and DC wipe logs.

Resources & references

• Ansible inventory patterns: inventories/<pod>/hosts, inventories/<pod>/host_vars/, inventories/<pod>/host_files/
• Prometheus target API: http://prometheus:9090/api/v1/targets
• nvme-cli man page: man nvme-format
• Provider console docs (follow your DC vendor guidance for secure wipe and returns)

↩️ Back to the Server Commissioning main page