Signing servers

Due to the sensitivity of Signing servers and the potential impact of any misconfiguration, the replacement of a Signing server should not be attempted by anyone who does not have a deep understanding of the Attestant method of managing Private Keys and Distributed Wallets.

Overall process

When the Dirk signing pool is first created it requires all 5 servers to be present before keys can be generated. This is to ensure that a rogue server cannot enter the pool later and disrupt the composite private key threshold.

The overall process for setting up Dirk production keys, including a dry run, is as follows:

  1. The coordinator initiates a video call with all primary owners - although this can be skipped if not on Production.

  2. The primary owners manage the creation of private keys on the servers they control.

  3. The coordinator uses ethdo on the client server to create distributed wallets in batches and then distributed accounts.

  4. The primary owners each make an encrypted backup with a passphrase known only to them of the Dirk key directory.

  5. The primary owners verify that their backup works.

  6. The primary owners make a second encrypted backup with a passphrase distributed to 3 other primary owners of signing servers using Shamir Secret Sharing. These owners are selected using the table in the "Business Continuity Plan".

  7. Verify that the second backup works.

Generate the wallets holding the private keys

Use the following script to generate 40 wallets. This should be run by the primary owners on their signer machines.

for wallet in $(seq 1 40); do 
    id=$(printf '%02d' $wallet)
    ethdo --base-dir=/home/dirk/wallets \
      wallet create \
       --type=distributed \
       --wallet=BatchXY${id}
done

Remember to edit the above for the pod you're working on. For example BatchXY=Batch04 for pod5, Batch05 for pod6 and so on. Note that the convention NetworkXY is used for networks other than production, e.g. Prater01

Ownership of the wallets should be set to dirk:

    chown -R dirk /home/dirk/wallets

Verify the Batch number is correct for the pod as errors can be hard to undo later:

    ethdo --base-dir=/home/dirk/wallets wallet list

If an error has been made, the following script can be used to undo the wallet generation:

   for wallet in $(seq 1 40); do
        id=$(printf '%02d' $wallet)
        ethdo --base-dir=/home/dirk/wallets \
          wallet delete --wallet=BatchXY${id}
   done

The final step is to restart the dirk instance on each server.

At this point the "Signer Info" spreadsheet should be updated with a note in the Data Flow Status column to indicate that this server has wallets in place, e.g. "Wallets generated."

Generate the private keys

Use the following script to generate 400 private keys in accounts held by Dirk. This should be run by the coordinator on the client-eu01 machine, separate from the signing servers. Due to the time taken to generate private keys in a distributed manner it may take a few hours to complete this step.

The reason that private keys are generated in batches is a combination of performance and segregation. First, having all private keys in a single wallet would mean that Dirk would need to search through a large number of entries every time a signature had to be made. Over time this would become unwieldy. Second, splitting keys across multiple wallets allows for easier backup and restore processes and the possibility of allowing a customer to be placed into a particular wallet.

If you are only considering a dry run it might be worth reducing the key count to 10 or so. Also, if you are running in a development environment you may find the certificates are named in line with the network, e.g. client-prater.attestant.io.key.

    su ops  
    #
    #
    for i in $(seq 1 40); do
        wallet=`printf '%02d' $i` 
        for j in $(seq 1 400); do
          id=`printf '%05d' $j`
          echo ${wallet}/${id}
          ethdo account create \
            --remote=signer-eu01.attestant.io:12381 \
            --server-ca-cert ${HOME}/certs/ca.crt \
            --client-cert ${HOME}/certs/client.attestant.io.crt \
            --client-key ${HOME}/certs/client.attestant.io.key \
            --account=BatchXY${wallet}/${id} \
            --signing-threshold=3 \
            --participants=5
        done
    done

In general unassigned keys are attached to customer's validators during the on-boarding process either as a single operation (via the client website) or as part of a larger batch where multiple keys can be assigned or made available to Dirk. The acli batch activate command provides more details.

Common errors are:

  • Failure to generate keys - check that dirk is the owner of the /home/dirk/wallets directory

  • Failure to find accounts - check that dirk has been restarted on each signing server after wallet generation

Confirm the validator keys are present on every signer

Obtain the accounts from each signer individually, counting them to confirm that the expected number of accounts are present:

    ethdo wallet accounts \
        --remote=signer-eu01.attestant.io:12381 \
        --server-ca-cert ${HOME}/certs/ca.crt \
        --client-cert ${HOME}/certs/client.attestant.io.crt \
        --client-key ${HOME}/certs/client.attestant.io.key \
        --wallet=Batch0001 | wc -l

This should return 400 (being the number of accounts created in the previous step). Repeat this command for each signer (i.e. change signer-eu01 to signer-eu02, then signer-eu03, signer-eu04 and finally signer-eu05).

Verify that the private keys are working

Create a signature, selecting one of the created accounts at random (in this example, Batch0002/00300):

    ethdo signature sign \
        --remote=signer-eu01.attestant.io:12381 \
        --server-ca-cert ${HOME}/certs/ca.crt \
        --client-cert ${HOME}/certs/client.attestant.io.crt \
        --client-key ${HOME}/certs/client.attestant.io.key \
        --account=BatchXY02/00300 \
        --data=0x000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f \
        --domain=0x202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f

Note that the data field is just a random block of 32 bytes that has no larger meaning. The domain field is another random block of 32 bytes that is used to provide unique signatures for the same data across logically separated instances (for example, across a production and test blockchain). The output should show the signature for the public key referenced in the account.

Obtain the composite public key of the selected account:

    ethdo account info \
        --remote=signer-eu01.attestant.io:12381 \
        --server-ca-cert ${HOME}/certs/ca.crt \
        --client-cert ${HOME}/certs/client.attestant.io.crt \
        --client-key ${HOME}/certs/client.attestant.io.key \
        --account=BatchXY02/00300

The output should show a composite public key with a signing threshold of 3 out of 5.

Verify the signature created in the first command with the composite public key obtained in the second command (replace the entries in bold as required):

ethdo signature verify \
    --data=0x000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f \
    --domain=202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f \
    --signature=0xb3d1e5c72ebc6d6dc06555201c632bbbf9262456f4fe305a8b47de36f472d10709783e0f6d22e4c1c3f4c6a7788deac519519bfb18f0a3f8c198abea7e209b9ffcee3994e81fcf7e67ba702c1814791eafac786b6c01e37287a0c53f00ccb3e1 \
    --signer=0xa3d34ad2746a4ab02ee8eed60ae6a9a4ea150b3ff404a0bd804b7d229c684d6cc7678ca4cfe911cb054f44a796bd3490 \
    --verbose

This should report "Verified". This shows that the signature matches the private key.

Backup the private keys

At this point each Dirk instance will have a collection of accounts within their local wallets. Each wallet needs to be backed up by both its primary and secondary owners. The backup for the primary owner should be carried out as follows:

  1. Primary owner (PO) logs into their signing server as root.

  2. PO verifies that the version of ethdo is 1.7.0+ copying in an appropriate version if needed (ethdo version)

  3. PO generates a strong passphrase 24+ mixed case alphanumeric (pwgen -B 24) and stores it safely for later use.

  4. While unobserved (no screen sharing) PO creates an environment variable and sets it to the passphrase using a leading space to hide this command from the history:

  PASSPHRASE=<passphrase>
  clear

  1. Once the screen is cleared, the PO can permit screen sharing

  2. PO makes a backup of the first wallet (changing the sequence start/end as directed by the coordinator for the pod):

    for i in $(seq 501 540); do 
        id=$(printf '%04d' $i); 
        ethdo wallet export \
        --base-dir=/home/dirk/wallets \
        --passphrase=${PASSPHRASE} \
        --wallet=Batch${id} > $(hostname)-Batch${id}-backup.dat
    done

Remember to update the index values appropriately for the pod (401-440 for pod 5, 501-540 for pod 6 and so on).

  1. The contents of the backup should be verified (adjusting the sequence start/end as before):
    for i in $(seq 501 540); do
        id=$(printf '%04d' $i); 
        ethdo wallet import \
            --verify \
            --passphrase=${PASSPHRASE} \
            --data=$(hostname)-Batch${id}-backup.dat
    done

The output from the above should be similar to:

Wallet name: Batch0502

Wallet type: distributed

Wallet UUID: 12341234-1234-1234-1234-12341234

Wallet accounts: 400

The Wallet UUID can be verified manually using ls -la /home/dirk/wallets.

  1. The PO will unset the passphrase to ensure that they have a working copy later.

unset PASSPHRASE

Delete the private keys

It is essential to have total confidence in the backup process using disposable data (or data that has minimal value).

  1. The PO will delete all the generated keys from their server leaving only the backups.
  for i in $(seq 501 540); do 
    batch=Batch$(printf '%04d' $i)
    ethdo --base-dir=/home/dirk/wallets wallet delete --wallet=${batch}
  done

  systemctl restart attestant-dirk
  1. The PO will verify that the wallet has been deleted (using the earlier UUID):

ls -la /home/dirk/wallets

  1. The other POs will delete their wallets so that 2 out of 5 signing servers do not have the keys.

  2. The POs who have deleted their wallets will restart and verify their dirk instances using:

systemctl restart attestant-dirk.service systemctl status attestant-dirk.service

  1. The coordinator will verify that signatures are still operational from the client-eu02 machine using the signature checking procedure earlier (3 out of 5 continues to run).

  2. A third PO will delete all the generated keys from their server so that 3 out of 5 signing servers do not have the keys.

  3. The coordinator will verify that signatures are now broken from the client-eu01 machine using the signature checking procedure earlier.

  4. Each PO makes a local copy of their backup:

scp -P 789 'root@signer-eu03.attestant.io:/root/*-backup.dat'

At this point the signing servers should be in a state where 3 out of 5 (or more) do not have their wallets in place. The backup file should be left on the machine as an additional backup location and for the secondary owner to use later in the process.

Restore the private keys

The primary owner (PO) should be able to restore all the private keys for their server using the encrypted backup as follows:

  1. Primary owner (PO) logs into their signing server as root (if needed).

  2. PO verifies that the version of ethdo is 1.7.0+ copying in an appropriate version if needed.

  3. PO retrieves their backup passphrase.

  4. While unobserved (no screen sharing) PO creates an environment variable and sets it to their recorded passphrase using a leading space to hide this command from the history:

PASSPHRASE= clear

  1. Once the screen is cleared, the PO can permit screen sharing.

  2. PO uploads their local copy of the encrypted backup archives to their server:

scp -P 789 pwd/signer-eu03-Batch0002-backup.tar.gz root@signer-eu03.attestant.io:/root/

  1. Decrypt and import the backup archives (adjusting the values and Batch appropriately):
  for i in $(seq 501 540); do 
    batch=Batch$(printf '%04d' $i); 
    ethdo --base-dir=/home/dirk/wallets wallet import \
        --passphrase=$PASSPHRASE \
        --data=/root/$(hostname)-$batch-backup.dat
done
  1. Ensure dirk owns the wallet data

chown -R dirk /home/dirk/wallets

  1. One PO should restart and verify their dirk instance:

systemctl restart attestant.service systemctl status attestant-dirk.service

  1. The coordinator should verify that 3 out of 5 signing servers are online and successfully signed a transaction.

  2. The remaining POs should now restart and verify their dirk instances:

systemctl restart attestant.service systemctl status attestant-dirk.service

  1. The coordinator will verify that all the Dirk instances are now participating in the signer pool.

  2. The PO will assist the secondary owner for the server (defined in the "Business Continuity Plan" document) in logging on to the machine using a screen share private to just the PO and the secondary owner.

  3. The secondary owner will verify the contents of the backups:

for i in $(seq 501 540); do 
  batch=Batch$(printf '%04d' $i); 
  ethdo --base-dir=/home/dirk/wallets wallet import \
    --verify \
    --passphrase=$PASSPHRASE \
    --data=/root/$(hostname)-$batch-backup.dat; 
done
  1. The secondary owner will view the verified passphrase (known to thePO) and make a local copy in a secure setting (e.g. encrypted key manager):

echo $PASSPHRASE

  1. The secondary owner will make a local copy of the encrypted backup using:

scp -P 789 root@signer-eu03.attestant.io:/root/*-backup.dat .

Note that shells such as zsh may require /root/*-backup.dat to escape the wildcard.

  1. The PO will now ensure that their passphrase is removed from the server environment:

unset PASSPHRASE

  1. The PO will close the shared screen session with the secondary owner.

Verify threshold signatures are working

To ensure that the pod is working properly, the final check is to engineer a situation where various combinations of threshold and below threshold signatures are taking place. In general the procedure for this is as follows:

  1. The coordinator will arrange for the POs for all signing servers to be present via a video call.

  2. The coordinator will ask the POs of signers 2 and 4 within the pod to shut down their dirk instances (e.g. signer-eu02, signer-eu04 in pod 1 or signer-eu17 and signer-eu19 in pod 4)

  3. The coordinator will run the following script on client-eu01 (or equivalent operations server) to confirm that signatures can be generated:

for i in $(seq 16 20); do 
    echo -n "${i}: "
    server=$(printf '%02d' $i)
    ethdo signature sign \
        --remote=signer-eu${server}.attestant.io:12381 \
        --server-ca-cert ${HOME}/certs/ca.crt \
        --client-cert ${HOME}/certs/<host>.attestant.io.crt \
        --client-key ${HOME}/certs/<host>.attestant.io.key \
        --account=Batch0308/00275 \
        --data=0x000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f \
        --domain=0x202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f
done

Note that the values in bold will need to be adjusted for the specific situation of the pod under test (e.g. 21-25 for pod 5)..

The output should show that successful signatures were received from signers 1, 3 and 5 within the pod but 2 and 4 failed.

  1. The coordinator will now ask for the PO of signer 3 to shut down dirk and re-run the script. The output should show no successful signatures as the participation of 3 is the minimum threshold to achieve a signature.

  2. The coordinator will now ask for dirk to be shut down on signer 5 and for signers 2 and 4 to be started.

  3. The coordinator will run the script and confirm that signatures are once again generated using signers 1, 2 and 4.

  4. The coordinator will now ask for dirk to be started on signers 3 and 5 and will rerun the script. The output should show signatures from all 5 signers and act as the final proof that all signers are communicating with each other and that the threshold signature is working correctly.

At this point the server should be marked as "Final Checks" in the "Server Info" spreadsheet to indicate that only final system testing is required before the pod can go live.

Create a live validator

As a final verification that the pod has been configured correctly, a live validator should be created as follows:

  1. The coordinator will manually prepare the attestantd configuration file (attestantd.yml) on the client API server to register the new pod so that the correct signers will be used.

  2. The PO providing the validator will use their client API to request a new validator

  curl -X POST \
   -H 'Content-type: application/json' \
   -H 'Accept: application/json' \
   -H "Authorization: Bearer xxxxxx" \
   -d '{"amount": "32000000000", \
        "withdrawal_eth1_address": "xxxxxx", \
        "name":"Validator 4","pod":"pod-4"}' \
    https://client.attestant.io/v1/eth2/validators  | jq

  1. The coordinator will update the operations database so that Batch0X01/00001 (the first validator) is assigned to the validator (t_unassigned_validators):

  2. The PO will deposit ETH to initiate the validator.

  3. The coordinator will verify /var/log/attestantd/attestantd.log and the database audit log (t_audit) to confirm that the validator has been assigned correctly.

  4. Given the nature of the Ethereum deposit queue it may take some time for the validator to start and any errors at that point should be reported.

Final steps to add and activate new wallets in vouch

The final step is to add the new wallets to the appropriate pod in ansible. Edit the file located at ops/ansible/inventories/podX/group_vars/all/main.yml for the correct pod, and add the wallets under the vouch_accountmanager_accounts field. This change must then be rolled out to all validator boxes in the pod so that vouch can recognize the new wallets.

After adding the wallets, the new keys need to be added to the unallocated validators table, making them available for assignment. This can be done using the following acli batch activate command:

    acli --remote=signer-euXX.attestant.io:12381 \
         --ca-cert ${HOME}/certs/ca.crt \
         --client-cert ${HOME}/certs/client.attestant.io.crt \
         --client-key ${HOME}/certs/client.attestant.io.key \
         batch activate \
           --network=Mainnet \
           --batch=Batch0XXX \
           --pod=pod-X