Introduction

This appendix provides the reader with sufficient information to begin using Ansible and Amazon S3 for automated software management.

Software depot

The Attestant software depot holds binaries for Attestant's internal infrastructure. These are specifically binaries that are not available to the public, and carry out operations such as report generation, client interaction, and internal monitoring.

Ansible is able to use the depot to install and verify softwares.

Location

The Attestant software depot is an Amazon S3 bucket at depot.attestant.io and has the following structure:

  • depot.attestant.io the bucket

    • bin the directory containing all of the binaries

      • <name>-<version> the specific binary

For example, version 1.2.0 of the Attestant binary financialsd can be found at s3://depot.attestant.io/bin/financialsd-1.2.0

Writing files to the depot

Writing files to the depot requires specific Amazon S3 user credentials. Only this user is allowed to upload files to the depot.

Reading files from the depot

The depot is configured to allow read access to binaries through use of bearer token authorisation. Specifically, a request using a URL providing the token in the Referer header can retrieve files. This provides a sensible balance between not allowing unauthorised access and allowing automated deployment of software.

Configuration management

Attestant uses Ansible for configuration management. Ansible uses remote connections to servers to carry out tasks, which removes the requirement (and security risk) for every Attestant server to have a software agent listening for connections.

Ansible groups tasks, such as creating a user or installing a piece of software, in bundles called "roles". Attestant has roles for all major components, as well as some for system configuration.

Ansible provides arbitrary grouping of hosts. A single host may be in the groups "geth", "vouch" and "prysm", for example. This allows for significant flexibility when it comes to defining which software should be operating on which servers.

Installing Ansible

Ansible should be installed on the client to allow playbooks to run. Typically this is done using an appropriate package manager (e.g. Homebrew for Mac, or apt for Linux):

brew install ansible

sudo apt-get install ansible

Clone the /ops repository

Ansible runs on client computers, connecting to the servers via SSH. The Ansible configurations are available in the Attestant /ops GitHub repository. To obtain this run from ~/attestant :

git clone https://github.com/attestantio/ops.git

Inside this repository is an ansible directory, containing all Ansible configuration information.

Do the following to install additional dependencies in the /ops repo after cloning:

cd ops/ansible

./install-dependencies.sh

Please refer to the "Software Development Lifecycle Procedure" for information about how changes are made to this repository.

Base inventory layout

The inventory defines which hosts are in which groups, and the variables that apply to them. The base inventory layout is as follows:

  • inventories/

    • development/

      • hosts

      • group_vars/

        • all

          • main.yml

          • vault.yml

      • host_vars/

    • operations/

      • hosts

      • group_vars/

        • all

          • main.yml

          • vault.yml

      • host_vars/

    • pod1/

      • hosts

      • group_vars/

        • all

          • main.yml

          • vault.yml

      • host_vars/

    • pod2/
      ...

At the top level there are directories named development and pod2. These ensure that the two environments are kept as separate as possible, reducing the chance of configuration mistakes.

Underneath each of these levels are three items: hosts, which is the list of hosts and their groups, group_vars, which is a directory containing group-specific variables, and host_vars, which is a directory containing host-specific variables. The use of both group-specific and host-specific variables allows per-server customisation if required without needing to create an individual configuration for every host.

The hosts file will look something like:

all:  
  hosts:  
    eth-val-p01-01.attestant.io:  
    eth-val-p01-02.attestant.io:  
    eth-val-p01-03.attestant.io:  
    client-eu01.attestant.io:  
    client-eu02.attestant.io:  
  
children:

  financialsd:
    hosts:
      client-eu01.attestant.io:

  fxd:
    hosts:
      client-eu01.attestant.io:
      client-eu02.attestant.io:

  vouch:
    hosts:
      eth-val-p01-01.attestant.io:
      eth-val-p01-02.attestant.io:
      eth-val-p01-03.attestant.io:

Base role layout

The role defines what software and configuration the hosts will have applied to them. The base role layout is /ops/ansible/roles/ with some examples of being:

  • ssh/files - Location of SSH public keys for authentication
  • geth/tasks/main.yml - Task list to install the geth application along with its version and checksum
  • ufw/vars/debian.yml - Variables for the ufw firewall application on Debian

Local SSH configuration for Ansible

Ansible requires an SSH connection from the client running Ansible (staff member's laptop/desktop) to the servers it will be updating. Because Attestant uses hardware-based public keys to access servers some setup is required to provide Ansible with suitable connections.

The remote server should have a copy of your public key so that it will allow you to log in. To do so, run:

ssh-copy-id -i ~/.ssh/.pub root@.attestant.io

where must be replaced with the SSH identity file of your hardware key. See "Attestant Server Public Keys" for details on how to create and manage these keys.

Next, a change must be made to the ~/.ssh/config file on the operations client that will be running Ansible. An example file is given below (bold indicates items that require changing for each staff member):

# Ansible
Host *.attestant.io
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/<id>-prd1
  ControlMaster auto
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlPersist 30m
  Port 789
  User root

# Overrides

Host operations-eu01.attestant.io
  User ops  

Host signer-eu01.attestant.io
  IdentityFile ~/.ssh/<id>-signer-eu01-1

where must be replaced with the SSH identity file of your hardware key. Note the use of overrides so that the majority of connections to servers can be made using the minimal command of

ssh .attestant.io

Once this is configured, an initial connection to any Attestant server will result in a persistent connection (for 30 mins) where further connections will not require hardware authentication, allowing Ansible to carry out its required tasks.

To terminate or kill this ssh persistent connection, use:

ssh -O stop .attestant.io

The SSH configuration file can be overridden on the command line as usual if a specific server requires a non-standard approach for some reason.

Tip for Apple Users: See the document “Attestant Apple Laptop Setup Procedure” for how to assign different background colours to terminals based on the domain you are logging into.

Using vault.yml for secrets

Secrets such as passwords and API keys are contained in vault.yml files within the relevant group or host’s directory. The secrets are encrypted; contact the CTO for access to the decryption key.

Note that whenever Ansible is run it will ask for the vault password to decrypt the secrets.

Running Ansible scripts

To run an Ansible script the operations user must be in the ops/ansible directory. From here, they should run the following command:

dev|prd-ansible-playbook -i inventories//hosts .yml --limit .attestant.io

Where is one of development or pod, is one of the roles that can be seen by running ls on the roles directory, and is the host on which to run the script.

For example, to configure fxd on the development server jack-d02.attestant.io the following command would be run:

dev|prd-ansible-playbook -i inventories/development/hosts fxd.yml --limit jack-d02.attestant.io

You only also update a few components on the same line, for example updating Prysm and Teku:

dev|prd-ansible-playbook -i inventories/development/hosts prysm.yml teku.yal --limit jack-d02.attestant.io

It is also possible to do a dry run by using the --check flag to see what would have been updated. Also adding the --verbose flag will give you a lot more detail of the changes that will be applied:

dev|prd-ansible-playbook -i inventories/development/hosts prysm.yml teku.yal --limit jack-d02.attestant.io --check --verbose