DevOps Integration for new apps

In order to onboard a new application to the DevOps stack there are two components:

  • Ansible setup - we need to codify they deployment and running operation of the application.
  • Monitoring setup - we need to integrate the service with the monitoring stack so that the Operations team can maintain it in production.

Ansible setup

Repository: https://github.com/attestantio/ops

In the ansible directory:

  • Inventories - relates to hosts and groups of hosts (pod)
  • Roles - software definitions

Inventories

Relationship:

  • a host has many roles (software elements) and host vars and host files.
  • a pod has many hosts and group vars the pod/hosts file defines which software (role) is running on which hosts.

Pod structure:

  • the group_vars defines variables for all hosts in that group (constant for all hosts
  • the host_files defines files specific to that host (not shared)
  • the host_vars defines variables specific to that host

Role

4 folders:

  • defaults - these are essentially app constants
  • handlers - where we define ansible handlers, this is typically a restart handler. A handler is something that is notified by another task or tasks and run once. Eg. if 3 tasks notify the same handler, the handler will be run once at the end.
  • tasks - this is where we define the ansible tasks. They do things like:
    • open firewalls
    • copy files
    • create directories
    • copy templates while injecting ansible variables in to them
    • create systemctl unit files
    • download software
  • templates - place where templates are stored

NB: We do not build software in the install steps. If the software needed does not have a binary/release distribution then we should build the required binary and upload it to S3 via the build server. See here for more details

App structure

  • all app related data/bins is stored in /home/<app_name>, and the app_name is the role_name
  • we tend to run all apps using systemctl and we create service unit files, e.g. /etc/systemd/system/attestant-vouch.service. We have a convention of prefixing the service name with attestant.
  • we log all logs for an app to /var/log/<app_name>/<app_name>.log, e.g. /var/log/vouch/vouch.log`

Running ansible

An example command that limits the host and app (role).

dev-ansible-playbook -i inventories/development-pod4-hoodi/hosts --limit eth-val-d04-01.attestant.io near.yaml

Monitoring setup

  • Need to open firewalls (see tasks called configure ufw for prometheus)
  • Ensure promgen is aware of the port (repository: https://github.com/attestantio/ops-tools)
  • Need to add alerts to ansible/roles/prometheus/files/rules in ops.
  • Need to run ansible on prometheus instance