EigenDA

Project documentation

https://docs.eigenda.xyz/operator-guides/overview

Monitoring

We have an EigenDA V2 dashboard located here for production.

Logs are in docker can be access with docker logs -f eigenda-native-node.

Maintenance

There are two containers that are running as part of this setup:

  • eigenda-native-node - the eigenda AVS software.
  • bin-reverse-proxy-1 - this is an nginx sidecar container that is used to provide rate limiting and request size limit protections.

The hyperlane validator is deployed via docker and controlled by the usual docker commands, e.g. docker start eigenda-native-node.

The docker containers are managed by a docker compose file stored in ansible. Configuration of the eigenda node and nginx sidecar are controlled via an environment file stored in ansible and installed at /home/docker/eigenda/bin/.env. The per environment values are injected in to these files via ansible.

There is a supporting script that is used for manual interaction with eigenda, located in /home/docker/eigenda/bin/run.sh. This script has various commands that can be performed:

  • ./run.sh list-quorums - this simply lists the quorums that the operator is registered to.
  • ./run.sh opt-in 0,1 - this attempts to opt in quorums supplied, e.g. the example is opting in to 0 and 1.
  • ./run.sh opt-out 0,1 - this attempts to opt out of all quorums that are supplied, e.g. the example is opting out of quorum 0 and 1.
  • ./run.sh update-socket - this updates the node ip and ports that are registered on-chain. We run this automatically on rollout as, when switching between hosts, the values can be cached.

SLAs

There are various SLAs defined in the eigen docs. The SLAs vary depending on the % of stake we have in any given quorum. If we drop below the SLAs then we can be ejected out of quorum for a cooldown period - currently 3 days.

Alerts

Alert NameDescriptionAction
EigenDA not in any quorumsWe are not part of any quorum when we should be.Try running /home/eigenda/bin/opt_in.sh to opt in to quorums.
EigenDA signing rate has dropped below 90%The signing rate is below all SLA.This indicates a problem with the node. Check the logs, resources and network connectivity.
EigenDA dispersal is offlineThe node dispersal port cannot be contacted.Check the node is running and that the dispersal port (typically 32005) can be reached.
EigenDA retrieval is offlineThe node retrieval port cannot be contacted.Check the node is running and that the retrieval port (typically 32004 external, 32014 internal) is responsive. This port is routed to the node via trafaek so check logs on trafaek also.
EigenDA latency is too highThe 95th percentile execution time for operations is over 500ms.This indicates a problem with the node. Check the logs, resources and network connectivity.
EigenDA request rate is highThe RPS for inbound requests has exceed 0.1 rps.This indicates an unexpectedly high inbound request rate. Need to investigate why and if this is expected. trafaek should rate limit request to the node to 10rps with a burst of 50.