Lido Oracle — Chorus One Knowledge Base

Audience: New Chorus One team members with basic Kubernetes/DevOps familiarity.
Last updated: July 2026 · Oracle v8.0.5 · KAPI 4.0.1

Table of Contents

• 1. What Is the Lido Oracle?
• 2. Full Stack Architecture
• 3. Component Deep Dive
  • 3.1 Oracle v8 (6 Docker containers)
  • 3.2 KAPI — Lido Keys API
  • 3.3 Validator Exit Stack
• 4. Networks
• 5. Ethereum Node Dependencies
• 6. Configuration Files
• 7. Secrets Architecture
• 8. Deployment
  • 8.1 Deploy / Upgrade Oracle
  • 8.2 Deploy / Upgrade KAPI
  • 8.3 Deploy / Upgrade Validator Ejector
• 9. Monitoring & Health Checks
• 10. Common Issues & Runbook
• 11. Repo Layout
• 12. Key Addresses & IDs

1. What Is the Lido Oracle?

Lido is the largest Ethereum liquid staking protocol. Users deposit ETH and receive stETH (a rebasing token representing their staked ETH + rewards). Lido uses a decentralized oracle committee — a set of trusted node operators — to report on-chain data that the protocol needs to function.

Chorus One is one of those oracle committee members. Our oracle member address is:

0x8dB977C13CAA938BC58464bFD622DF0570564b78

Why does this matter? If our oracle goes offline, we miss reporting windows. Lido monitors this and will page us. Missing too many frames can result in slashing or removal from the committee.

What does the oracle report?

There are four oracle sub-processes, each running as a separate container:

2. Full Stack Architecture

3. Component Deep Dive

3.1 Oracle v8 (6 Docker containers)

Runs on a bare-metal host managed by Ansible. All 6 containers use the same Docker image — the entrypoint command switches behavior.

Image: 176395444877.dkr.ecr.us-east-1.amazonaws.com/chorusone/lido-oracle:<version_tag>
ECR repo: chorusone/lido-oracle (us-east-1, Chorus One AWS account)

Config files on the host:

/etc/docker/compose/ethereum-lido-oracle-v8-mainnet/
├── docker-compose.yml       ← generated by Ansible
└── keys.toml                ← tells fetch_keys.py where to get Vault secrets

Secrets are fetched from Vault at boot and written to ramdisk:

/dev/shm/vault-secrets/ethereum-lido-oracle-v8/mainnet/env-v8

This file contains MEMBER_PRIV_KEY, IPFS tokens, DB password, etc.

Vault path for oracle secrets:

ethereum/lido-oracle/mainnet

3.2 KAPI — Lido Keys API

KAPI is a REST API that indexes all validator keys registered with Lido. The oracle queries it to know which validators belong to each module. Without KAPI, the oracle cannot report.

Runs in Kubernetes (k8s-prod01, default namespace).

Key files:

Vault secret for KAPI:

secret/k8s/default/lido-keys-api-mainnet  →  field: DB_PASSWORD

Vault auth backend: kubernetes (note: Hoodi uses kubernetes-prod01 — different!)

KAPI health check:

kubectl exec -n default deploy/lido-keys-api-mainnet -- curl -s http://127.0.0.1:3000/v1/status
# Should return: {"appVersion":"4.0.1","chainId":1,...}

3.3 Validator Exit Stack

When Lido decides a validator needs to exit (e.g. withdrawal requests exceed buffer), it posts an on-chain exit request. We must respond by submitting a signed voluntary exit to the beacon chain. This is handled by a three-component stack, all running in Kubernetes:

⚠️ Name collision warning: There is also a lido-oracle-v8-ejector Docker container on the oracle host (part of the oracle suite). That is a different thing — it's an oracle process that detects and reports exit requests on-chain. The components below are what actually perform the exits.

Component roles

Full exit flow step by step

Note: Submitting a voluntary exit multiple times is idempotent and carries no penalty. If unsure whether an exit was submitted, it's safe to restart the ejector and let it reprocess.

Key config values (mainnet statefulset patch)

EXECUTION_NODE: http://100.114.206.5:18545   # eth-archive03 (ARCHIVE required)
CONSENSUS_NODE: http://100.114.206.5:15052
LOCATOR_ADDRESS: 0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb
STAKING_MODULE_ID: "1"                        # Curated module
OPERATOR_ID: "3"                              # Chorus One
VALIDATOR_EXIT_WEBHOOK: http://validator-ejector-listener:8080/exit
DRY_RUN: "false"
BLOCKS_PRELOAD: "40000"                       # How many blocks back to scan on startup
BLOCKS_LOOP: "64"                             # Blocks per polling cycle

Archive node required: validator-ejector needs an archive execution node because it scans historical blocks. Do not point it at a pruned node.

Config file locations

kubernetes/base/evm/validator-ejector/
├── statefulset.yaml          ← base: probes, resources, common env vars
├── serviceaccount.yaml
└── kustomization.yaml

kubernetes/clusters/prod01/evm/validator-ejector/mainnet/
├── kustomization.yaml        ← namespace: eth-mainnet-lido
└── statefulset.yaml          ← patch: EL/CL endpoints, operator config, oracle allowlist

4. Networks

We run on both mainnet and Hoodi (testnet). Configuration differences:

Gotcha: Hoodi KAPI uses KUBERNETES_BACKEND=kubernetes-prod01. If you copy the mainnet config, this will be wrong and Vault auth will fail silently.

5. Ethereum Node Dependencies

All oracle components need an execution layer (EL) and consensus layer (CL) node. We use eth-archive03 (Besu v26.7) for mainnet.

Why Besu and not Reth? Reth v2.4.1 has a timing issue where a newly-finalized block is available in the engine API before it is queryable via eth_getBlockByHash. The oracle queries by exact block hash immediately after finalization, hitting this window and receiving -32001: block not found. Besu makes blocks available immediately. We switched in July 2026 when Lido flagged our oracle offline.

6. Configuration Files

Oracle (Ansible group_vars)

provision/ansible/environments/cs/group_vars/
├── ethereum_lido_oracle_v8_mainnet.yaml   ← mainnet config
└── ethereum_lido_oracle_v8_hoodi.yaml     ← hoodi config

Key fields in mainnet:

ethereum_lido_oracle_v8:
  version_tag: "8.0.5"                           # Docker image tag
  execution_endpoint: http://100.114.206.5:18545  # Besu EL
  consensus_endpoint: http://100.114.206.5:15052  # Besu CL
  keys_api_endpoint: http://lido-keys-api-mainnet.default.svc.k8s.prod.chorus1.net:8080
  lido_locator_address: "0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb"
  performance_db_host: "postgres-proxy.default.svc.k8s.prod.chorus1.net"

KAPI (Kustomize overlay)

kubernetes/clusters/prod01/evm/lido-keys-api/mainnet/
├── kustomization.yaml       ← entry point
├── deployment.yaml          ← patch: image + env
└── configmap-secrets.yaml   ← Vault DB_PASSWORD mapping

kubernetes/base/evm/lido-keys-api/
└── deployment.yaml          ← base: DB_HOST, ports, probes, resources

7. Secrets Architecture

Oracle secrets Vault path: ethereum/lido-oracle/mainnet (contains MEMBER_PRIV_KEY, IPFS tokens, DB password)

KAPI secret Vault path: secret/k8s/default/lido-keys-api-mainnet → field DB_PASSWORD

How to read KAPI's DB password manually (for debugging):

vault kv get -field=DB_PASSWORD secret/k8s/default/lido-keys-api-mainnet

8. Deployment

8.1 Deploy / Upgrade Oracle

The oracle runs on a bare-metal host managed by Ansible. To upgrade or restart:

# 1. Update the version_tag in group_vars
vim provision/ansible/environments/cs/group_vars/ethereum_lido_oracle_v8_mainnet.yaml
# Change: version_tag: "8.0.5"

# 2. Build and push the image to ECR (if a custom image is needed)
#    For official lidofinance/oracle images, skip this step

# 3. Run the Ansible playbook
cd provision/ansible
ansible-playbook -i environments/cs base.yaml --tags ethereum_lido_oracle_v8 -l <host>

ECR tags are immutable — you cannot overwrite an existing tag. Always use a new tag.

8.2 Deploy / Upgrade KAPI

KAPI runs in Kubernetes and is managed by kubectl apply -k.

# Edit the image version in the overlay patch
vim kubernetes/clusters/prod01/evm/lido-keys-api/mainnet/deployment.yaml
# Change: image: 176395444877.dkr.ecr.us-east-1.amazonaws.com/chorusone/lido-keys-api:<new-version>

# Apply via kustomize
kubectl apply -k kubernetes/clusters/prod01/evm/lido-keys-api/mainnet/

# Watch rollout
kubectl rollout status deployment/lido-keys-api-mainnet -n default

Preview changes before applying:

kubectl diff -k kubernetes/clusters/prod01/evm/lido-keys-api/mainnet/

⚠️ If you see selector is immutable errors, the deployment must be deleted and recreated:

kubectl delete deployment lido-keys-api-mainnet -n default
kubectl apply -k kubernetes/clusters/prod01/evm/lido-keys-api/mainnet/

8.3 Deploy / Upgrade Validator Ejector

The ejector runs in Kubernetes and is managed by kubectl apply -k, the same as KAPI.

# Edit image version in the base statefulset (or overlay if overridden)
vim kubernetes/base/evm/validator-ejector/statefulset.yaml
# Change: image: 176395444877.dkr.ecr.us-east-1.amazonaws.com/chorusone/lido-validator-ejector:<new-tag>

# Preview changes
kubectl diff -k kubernetes/clusters/prod01/evm/validator-ejector/mainnet/

# Apply
kubectl apply -k kubernetes/clusters/prod01/evm/validator-ejector/mainnet/

# Watch rollout
kubectl rollout status statefulset/validator-ejector -n eth-mainnet-lido

StatefulSet note: Unlike Deployments, StatefulSets do not get an error on selector changes — but they do require kubectl rollout restart to pick up new env var changes if the pod spec hash didn't change:

kubectl rollout restart statefulset/validator-ejector -n eth-mainnet-lido

9. Monitoring & Health Checks

Quick status checks

# Oracle containers (on the oracle host)
docker ps | grep lido-oracle-v8

# Oracle logs (accounting is the most important)
docker logs lido-oracle-v8-accounting-mainnet --tail=50

# KAPI pod
kubectl get pods -n default | grep lido-keys-api-mainnet

# KAPI API status
kubectl exec -n default deploy/lido-keys-api-mainnet -- \
  curl -s http://127.0.0.1:3000/v1/status | jq .

# Validator ejector pods (namespace: eth-mainnet-lido)
kubectl get pods -n eth-mainnet-lido

# Validator ejector logs
kubectl logs -n eth-mainnet-lido statefulset/validator-ejector --tail=50

# Validator ejector listener logs
kubectl logs -n eth-mainnet-lido deploy/validator-ejector-listener --tail=50

# Ethereum key reporter (namespace: default)
kubectl logs -n default deploy/ethereum-key-reporter --tail=50

# Check validator exit status on beacon chain (by pubkey or index)
curl -s https://ethereum-beacon-api.publicnode.com/eth/v1/beacon/states/head/validators/<pubkey_or_index> | jq .

What healthy oracle logs look like

[INFO] Current frame: ref_slot=14875199
[INFO] Waiting for the next finalized slot...
[INFO] Finalized slot: 14875200, block hash: 0x...
[INFO] Fetched data from KAPI

Healthy = cycling every ~12 seconds, ERRORS: 0.

Lido Oracle Dashboard

Lido maintains a public monitoring page. If you get a Telegram alert saying "Bitwise accounting oracle offline", check:

1. Is the accounting container running? docker ps | grep accounting
2. Are there errors in the log? docker logs lido-oracle-v8-accounting-mainnet --tail=100
3. Is KAPI healthy? (see above)
4. Is the EL/CL node reachable? curl http://100.114.206.5:18545

10. Common Issues & Runbook

KAPI CrashLoopBackOff

Oracle "block not found" errors

Symptom: Web3py exception: block not found: hash 0x... in accounting/csm/cm logs.

Cause: The execution node has a lag between the CL finalizing a block and that block being available via RPC (seen with Reth v2.4.1).

Fix: Switch execution_endpoint and consensus_endpoint to Besu (100.114.206.5) in the oracle group_vars, then re-run Ansible.

KAPI getSigningKeys CALL_EXCEPTION (background, non-fatal)

Symptom: CALL_EXCEPTION: getSigningKeys(uint256,uint256,uint256) in KAPI logs.

Cause: The old custom KAPI image (e5433ccdb4f7edc823a6c76e1aeff0cd86b9dc4f) has an ABI incompatible with the current Lido curated module v4 contracts. The official 4.0.1 image fixes this for most cases; some background sync jobs for curated-onchain-v1 may still log this but it does not crash the pod.

Fix: Ensure image is 4.0.1 or newer (official lidofinance/lido-keys-api or our ECR mirror).

Validator ejector not processing exits

First thing to try: Restart both pods. On startup, validator-ejector reloads historical messages and validator-ejector-listener refreshes its key cache.

kubectl rollout restart statefulset/validator-ejector -n eth-mainnet-lido
kubectl rollout restart deploy/validator-ejector-listener -n eth-mainnet-lido

Check if a validator has already exited (safe to resubmit if unsure — idempotent):

# By validator index or pubkey
curl -s https://ethereum-beacon-api.publicnode.com/eth/v1/beacon/states/head/validators/<index> | jq '.data.status'
# "active_exiting" or "exited_unslashed" = exit in progress or done

Check key-reporter cache (validates that the pubkey → Vault path mapping exists):

curl -s http://ethereum-key-reporter.default.svc.k8s.prod.chorus1.net:8000/v1/ethereum/mainnet/lido/validators | jq .

Useful external links:

• Mainnet validator explorer (search by pubkey): https://beaconcha.in/
• Hoodi testnet validator explorer: https://hoodi.beaconcha.in/
• Ethereum key Vault path structure: see Notion → Ethereum keys structure for New Vault

Oracle offline alert from Lido

1. Check oracle containers: docker ps | grep lido-oracle
2. Check logs for errors: docker logs lido-oracle-v8-accounting-mainnet --tail=200
3. Check KAPI is healthy (oracle cannot report without it)
4. Check EL/CL node connectivity
5. If stopped: re-run Ansible playbook to bring containers back

Lido-requested oracle stop (for upgrades)

Lido periodically asks all oracle committee members to stop, update, and restart. Process:

1. docker compose -f /etc/docker/compose/ethereum-lido-oracle-v8-mainnet/docker-compose.yml down
2. Update version_tag in group_vars
3. Re-run Ansible playbook
4. Confirm containers are back up
5. Reply to Lido Telegram that you're online

11. Repo Layout

chorus-infrastructure/
│
├── provision/ansible/
│   ├── environments/cs/group_vars/
│   │   ├── ethereum_lido_oracle_v8_mainnet.yaml   ← oracle mainnet config
│   │   └── ethereum_lido_oracle_v8_hoodi.yaml     ← oracle hoodi config
│   └── roles/
│       └── chorusone.ethereum-lido-oracle-v8/
│           ├── defaults/main.yml                  ← default ports, dirs
│           ├── tasks/main.yml                     ← Ansible tasks
│           └── templates/
│               ├── docker-compose.yml.j2          ← generates compose file
│               └── keys.toml.j2                   ← Vault secret fetch config
│
└── kubernetes/
    ├── base/evm/lido-keys-api/
    │   ├── deployment.yaml                        ← base: DB_HOST, ports, probes
    │   ├── service.yaml
    │   ├── serviceaccount.yaml
    │   └── clusterrolebinding.yaml
    └── clusters/prod01/evm/
        ├── lido-keys-api/
        │   ├── mainnet/
        │   │   ├── kustomization.yaml
        │   │   ├── deployment.yaml                ← patch: image + env
        │   │   └── configmap-secrets.yaml         ← Vault DB_PASSWORD
        │   └── hoodi/
        │       ├── kustomization.yaml
        │       ├── deployment.yaml
        │       ├── configmap-secrets.yaml
        │       └── namespace.yaml
        └── validator-ejector/
            └── mainnet/
                ├── kustomization.yaml         ← namespace: eth-mainnet-lido
                └── statefulset.yaml           ← patch: EL/CL endpoints, operator config

kubernetes/base/evm/validator-ejector/
├── statefulset.yaml                           ← base: probes, resources, common env
├── serviceaccount.yaml
└── kustomization.yaml

12. Key Addresses & IDs