Monitoring and Alerting

Project Documentation

• NEAR Protocol Validator Guide
• NEAR Nodes Validator Bootcamp

Monitoring

• Dev
• Prod

Maintenance

There are two key components running as part of the validator setup:

• /home/neard/bin/neard: The validator client software, managed via a systemd service.
• /home/neard/bin/ping.sh: A script that sends staking proposals; scheduled via a cron job to run every 8 hours.

Proposal Logic

To become a validator, a node must meet the minimum required stake, known as the seat price. To participate, it needs to submit a staking proposal every epoch with the required amount of stake.

Proposals target epoch +2, which means if you submit a proposal now and it’s approved, the validator will receive a seat 3 epochs later.

Each NEAR epoch has 43,200 blocks, and with an average block time of about 1 second, this results in an epoch duration of roughly 12 hours.

To ensure continuous participation, the script is designed to automatically submit a proposal every epoch.

For more details on how seat prices are calculated, check out this guide.

Logs

• /var/log/neard/neard.log: Logs for the validator client
• /var/log/neard/ping.log: Logs for the staking proposal script

SLAs

Uptime is measured by how many chunks we produce or validate compared to how many we’re expected to at a given time (i.e., actual/expected rate). It depends on three roles: block production, chunk production, and chunk validation. At the moment, we only have 1 role which is chunk validation. To be able to get the other roles, we need to be in the top 100 validators weighted by the total number of stakes delegated.

We must maintain a chunk endorsement/validation rate above 90% (we've set an alert at 92% to allow some buffer). Missing blocks or chunks is the primary reason a validator is removed from the active pool. The second most common reason is not having enough stake in our staking pool. The action needed is to check the current seat price or minimum stake required for all active validators. Additionally, if we don’t propose a block every epoch, we could also lose our active status.

As a side note, for block and chunk producers, uptime is calculated differently because the system prioritizes chunk production over validation. For example, cosmose.poolv1.near has 100% uptime: 237/237 blocks, 1684/1684 chunks, but "only" 91378/92274 chunk validations. Another one, bisontrails.poolv1.near, shows 99.91% uptime with 243/243 blocks, 2165/2167 chunks, and 86747/87414 validations. So chunk validation isn’t the main factor in their uptime, especially if chunk production isn’t perfect.

Failover

We maintain a secondary node in a different location to handle potential hardware or network failures on the primary/active node. The failover node uses a different node key and does not have the validator key. It shares the same configs as the active node, with two key differences:

• The active node has the file /home/neard/.near/validator_key.json
• The tracked_shadow_validator field is absent in the active node's /home/neard/.near/config.json

At a high level, the failover process involves:

• Copying validator key to the failover node
• Removing the validator key from the original node
• Deleting the tracked_shadow_validator field in the config file
• Swapping the node key
• Restarting both services

Instruction

1. Check out a new branch from master inside ops.
2. Set the desired host as active using the neard_active_instances flag. Make sure only one instance is listed — there should only be one active instance at any given time. If more than one is active, our chunk endorsement rate will suffer due to throttling.
3. SSH login to all hosts in the pod.
4. Run the Ansible playbook targeting all hosts to trigger the failover process as described above and ensure to set the neard_failover_enabled var.

For example, in dev:

dev-ansible-playbook -i inventories/development-pod5-near/hosts neard.yml --extra-vars 'neard_failover_enabled=true'

And an example in prod:

prd-ansible-playbook -i inventories/pod10/hosts neard.yml --extra-vars 'neard_failover_enabled=true' | tee -a ~/ansible_logs/near-val-p10-xxxxxx.txt

5. Create a PR with your changes and get it reviewed.

Alerts