Metrics

DRL exposes a Prometheus-compatible metrics endpoint on the standard port (default 9090) at /metrics. A /health liveness endpoint is available on the same port.

The endpoint uses a dedicated private registry — it does not expose Go runtime or process metrics by default, keeping the surface clean and DRL-specific.

Endpoint configuration

metrics {
    port 9090
}

Scrape target for Prometheus:

- job_name: drl
  static_configs:
    - targets: ["<drl-host>:9090"]

Metric reference

gRPC

Key alerts:

• Sudden drop in drl_grpc_check_total → DRL is unreachable from Envoy.
• Sustained spike in OVER_LIMIT share → active attack or misconfigured rate limit.

Rate limiting

Key alerts:

• drl_ratelimit_propagation_latency_ms p99 > 500 ms → gossip is lagging; check cluster network and drl_membership_cluster_size.

Cache

Both blocklist and accounting report under the same metric names, distinguished by the cache_type label.

cache_type label values: blocklist, accounting.

Key alerts:

• drl_cache_evictions_total{cache_type="blocklist"} > 0 → blocklist is full; blocked entities are being silently dropped.
• drl_cache_memory_bytes approaching max_size_mb × 1,048,576 → consider increasing the configured limit.

Accounting

The ratio remote / (local + remote) reflects how evenly traffic is distributed across the hash ring. In a balanced N-node cluster this should be approximately (N−1) / N.

Key alerts:

• drl_accounting_remote_increments_total near zero on a multi-node cluster → the hash ring may not have converged; check drl_membership_cluster_size.

Membership

Key alerts:

• drl_membership_cluster_size < expected node count → a node has left or become unreachable; block propagation will be incomplete until it recovers or is replaced.

Handover

Handover metrics track the graceful counter-migration that happens when a node leaves the cluster, ensuring the departing node’s owned accounting counters are transferred to a new owner before it shuts down.

Key alerts:

• drl_accounting_handover_failed_total > 0 → investigate network connectivity between nodes at shutdown time.

Recommended Grafana dashboard panels