Documentation
Docs37. Monitoring & Alerting

37. Monitoring & Alerting

The Kortana node daemon incorporates an enterprise-grade, real-time observability subsystem designed for high-availability production environments (§34.1–§34.3). To meet the stringent performance demands of validating nodes and high-volume RPC gateways, the telemetry engine is implemented natively in modern C++23 (telemetry/metrics.hpp and telemetry/metrics_server.cpp) without incurring the binary bloat or runtime overhead of external monitoring libraries.

A fundamental design invariant of the Kortana telemetry architecture is that metrics recording must never degrade node performance. Every metric measurement in kortana-node executes as a lock-free, relaxed atomic operation (std::memory_order_relaxed and std::bit_cast<uint64_t>) on pre-resolved pointer handles. Recording an execution duration, incrementing a transaction counter, or updating a mempool size takes zero heap allocations, zero mutex acquisitions, and zero string formatting overhead, ensuring that high-frequency code paths (including the dPOH generator loop, block commit pipeline, and P2P gossip handlers) remain entirely unblocked. Metric aggregation and text serialization occur strictly on scrape demand on the scraper's isolated background thread.

The built-in HTTP metrics daemon (MetricsServer) runs on dedicated TCP Port 9090, binding to the loopback interface (127.0.0.1:9090) by default for security isolation. Production reverse proxies can expose these telemetry endpoints to Prometheus servers, Grafana dashboards, and automated site reliability engineering (SRE) alerting systems.

37.1 Prometheus Metrics

The Kortana daemon exposes its internal state in standard, line-oriented Prometheus text exposition format via the http://127.0.0.1:9090/metrics endpoint. The text renderer organizes metrics strictly according to official Prometheus standards, grouping series under # HELP descriptions and # TYPE declarations while automatically escaping dynamic label strings (such as RPC method names) to prevent stream corruption.

The NodeMetrics registry (telemetry/metrics.hpp) exposes comprehensive operational counters, gauges, and histograms across every protocol subsystem:

  • Consensus & Block Production:
    • kortana_block_height (Gauge): The current canonical block height finalized with a valid Quorum Certificate.
    • kortana_block_time_seconds (Histogram): Bucketed duration of block processing, execution, and state commitment.
    • kortana_consensus_round_duration (Histogram): Time taken to aggregate BLS12-381 validator signatures into Quorum Certificates.
    • kortana_poh_hashes_per_second (Gauge): Instantaneous SHA-256 hash generation rate of the local dPOH generator thread.
    • kortana_poh_sequence (Counter): Monotonically increasing sequence count of all continuous dPOH ticks since genesis.
  • Transactions & Mempool:
    • kortana_tx_count_total (Counter): Cumulative count of all transactions executed across both virtual machines.
    • kortana_mempool_size (Gauge): Total count of valid transactions currently buffered in the mempool priority queue.
    • kortana_mempool_bytes (Gauge): Total heap memory consumption of the active mempool buffer arena.
  • Networking & P2P Fabric:
    • kortana_peer_count (Gauge): Number of active, authenticated, and scored P2P connections in the routing table.
    • kortana_sync_progress (Gauge): Percentage completion of block, fast, or snapshot state synchronization.
  • Virtual Machines & Storage:
    • kortana_kil_crossing_duration_seconds (Histogram): Latency of cross-VM execution transitions between KEVM and KVM.
    • kortana_state_trie_size_bytes & kortana_storage_size_bytes (Gauge): Physical disk utilization across all 14 RocksDB column families.
    • kortana_krs_invariant_failure (Counter): Critical runtime invariant error monitor (strictly zero in normal operation).
    • kortana_rpc_requests_total & kortana_rpc_latency_seconds (Labeled Histogram): Per-method JSON-RPC query volume and execution latency.

37.2 Alert Rules

To protect validator infrastructure against silent consensus failures, hardware bottlenecks, and network isolation, node operators should configure Prometheus Alertmanager or PagerDuty rules against the following critical telemetry thresholds:

  • Critical: Stalled Block Height (KortanaChainStalled):

    alert: KortanaChainStalled
expr: increase(kortana_block_height[5m]) == 0
for: 3m
labels:
  severity: critical
annotations:
  summary: "Kortana node is not advancing block height"
  description: "Block height has ceased increasing for >3 minutes. Check validator connection and consensus pacemaker."

    Signals that the local node has desynchronized from the network or that the consensus engine is experiencing consecutive view-change timeouts.

  • High: Peer Table Depletion (KortanaPeerIsolation):

    alert: KortanaPeerIsolation
expr: kortana_peer_count < 4
for: 3m
labels:
  severity: high
annotations:
  summary: "Node is isolated from P2P network"
  description: "Connected P2P peers dropped below minimum fault tolerance threshold (4 peers). Verify Port 30303 reachability."

  • Critical: dPOH Clock Stalling (KortanaPohClockStall):

    alert: KortanaPohClockStall
expr: rate(kortana_poh_sequence[1m]) < 400000
for: 2m
labels:
  severity: critical
annotations:
  summary: "dPOH SHA-256 generation rate degraded"
  description: "dPOH hash rate is below 400,000 hashes/sec. Dedicated generator thread may be experiencing CPU throttling."

  • Warning: Mempool Saturation (KortanaMempoolSaturated):

    alert: KortanaMempoolSaturated
expr: kortana_mempool_bytes > 200000000
for: 5m
labels:
  severity: warning
annotations:
  summary: "Mempool memory approaching capacity"
  description: "Mempool heap allocation exceeds 200 MB. Low-fee transactions may begin experiencing eviction."

  • Fatal: Protocol Invariant Violation (KortanaInvariantViolation):

    alert: KortanaInvariantViolation
expr: increase(kortana_krs_invariant_failure[1m]) > 0
for: 0m
labels:
  severity: page
annotations:
  summary: "FATAL: On-chain invariant failure detected"
  description: "A formal protocol invariant has been violated. Immediate manual investigation required."

37.3 Health Checks

The Kortana daemon provides automated liveness and readiness health probing via the http://127.0.0.1:9090/health endpoint (and through the primary RPC interface at http://127.0.0.1:8545/health). Engineered specifically for Kubernetes pod probes, AWS Application Load Balancer target groups, and automated systemd watchdogs (ops/healthcheck.sh), the endpoint performs real-time internal subsystem health evaluations.

The health endpoint responds with a structured JSON payload representing the node's authoritative operating state (HealthSnapshot::to_json() in telemetry/metrics_server.cpp):

{
  "status": "healthy",
  "chain_id": "72511",
  "height": 1048576,
  "sync_status": "synced",
  "peers": 28,
  "poh_sequence": 67108864,
  "uptime_seconds": 86400
}

The daemon evaluates its health status dynamically according to strict operational criteria:

  • HTTP 200 OK ("status": "healthy"): Returned when the node daemon is fully synchronized with the canonical chain tip (sync_status == "synced"), has at least one active, scored P2P peer connection, is actively advancing its dPOH tick sequence, and has successfully passed RocksDB startup WAL integrity verification.
  • HTTP 503 Service Unavailable ("status": "unhealthy"): Returned during initial startup WAL replay, snapshot importation, peer isolation (0 peers), or when block synchronization falls significantly behind the network head. This status automatically causes upstream load balancers to route Web3 traffic away from unready instances without dropping user connections.

Automated DevOps orchestration pipelines can execute the bundled diagnostic script ops/healthcheck.sh to query this endpoint locally, verifying sub-millisecond node responsiveness before staging traffic.

Part IX — CLI Reference

Previous 36. Operations RunbookNext38. `kortana-cli`