27. Metrics & Health Endpoint
The Kortana node daemon incorporates an enterprise-grade, real-time observability subsystem designed for automated infrastructure orchestration, Prometheus metric collection, and Kubernetes health probing (§34.1–§34.3). Built directly into the core C++23 daemon (telemetry/metrics.hpp and telemetry/metrics_server.cpp), the telemetry server runs on dedicated TCP Port 9090 without external library dependencies.
The telemetry architecture adheres strictly to a zero-overhead design guarantee: metrics recording must never introduce execution latency or lock contention on consensus or state transition paths. All metric operations (increment, set, observe) execute using lock-free relaxed atomic memory primitives (std::memory_order_relaxed and std::bit_cast<uint64_t>) on pre-resolved pointer handles in NodeMetrics. Formatting, label escaping, and Prometheus text rendering occur strictly on demand on the HTTP scraper's thread, guaranteeing that production transaction processing remains completely jitter-free.
27.1 /metrics (Prometheus Format)
The /metrics endpoint on TCP Port 9090 (http://127.0.0.1:9090/metrics) serves operational telemetry in standard, line-oriented Prometheus text exposition format. The endpoint groups metrics under # HELP descriptions and # TYPE declarations (Counters, Gauges, and Histograms), with automatic string escaping to prevent label injection vulnerabilities.
Key metrics exposed by the node registry include:
- Consensus & Block Production:
kortana_block_height: Current finalized block height finalized with a Quorum Certificate.kortana_block_time_seconds: Bucketed histogram of block execution and state trie commit durations.kortana_consensus_round_duration: HotStuff view-change and BLS12-381 signature aggregation latency.kortana_poh_hashes_per_second: Instantaneous SHA-256 hash generation rate of the dedicated dPOH thread.kortana_poh_sequence: Monotonically increasing sequence count of continuous dPOH ticks.
- Transactions, Mempool & Networking:
kortana_tx_count_total: Cumulative count of all transactions executed across both virtual machines.kortana_mempool_size&kortana_mempool_bytes: Count and heap memory footprint of pending transactions.kortana_peer_count: Number of active, authenticated, and scored P2P connections.
- Dual-VM Engine & Storage:
kortana_kil_crossing_duration_seconds: Histogram measuring execution latency when crossing between KEVM and KVM.kortana_state_trie_size_bytes&kortana_storage_size_bytes: Disk storage utilization across all 14 RocksDB column families.kortana_krs_invariant_failure: Critical runtime invariant error monitor (strictly zero in production).
27.2 /health (Liveness & Readiness Probes)
The /health endpoint on TCP Port 9090 (and duplicated on Port 8545) provides real-time liveness and readiness probing for cloud load balancers (AWS ALB/NLB, Google Cloud Armor), Kubernetes ingress controllers, and automated systemd monitoring agents (ops/healthcheck.sh).
The endpoint returns an authoritative, structured JSON document (HealthSnapshot::to_json() in telemetry/metrics_server.cpp):
{ "status": "healthy", "chain_id": "72511", "height": 1048576, "sync_status": "synced", "peers": 28, "poh_sequence": 67108864, "uptime_seconds": 86400 }
The node evaluates its health status dynamically:
- HTTP 200 OK (
"status": "healthy"): Returned when the daemon is fully synced (sync_status == "synced"), has at least one active scored P2P peer, is actively advancing its dPOH tick sequence, and has verified database WAL integrity. - HTTP 503 Service Unavailable (
"status": "unhealthy"): Returned during initial startup WAL replay, snapshot importation, peer isolation (0 peers), or catastrophic state mismatch. This status automatically causes upstream load balancers to route Web3 traffic away from unready instances without dropping user connections.
Automated DevOps orchestration pipelines can execute ops/healthcheck.sh to query this endpoint locally, verifying sub-millisecond node responsiveness before staging traffic.