36. Operations Runbook
The Operations Runbook provides diagnostic procedures for troubleshooting and resolving node operational anomalies.
36.1 Node is Down (Crash Loops)
Inspect systemd logs using journalctl -u kortanad -e and check kernel messages for Out-Of-Memory events (dmesg -T | grep -i oom). If the process was terminated by the OOM killer, reduce the RocksDB block cache size in config.toml and ensure adequate RAM is allocated.
36.2 Chain Has Stalled
If the block height ceases advancing, check ktn_syncStatus and verify that P2P Port 30303 is reachable from external nodes. Verify that active validators are proposing blocks; if a leader is offline, the pacemaker should trigger a view change within 4,500 ms.
36.3 High Memory Consumption
When a node exhibits abnormal memory growth, operators should investigate mempool transaction buffer sizes, WebSocket subscriber queues, and RocksDB memtable allocations. In config.toml, ensure that mempool.max_memory_mb is bounded (default 512 MB) and that storage.write_buffer_size_mb is capped at 64 MB per column family.
Operators can invoke jemalloc profiling or inspect /metrics to identify whether memory is held by RocksDB block caches, pending uncommitted state batches, or network buffer arenas. Restarting the daemon flushes fragmented heap arenas and recovers baseline memory usage.
36.4 Disk is Full
kortanad includes an automated safety guard that halts the daemon if free disk space falls below 2 GB, preventing RocksDB database corruption. Expand the underlying NVMe storage volume and restart the service.
36.5 Validator Not Proposing
Verify that kortana-validator is running and connected to kortanad via its IPC socket. Confirm via ktn_getValidators that the validator's bonded stake meets the minimum threshold and that the node is active in the current epoch schedule.
36.6 State Root Mismatch on Startup
If a node reports a state root mismatch during startup WAL replay, stop the daemon, take a backup of ~/.kortana/data/, and re-synchronize the node using Snapshot Sync (kortanad --sync-mode snapshot).
36.7 P2P Peer Isolation
When a node reports zero connected peers, operators must troubleshoot physical network and firewall configurations. Verify that TCP and UDP Port 30303 are open and forwarded correctly through router NATs (ufw allow 30303/tcp && ufw allow 30303/udp).
Check that the server's public IP address matches the declared external address in config.toml, test reachability to official bootnode endpoints, and verify that the local system time is synchronized via NTP (as massive clock skews can cause SIGMA handshake timeouts).
36.8 Replaying Missed Epochs
If a node is offline for multiple days, initiate fast sync or snapshot sync to catch up to the current epoch tip within minutes.
36.9 Suspected Key Compromise
If an operator suspects private key compromise, broadcast an immediate Unstake transaction (Type 0x31) using the operator key to move bonded DNR into the timelocked unbonding queue, and immediately transfer liquid balances to a fresh hardware wallet.