36. Operations Runbook

When an issue arises at 3:00 AM, refer to this runbook.

36.1 Node is Down (Crash Loops)

• Check OOM: Run dmesg -T | grep -i oom. If the node was killed by the OS, your RocksDB cache is set too high. Reduce db.block_cache_mb in config.toml.
• Check Config: Run kortanad --check.

36.2 Chain Has Stalled

• Check ktn_syncStatus. Is the node waiting on a specific block?
• Check peer count. If net_peerCount is 0, check your firewall (Port 30303 TCP/UDP).

36.4 Disk is Full

• The node will gracefully shut down if disk space drops below 2 GB.
• Fix: Provision a larger NVMe drive, stop the node, rsync the data directory to the new drive, update mounts, and restart.

36.5 Validator Not Proposing

• Check the logs of the kortana-validator process. Is it connected to the kortanad socket?
• Is the node fully synced? A validator will not propose blocks if it is behind the network tip.

36.6 State Root Mismatch on Startup

• Symptom: "FATAL: State root mismatch after WAL replay."
• Fix: The hardware may have corrupted data before it was fsynced. Delete the state/ directory and restart the node with --sync-mode snapshot to download a fresh trie.

36.9 Suspected Key Compromise

If you suspect your validator's operator key (operator.key) is compromised:

1. Immediately broadcast a UndelegateStake transaction from a secure machine.
2. The funds will enter the unbonding period.
3. The attacker cannot instantly drain the funds, giving you a window to out-gas them on the withdrawal transaction when the unbonding period ends.

---