> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sei.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Detailed guide for troubleshooting node problems.

export const RandomPeers = ({format = 'bash', network = 'mainnet', count = 5}) => {
  const PEERS = {
    mainnet: ['d53ab7681ed0df3d3249fc0132df7c3a131c9c1a@45.250.253.40:51656', '4ee8aea5bb58e9038d72e74322e3bba755287398@202.8.8.183:11956', '81409623ae4da3ec7b400cf640dea0b0999a964b@57.128.230.96:51556', '61a5be64b5215786fe5da712584678d0626636b5@87.249.137.71:51556', 'f83c536f43df9a5d900cd3c2f702c04d7dddc7b5@136.243.67.45:11956', 'b8d600a2f568576b5a2df5ee649cf9b809389064@162.19.62.176:26756', '57860b18ed3e1bbe8901ba73f2e63c7e6fe8b3d3@57.129.54.81:26656', '04fc6bba6c5c33034811612dca31c7adda24b299@91.134.60.37:16856', 'de64b779c7f4091e6f1765f5ca4c46f9d3011732@65.108.70.106:46656', '3be6b24cf86a5938cce7d48f44fb6598465a9924@p2p.state-sync.pacific-1.seinetwork.io:26656', '70e0c91b83b5ed1beaca798267f2debdf97dac10@18.156.6.83:26656', 'dd6b1ae002a15c1c8a38e05660f49a93c75d4159@148.251.181.225:26656'],
    testnet: ['71beea83970431f55816eee5f066a611a1dc80f7@p2p.state-sync.atlantic-2.seinetwork.io:26656', '65c257f9275beb1b99ca169ef89743c034b15db0@3.76.192.224:26656', '33588592e477c5238ff2a5d8dc765f85790ef853@23.109.47.225:26656', 'babc3f3f7804933265ec9c40ad94f4da8e9e0017@testnet-seed.rhinostake.com:11956', '8542cd7e6bf9d260fef543bc49e59be5a3fa9074@seed.publicnode.com:56656']
  };
  const pickRandom = (arr, n) => {
    const shuffled = [...arr];
    for (let i = shuffled.length - 1; i > 0; i--) {
      const j = Math.floor(Math.random() * (i + 1));
      [shuffled[i], shuffled[j]] = [shuffled[j], shuffled[i]];
    }
    return shuffled.slice(0, Math.min(n, arr.length));
  };
  const CopyIcon = ({className}) => <svg role="img" aria-label="Copy" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
      <path d="M7 7m0 2.667a2.667 2.667 0 0 1 2.667 -2.667h8.666a2.667 2.667 0 0 1 2.667 2.667v8.666a2.667 2.667 0 0 1 -2.667 2.667h-8.666a2.667 2.667 0 0 1 -2.667 -2.667z" />
      <path d="M4.012 16.737a2.005 2.005 0 0 1 -1.012 -1.737v-10c0 -1.1 .9 -2 2 -2h10c.75 0 1.158 .385 1.5 1" />
    </svg>;
  const CheckIcon = ({className}) => <svg role="img" aria-label="Copied" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
      <path d="M5 12l5 5l10 -10" />
    </svg>;
  const ShuffleIcon = ({className}) => <svg role="img" aria-label="Shuffle" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
      <path d="M16 3h5v5" />
      <path d="M4 20l16.2 -16.2" />
      <path d="M21 16v5h-5" />
      <path d="M15 15l6 6" />
      <path d="M4 4l5 5" />
    </svg>;
  const pool = PEERS[network] ?? [];
  const [peers, setPeers] = useState(() => pool.slice(0, count));
  const [copied, setCopied] = useState(false);
  const shuffle = () => {
    setPeers(pickRandom(pool, count));
  };
  useEffect(() => {
    shuffle();
  }, [network, count]);
  if (pool.length === 0) {
    return <div className="not-prose w-full">
        <pre className="m-0 p-3 rounded-md bg-neutral-100 dark:bg-neutral-800 text-sm opacity-70" style={{
      fontFamily: 'var(--sei-font-mono)'
    }}>
          No peers configured for network “{network}”.
        </pre>
      </div>;
  }
  const peerString = peers.join(',');
  const displayText = format === 'toml' ? `persistent_peers = "${peerString}"` : `PEERS="${peerString}"`;
  const handleCopy = async () => {
    try {
      await navigator.clipboard.writeText(displayText);
      setCopied(true);
      setTimeout(() => setCopied(false), 2000);
    } catch {}
  };
  return <div className="not-prose w-full">
      <div className="relative">
        <pre className="m-0 px-4 py-3 pr-28 rounded-lg bg-neutral-100 dark:bg-neutral-900 text-[12.5px] leading-[1.55] whitespace-pre-wrap break-all border border-neutral-200 dark:border-neutral-700 text-neutral-900 dark:text-neutral-100" style={{
    fontFamily: 'var(--sei-font-mono)'
  }}>
          <code style={{
    fontFamily: 'var(--sei-font-mono)'
  }}>{displayText}</code>
        </pre>
        <div className="absolute top-2 right-2 flex items-center gap-1.5">
          <button type="button" onClick={shuffle} aria-label="Shuffle peers" title="Shuffle" className="inline-flex items-center gap-1 px-2 py-1 rounded-md text-[11px] font-semibold border border-neutral-300 dark:border-neutral-600 bg-white/80 dark:bg-neutral-800/80 text-neutral-700 dark:text-neutral-200 hover:bg-white dark:hover:bg-neutral-800 transition-colors">
            <ShuffleIcon />
            Shuffle
          </button>
          <button type="button" onClick={handleCopy} aria-label="Copy peers" title="Copy to clipboard" className="inline-flex items-center gap-1 px-2 py-1 rounded-md text-[11px] font-semibold border border-neutral-300 dark:border-neutral-600 bg-white/80 dark:bg-neutral-800/80 text-neutral-700 dark:text-neutral-200 hover:bg-white dark:hover:bg-neutral-800 transition-colors">
            {copied ? <>
                <CheckIcon className="text-green-600" />
                Copied
              </> : <>
                <CopyIcon />
                Copy
              </>}
          </button>
        </div>
      </div>
    </div>;
};

Understanding common errors and their solutions helps maintain a healthy node
operation.

## Common Error Codes

Here are the most frequent errors you might encounter and their solutions:

### Consensus Errors

When you encounter consensus errors, quick and appropriate action is essential:

```text theme={"dark"}
Error: "Consensus failure - height halted"
Solution: Check for network upgrades or chain halts
Command: seid status
```

```text theme={"dark"}
Error: "Private validator file not found"
Solution: Restore validator key or check file permissions
Location: $HOME/.sei/config/priv_validator_key.json
```

```text theme={"dark"}
Error: "Duplicate signature"
Solution: IMMEDIATELY STOP NODE - potential double signing risk
Action: Check validator operation on other machines
```

### Network Errors

Network errors can prevent your node from participating in consensus:

```text theme={"dark"}
Error: "Dial tcp connection refused"
Solution: Check network connectivity and firewall rules
Commands:
  - netstat -tulpn | grep seid
  - ufw status
```

```text theme={"dark"}
Error: "No peers available"
Solution: Verify peer connections and network config
Commands:

- curl localhost:26657/net_info
```

### Database Errors

Database corruption can require immediate attention:

```text theme={"dark"}
Error: "Database is corrupted"
Solution: Reset database or restore from backup
Commands:
  - seid tendermint unsafe-reset-all
  - cp -r backup/data $HOME/.sei/
```

### Diagnostic Commands

These commands help you investigate issues and monitor your node:

```bash theme={"dark"}
# Check node synchronization
seid status

# Check validator status
seid query staking validator $(seid tendermint show-validator)

# Monitor real-time logs
journalctl -fu seid -o cat

# View system resource usage
top -p $(pgrep seid)
```

## AppHash Mismatch Errors

If you encounter an AppHash mismatch, you'll need to capture the state for comparison with a known good version:

```bash theme={"dark"}
# For SeiDB (most non-archive nodes):
git clone https://github.com/sei-protocol/sei-db.git
cd sei-db/tools
make install
systemctl stop seid
seidb dump-iavl -d $HOME/.sei/data/committer.db -o /home/ubuntu/iavl-dump
systemctl restart seid

# For Legacy IAVL DB:
seid debug dump-iavl <latest height>
```

Always include the app hash, commit hash, and block height from your logs when reporting issues.

### Identifying AppHash Errors

AppHash errors typically appear in logs as:

```text theme={"dark"}
ERR wrong Block.Header.AppHash. Expected [EXPECTED_HASH], got [ACTUAL_HASH]
block_id={"hash":"...","parts":{"hash":"...","total":1}} height=[HEIGHT]
```

**Common Causes:**

* Using incorrect node version during sync (ensure you're on the latest version)
* Corrupted or incorrectly applied snapshots
* Database inconsistencies from improper shutdowns
* Syncing with outdated or incompatible peers

**Resolution Steps:**

1. **Stop the node immediately.**

2. **Try a node rollback first:**, see [here](/node/troubleshooting#node-rollback)

3. **If rollback fails, restore from a fresh snapshot:**

   * Download a recent snapshot from trusted providers (Polkachu, PublicNode)
   * Ensure you're using the correct node version
   * Verify peer configurations are up to date

4. **Restart the node and monitor logs for continued errors**

### Peer Connection Issues as AppHash Red Herrings

**Important Note:** Peer connection failures are often symptoms of underlying AppHash errors, not the root cause.

When you see extensive peer connection errors like:

```text theme={"dark"}
ERR failed to handshake with peer
ERR failed to send request for peers
ERR peer handshake failed endpoint={} err=EOF
```

**Don't focus solely on fixing peer connections first.** Instead:

1. **Scan your logs carefully** for AppHash errors that may appear intermittently
2. **Look for the actual error pattern:**
   ```text theme={"dark"}
   ERR wrong Block.Header.AppHash. Expected [HASH], got [HASH]
   ```
3. **Check if your node is stuck** at a specific height despite peer connection attempts

**Why This Happens:**

* AppHash mismatches prevent proper block validation
* Node cannot advance to new blocks due to state inconsistency
* Peers may reject connections from nodes with corrupted state
* Network appears to be the problem when it's actually a local state issue

**Debugging Approach:**

1. **First, check for AppHash errors** in your logs (search for "wrong Block.Header.AppHash")
2. **If AppHash errors are found**, treat this as the primary issue
3. **Only focus on peer connection fixes** if no AppHash errors exist

This approach can save hours of debugging time by addressing the root cause rather than symptoms.

### Peer Connection and Handshake Issues

**Identifying Peer Issues:**

Look for these error patterns in your logs:

```text theme={"dark"}
ERR failed to handshake with peer err="expected to connect with peer \"[EXPECTED_ID]\", got \"[ACTUAL_ID]\""
ERR failed to send request for peers err="no available peers to send a PEX request to (retrying)"
ERR peer handshake failed endpoint={} err=EOF module=p2p
```

**Common Causes:**

* Outdated peer configurations with mismatched node IDs
* Network infrastructure changes on peer side
* Firewall blocking connections on port 26656
* DNS resolution issues

**Resolution Steps:**

1. **Update peer configurations** with current node IDs:

   <RandomPeers network="mainnet" format="toml" />

2. **Verify network connectivity:**

   ```bash theme={"dark"}
   # Test connection to peer endpoints
   nc -zv p2p.state-sync-0.pacific-1.seinetwork.io 26656

   # Check if port 26656 is open for inbound connections
   netstat -tulpn | grep :26656
   ```

3. **Check current peer status:**
   ```bash theme={"dark"}
   curl http://localhost:26657/net_info | jq '.result.peers | length'
   curl http://localhost:26657/lag_status | jq .
   ```

### Sync Performance Issues

**Identifying Sync Problems:**

Monitor these indicators:

```bash theme={"dark"}
# Check sync status and lag
curl http://localhost:26657/lag_status | jq .

# Monitor if height is progressing
curl http://localhost:26657/status | jq '.result.sync_info'
```

**Common Solutions:**

1. **Increase packet payload size** for large block processing:

   ```toml theme={"dark"}
   # In config.toml [p2p] section
   max-packet-msg-payload-size = 1024000  # Increase from default 102400
   ```

2. **Optimize mempool settings** in `config.toml`:

   ```toml theme={"dark"}
   # In [mempool] section
   keep-invalid-txs-in-cache = true
   ttl-duration = "5s"
   ttl-num-blocks = 5
   ```

3. **If node gets stuck at specific height:**
   * Try restarting the node
   * If restart doesn't help, perform rollback
   * Consider taking a fresh snapshot

**Warning Signs to Watch For:**

* Current height not increasing over time
* Increasing lag between current height and max peer height
* Repeated timeout errors in logs
* Mempool size consistently reaching limits

## Crash and Panic Debugging

For crashes, panics, or nil pointer exceptions:

* Capture at least 1,000 lines of logs leading up to the crash
* Or collect 15 minutes of log data, whichever provides more context
* Include the full stack trace if available

### Logging Configuration

Proper logging configuration is essential for debugging and monitoring:

```toml theme={"dark"}
# In config.toml
# Set appropriate log level
log_level = "debug"  # Use "trace" for maximum detail

# Choose log format
log_format = "json"  # Use "plain" for human-readable logs
```

Configure log rotation to manage storage effectively:

```bash theme={"dark"}
# Example logrotate configuration
sudo tee /etc/logrotate.d/seid << EOF
/var/log/seid/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 sei sei
    sharedscripts
    postrotate
        systemctl reload seid
    endscript
}
EOF
```

Enable core dumps for crash analysis:

```bash theme={"dark"}
# Set unlimited core dump size
ulimit -c unlimited

# Configure core dump location
echo "/tmp/core.%e.%p" > /proc/sys/kernel/core_pattern
```

## Other common Issues and Fixes

1. **Sync Problems**

   * Check available disk space (`df -h`)
   * Ensure proper peer connections (`curl http://localhost:26657/net_info`)
   * Verify firewall settings (port 26656 open)

2. **Performance Issues**

   * Monitor system resources (`htop` or `iotop`)
   * Check disk I/O performance (`iostat`)
   * Analyze network traffic (`iftop`)

3. **Database Issues**

   * Run database integrity checks using:

     ```bash theme={"dark"}
     seid debug dump-db | grep -i error
     ```

     If errors are detected, consider restoring from a recent backup.

   * Consider pruning excessive historical data by adjusting `ss-keep-recent` in
     `app.toml` or running:

     ```bash theme={"dark"}
     seid tendermint unsafe-reset-all --home $HOME/.sei --keep-addr-book
     ```

     Alternatively, manually remove old state snapshots to free up space:

     ```bash theme={"dark"}
     rm -rf $HOME/.sei/data/snapshots/*
     ```

## Node Rollback

To rollback a node from an AppHashed state, you need to stop the node first.
Do this in your preferred way.

Next, do a rollback with:

```bash theme={"dark"}
seid rollback
```

Then, restart the node.

In case you see the following error while trying to rollback:

```bash theme={"dark"}
failed to initialize database: resource temporarily unavailable
```

This means that you did not shutdown the node properly. Try to shutdown or kill the `seid` process directly in that case. If this doesn't help, restart your machine.

Then try the rollback steps again.