Troubleshooting

This page tries to answer how to deal with the most popular issues.

Database

Slow database inserts and updates

If you're:

Running behind the tip
Have slow canonical commit time according to the Canonical Commit Latency time chart on Grafana dashboard (more than 2-3 seconds)

Seeing warnings in your logs such as

2023-11-08T15:17:24.789731Z WARN providers::db: Transaction insertion took too long block_number=18528075 tx_num=2150227643 hash=0xb7de1d6620efbdd3aa8547c47a0ff09a7fd3e48ba3fd2c53ce94c6683ed66e7c elapsed=6.793759034s

then most likely you're experiencing issues with the database freelist. To confirm it, check if the values on the Freelist chart on Grafana dashboard is greater than 10M.

Currently, there are two main ways to fix this issue.

Compact the database

It will take around 5-6 hours and require additional disk space located on the same or different drive.

Clone Reth

git clone https://github.com/paradigmxyz/reth
cd reth

Build database debug tools
```
make db-tools
```
Run compaction (this step will take 5-6 hours, depending on the I/O speed)
```
./db-tools/mdbx_copy -c $(reth db path) reth_compact.dat
```
Stop Reth

Backup original database

mv $(reth db path)/mdbx.dat reth_old.dat

Move compacted database in place of the original database
```
mv reth_compact.dat $(reth db path)/mdbx.dat
```
Start Reth
Confirm that the values on the Freelist chart is near zero and the values on the Canonical Commit Latency time chart is less than 1 second.
Delete original database
```
rm reth_old.dat
```

Re-sync from scratch

It will take the same time as initial sync.

Stop Reth
Drop the database using by remove the db and static file directories found in your data_dir
Start reth

Database write error

If you encounter an irrecoverable database-related errors, in most of the cases it's related to the RAM/NVMe/SSD you use. For example:

Error: A stage encountered an irrecoverable error.

Caused by:
0: An internal database error occurred: Database write error code: -30796
1: Database write error code: -30796

Error: A stage encountered an irrecoverable error.

Caused by:
0: An internal database error occurred: Database read error code: -30797
1: Database read error code: -30797

Check your memory health: use memtest86+ or memtester. If your memory is faulty, it's better to resync the node on different hardware.

Check database integrity:

git clone https://github.com/paradigmxyz/reth
cd reth
make db-tools
./db-tools/mdbx_chk $(reth db path)/mdbx.dat | tee mdbx_chk.log

If mdbx_chk has detected any errors, please open an issue and post the output from the mdbx_chk.log file.

Concurrent database access error (using containers/Docker)

If you encounter an error while accessing the database from multiple processes and you are using multiple containers or a mix of host and container(s), it is possible the error is related to PID namespaces. You might see one of the following error messages.

mdbx:0: panic: Assertion `osal_rdt_unlock() failed: err 1' failed.

pthread_mutex_lock.c:438: __pthread_mutex_lock_full: Assertion `e != ESRCH || !robust' failed

If you are using Docker, a possible solution is to run all database-accessing containers with --pid=host flag.

For more information, check out the Containers section in the libmdbx README.

Hardware Performance Testing

If you're experiencing degraded performance, it may be related to hardware issues. Below are some tools and tests you can run to evaluate your hardware performance.

If your hardware performance is significantly lower than these reference numbers, it may explain degraded node performance. Consider upgrading your hardware or investigating potential issues with your current setup.

Disk Speed Testing with IOzone

Test disk speed:

iozone -e -t1 -i0 -i2 -r1k -s1g /tmp

Reference numbers (on Latitude c3.large.x86):

Children see throughput for 1 initial writers = 907733.81 kB/sec
Parent sees throughput for 1 initial writers = 907239.68 kB/sec
Children see throughput for 1 rewriters = 1765222.62 kB/sec
Parent sees throughput for 1 rewriters = 1763433.35 kB/sec
Children see throughput for 1 random readers = 1557497.38 kB/sec
Parent sees throughput for 1 random readers = 1554846.58 kB/sec
Children see throughput for 1 random writers = 984428.69 kB/sec
Parent sees throughput for 1 random writers = 983476.67 kB/sec

Test disk speed with memory-mapped files:

iozone -B -G -e -t1 -i0 -i2 -r1k -s1g /tmp

Reference numbers (on Latitude c3.large.x86):

Children see throughput for 1 initial writers = 56471.06 kB/sec
Parent sees throughput for 1 initial writers = 56365.14 kB/sec
Children see throughput for 1 rewriters = 241650.69 kB/sec
Parent sees throughput for 1 rewriters = 239067.96 kB/sec
Children see throughput for 1 random readers = 6833161.00 kB/sec
Parent sees throughput for 1 random readers = 5597659.65 kB/sec
Children see throughput for 1 random writers = 220248.53 kB/sec
Parent sees throughput for 1 random writers = 219112.26 kB/sec

RAM Speed and Health Testing

Check RAM speed with lshw:

sudo lshw -short -C memory

Look for the frequency in the output. Reference output:

H/W path              Device          Class          Description
================================================================
/0/24/0                               memory         64GiB DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
/0/24/1                               memory         64GiB DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
...

Test RAM health with memtester:
```
sudo memtester 10G
```
This will take a while. You can test with a smaller amount first:
```
sudo memtester 1G 1
```
All checks should report "ok".

Botanix Federation book