On January 25th, 2024, at approximately 08:28 UTC, the enactment of Kusama runtime RT1001000 caused the relay chain to reject blocks from all parachains including Moonriver (this occurred at Moonriver block 6012045).
Consequently, all Kusama parachains stalled, as they could not produce subsequent blocks due to the lingering issues in Kusama. The issue seems linked to code changes to KusamaGo to page https://github.com/paritytech/polkadot-sdk/issues/3058, as it prepares to activate Asynchronous Backing.Go to page https://wiki.polkadot.network/docs/learn-async-backing
The stall lasted one Kusama epoch, which is set to 1 hour. When the new epoch started, all Kusama parachains began producing blocks again, including Moonriver. Nevertheless, some RPC providers noted that specific calls through the Ethereum JSON-RPC endpoints were failing. Not all nodes were affected.
After further investigation, the team pinpointed the issue to a Frontier database corruption resulting from the Kusama stall.
If you are running a node, you can verify if your node is affected by executing the following call (assuming you are running the default ports):
curl –location ‘https://localhost:9944 \
–header ‘Content-Type: application/json’ \
–data ‘{
“jsonrpc”:”2.0″,
“id”:1,
“method”:”eth_getBlockByHash”,
“params”: [“0x69856c1b38ee35715263f144ebd3624b47ebcef9346c05bce54cbf5a629228a8”, false] }’
If you see a null response, your Frontier database is corrupted. To fix the issue, take the following steps:
Stop your node
Remove (or rename in case you want to keep a backup) the Frontier database folder, this is normally located in “/path/to/data/chains/moonriver/frontier” – For example, you can do “rm -rf /path/to/data/chains/moonriver/frontier/*” or “mv /path/to/data/chains/moonriver/frontier /path/to/data/chains/moonriver/frontier_backup”
Start the node
The Frontier database starts reconstructing itself from the top block to genesis. This means that you can retrieve the latest block information as soon as the node is restarted, and it should take roughly 20 minutes to be fully reconstructed. The database’s fully reconstructed size is approximately 5+ GB. You can check if the database reconstruction is complete by querying Block 1 as follows:
curl –location ‘https://localhost:9944’ \
–header ‘Content-Type: application/json’ \
–data ‘{
“jsonrpc”:”2.0″,
“id”:1,
“method”:”eth_getBlockByHash”,
“params”: [“0xb1349282e7b5f0f17a152b8d6a8f719690a3a6cc661ad2e2c6a32e14bf7d395a”, false] }’
The call should not return null.
Summary
- At approximately 08:28 UTC, the enactment of Kusama runtime RT1001000 caused the relay chain to reject blocks from all parachains including Moonriver (this occurred at Moonriver block 6012045).
- At approximately 09:30 UTC, the situation resolved itself following the completion of the current Kusama epoch and Moonriver began to produce blocks successfully.
- However, a subset of Moonriver RPC nodes began returning an incorrect response for certain EVM requests leading to errors within some wallets, dapps and block explorers.
- These errors were traced to a corruption of an internal Frontier database within affected nodes. The Frontier database is related to the Ethereum emulation layer that runs on Moonriver.
- Over the following several hours, node operators began the process of completing a recovery process to restore service.
- As of 17:00 UTC, most node operators had completed the recovery steps leading to normal operations for most users and applications.
- Some node operators may have yet to complete this recovery process so some users and dapp developers may still experience errors in some cases.
- The patience and understanding of the community is greatly appreciated.
- The Moonbeam team continues to investigate the trigger of the database corruption to improve the resiliency of services should a similar incident occur in the future.