The Moonbeam network officially launched on January 11, 2022. On launch, transfers and EVM functionality were enabled and SUDO access had been revoked. Due to the fact that Moonbeam was the first parachainGo to page https://moonbeam.network/education/what-is-a-parachain/ to become fully operational on Polkadot, that it is one of the first complex Substrate-based parachain implementations offering generic smart contract functionality, and that it was also introducing many novel features based on Substrate and Polkadot technology, the developers of Moonbeam together with the Moonbeam Foundation decided to develop and deploy a unique security feature called Maintenance Mode since the initial release.
The maintenance pallet is a module that is part of all Moonbeam runtimes and is intended to be used only in extremely rare cases of existential threats to the network. The mode effectively suspends all transaction processing and EVM execution while regular block production continues. Governance and Staking operations continue to function. As described further below, Maintenance Mode is initiated by the Technical Committee, which was appointed by the Moonbeam Foundation and early active members of the development community.
The feature was designed to support emergency upgrades on the network. Upgrading the network always has a 1-hour delay between the announcement and the enactment, due to how the relay and parachain upgrade systems work. In the case of an existing threat, the network would be vulnerable during that time. Enabling maintenance mode allows the network to upgrade while blocking any bad actor attempting to exploit the disclosed vulnerability.
Another use case for the maintenance mode is during complex runtime migrations that would require the blockchain to migrate data during multiple blocks. Such migration could be at risk if other transactions could get included in the same blocks. Enabling maintenance mode during those blocks aims to prevent that risk.
Once the maintenance mode is activated, the runtime will automatically fail to execute balance transfers, smart contract calls, etc, (a full list can be found here: https://github.com/PureStake/moonbeam/blob/1f99e6013a8720083c11ea99c1bb90c2c6050721/runtime/moonbeam/src/lib.rs#L900Go to page https://github.com/moonbeam-foundation/moonbeam/blob/1f99e6013a8720083c11ea99c1bb90c2c6050721/runtime/moonbeam/src/lib.rs#L900). Additionally, collator nodes will refuse to add those transactions to the blocks to prevent the collection of fees for failing transactions.
Maintenance Mode can only be enacted through Moonbeam governance through a “yes” vote by at least two-thirds of the members of the Technical Committee, which currently consists of 5 members from the Moonbeam Foundation and PureStake.
- Maintenance mode does NOT change any permissions to execute privileged transactions.
- Maintenance mode does NOT change any logic of the allowed transactions.
- Maintenance mode applies to ALL accounts, with the SAME rules.
The Maintenance Mode can be deactivated by the Technical Committee, using the same process as when it was activated
Decision Process for Maintenance Mode Activation
In an emergency situation, the Technical Committee convenes to discuss the issue and evaluate the risk to the network. At least 4 members of the committee must be present. Maintenance Mode will be considered if the incident is classified as an existential threat to the network and all other options have been exhausted. At that time, a proposal to enact maintenance will be submitted to the blockchain and two-thirds of the Technical Committee (currently 4 out of 5 members) have to vote “yes” to activate maintenance.
It is anticipated that this process will be deprecated following the production deployment of Governance v2Go to page https://medium.com/polkadot-network/gov2-polkadots-next-generation-of-decentralised-governance-4d9ef657d11b features on Kusama and Polkadot, which will be incorporated into Moonbeam. Governance v2 will further decentralize decision-making on the network.
First use of Maintenance Mode – Immunefi Bug Report
On June 27, 2022, the Moonbeam Foundation received a bug reportGo to page https://moonbeam.network/blog/security-patch-for-integer-truncation-bug/ via the Immunefi bug bounty platform about a bug in the Frontier library that could lead to dangerous inconsistency between the runtime and the EVM environment. The bug could be exploited to mint unsecured assets on the network. The Frontier library is developed by the Polkadot ecosystem and maintained by Parity. It is used among numerous parachains on Polkadot that were all vulnerable to the exploit. Moonbeam developers informed Parity and affected parachain projects and decided to develop and deploy a hotfix as soon as possible.
About 4 hours after the bug disclosure, one of the affected parachain projects inadvertently shared details about the issue on its public Github repository while Moonbeam was still vulnerable. At this point, Maintenance Mode was activated to prevent any possible exploit of the still unprotected network which was increasingly likely due to the accidental disclosure. A few hours later, the bug was patched through a network runtime upgrade and Maintenance Mode ended. The network resumed normal operation.
Maintenance Mode in Response to the Nomad Bridge Hack
Reports indicated irregular transactions occurring on the Ethereum side of a smart contract utilized by Nomad, a third-party bridge application deployed on the Moonbeam network.
Because the impacts appeared to be widespread and because the root cause of the issue was not immediately known, Maintenance Mode was activated. Maintenance Mode ended and normal network operations resumed soon after the source of the vulnerability was attributed to the third-party Nomad bridge contract itself and not in the Moonbeam codebase.