Moonbeam Team Releases Urgent Security Patch for Custom Precompiles

Updated on June 21, 2022

On Friday, May 27, 2022, Moonriver and Moonbeam both received urgent upgrades through runtimes 1503 and 1504 in order to resolve a security issue that was responsibly disclosed by an independent white-hat hacker earlier that morning (ET). The issue has been resolved and the vulnerability is no longer exploitable on either network. The initial indication is that the bug was never exploited, though the team will continue to investigate.

The details behind the security issue were not immediately disclosed to the public in order to both prevent exploitation on our own networks and to allow sufficient time to notify other parachainsGo to page https://moonbeam.network/education/what-is-a-parachain/ that we are aware are using the same code so they could patch their own instances. We have confirmed that these teams have verified that they are no longer impacted, and are now able to provide further details on the emergency upgrade(s).

About the Security Issue

On the morning of May 27, the Moonbeam team received an ImmunefiGo to page https://immunefi.com/bug-bounty/moonbeamnetwork/ bug report concerning a potential security vulnerability within FrontierGo to page https://github.com/polkadot-evm/frontier (the Substrate pallet that provides core Ethereum compatibility features within the Polkadot ecosystem, which Moonbeam helped create). The operations and development teams immediately fielded this report and investigated the vulnerability. Once verified, a fix was prepared and a deployment plan was put in place. The Parity team was also notified and began outreach to several parachain teams.

Runtime 1503 was deployed and live on Moonbeam and Moonriver by that afternoon. While this runtime did remove the vulnerability, it introduced an unintended bug which was patched later in the evening with runtime 1504. The issue was resolved in its entirety and the networks were operating as expected by 9pm ET — 12 hours after receiving the initial report.

The issue was reported by white hat hacker pwning.ethGo to page https://twitter.com/i/flow/login?redirect_after_login=%2FPwningEth, who has been awarded the maximum reward amount through the Immunefi bug bounty program. It is estimated the vulnerability could have impacted up to $100M in funds.

Vulnerability Scope

The vulnerability concerns calls to non-standard Ethereum precompiles. Those are addresses allowing the EVM, through smart contracts, access to some of Moonbeam’s core features (like our XC-20, staking, and democracy pallets) that do not exist in the base EVM. Using a DELEGATECALL, a malicious smart contract could access the precompile storage of another party via a callback.

This is not a problem for typical users, as it would require them to send a transaction to the malicious smart contract. However, it is an issue for other smart contracts allowing arbitrary calls to external smart contracts. For example, this is the case of some smart contracts allowing callbacks. In those situations, a malicious user could make a DEX execute a call to the malicious smart contract that would be able to access the precompiles pretending to be the DEX and possibly transfer its balance to any other address.

Remediation

The runtime 1503Go to page https://github.com/moonbeam-foundation/moonbeam/releases/tag/runtime-1503 quickly disabled all DELEGATECALL to prevent the vulnerability from being exploited.

The runtime 1504Go to page https://github.com/moonbeam-foundation/moonbeam/releases/tag/runtime-1504 re-established DELEGATECALL for Ethereum standard precompiles and for smart contracts (while keeping it disabled for custom precompiles)

In between those two runtime upgrades, transactions to smart contracts relying on DELEGATECALL failed and were marked as reverted.

The fixed code is available here: https://github.com/PureStake/moonbeam/pull/1551/files#diff-1404c7c10c00572e96e647084d96576f592a2c18e70f53cdcfec2daa98adb60cR119Go to page https://github.com/moonbeam-foundation/moonbeam/pull/1551/files#diff-1404c7c10c00572e96e647084d96576f592a2c18e70f53cdcfec2daa98adb60cR119

Existing Security Measures

Code Audits

The Moonbeam team has an ongoing service relationship with two auditing firms (SR-Labs and NCC) that perform continuous, incremental, and full snapshot code audits. The codebases on both Moonriver and Moonbeam were in scope for this auditing process. Additionally, the team is looking into engaging a third firm to further expand audit coverage.

Bug Bounty Program

Last year, the Moonbeam project established an Immunefi bug bountyGo to page https://immunefi.com/bug-bounty/moonbeamnetwork/ program to encourage additional security testing on the Moonbeam codebase. This issue was submitted through that program and was the first significant report related to Moonbeam. As is evident from this disclosure, this bug bounty program has already proven to be extremely valuable in maintaining a secure codebase.

Internal Testing

Before every release, the Moonbeam team thoroughly tests the code for completeness and to identify any potential technical issues using both automated and manual tests. This internal testing takes place over a period of more than one week, and happens across several internal test networks in addition to the public TestNet, Moonbase Alpha. Typically, releases are staggered between Moonriver and Moonbeam by approximately two weeks or more. Due to the urgent nature of this issue, an abbreviated testing process was completed and the upgrades were deployed to both networks at the same time in an effort to limit the exposure window.

New Security Measures as a Result of This Event

New Automated Testing Process in Development

While there is already a comprehensive CI (continuous integration) process in place for publicly available code, the Moonbeam team plans to create an additional private build and CI stack to enable rapid releases in the event of sensitive security fixes that need to be protected from public disclosure. This will aid the team in preventing technical issues from manifesting during fast rollouts, helping ensure that new issues are not inadvertently introduced.

Additionally, the team may add delays between the Moonriver and Moonbeam patches when appropriate and where possible, depending on the severity of the issue.

Communications

The team is also working to develop additional private and public disclosure channels to more quickly acknowledge issues, share information, and (when it’s safe to do so) share details for the sake of transparency and learning.

Communications and Response Timeline

Once the Immunefi notification was received, the Moonbeam team quickly fielded the security report and began working on a fix. The first priority was to address the immediate security vulnerability as promptly as possible on both chains.

Once the fix was deployed and the Moonbeam/Moonriver networks were no longer at risk, the team began providing additional details on the issue with ecosystem teams and other parachains using the same Frontier pallet. Over the weekend, we worked with these other parachain teams to determine the extent of their exposure and to recommend a remediation path before the bug is disclosed. We were unable to disclose the full details until it was confirmed that these chains had sufficient time to check the report and issue a fix if necessary.

In an effort to more thoroughly detail the events leading to the security patch, the Moonbeam team has provided a breakdown of the steps taken and communications sent regarding this event.

All times provided are in Eastern Daylight Time (EDT).

5/27/2022

8:54 AM: The Moonbeam Foundation receives two critical bug reports through Immunefi which could lead to loss of funds on Moonbeam and Moonriver.

9:00 AM: Moonbeam operations and engineering teams meet to review the reports and confirm their severity and potential impact.

10:31 AM: The reported findings are confirmed and are acknowledged as high severity. The engineering team begins working on a patch.

11:51 AM: A preliminary runtime hotfix is prepared and testing of the patch begins.

1:45 PM: A hotfix rollout plan is proposed that will upgrade both Moonriver and Moonbeam simultaneously following a successful test on the Moonbase Alpha TestNet in an effort to reduce the vulnerability window.

2:00 PM: Following a council vote, both upgrades were approved and fast-tracked through democracy. Following a 30-minute democracy voting period and a 60-minute wait time until enactment, the upgrades will be live.

2:19 PM: Upgrade votes go into democracy and are voted upon.
https://moonriver.polkassembly.network/referendum/95Go to page https://moonriver.polkassembly.network/referendum/95
https://moonbeam.polkassembly.network/referendum/42Go to page https://moonbeam.polkassembly.network/referendum/42

3:53 PM: Both Moonbeam and Moonriver are successfully upgraded to runtime 1503.

4:00 PM: Following the vulnerability patch, the Moonbeam Foundation initiates a response to the bug reporter on Immunefi confirming the vulnerability and the subsequent patch.

4:25 PM: Receive first report of technical issue with smart contracts deployed on Moonriver/Moonbeam.

4:40 PM: Engineering team confirms a new technical (non-security) bug introduced by runtime 1503 that interferes with delegate calls, impacting DeFi projects on both networks. The team begins preparing a second hotfix to resolve the new issue.

5:37 PM: The Moonbeam team notifies the Parity team regarding the security issue and its resolution. The Parity team begins notifying other potentially impacted teams.

6:00 PM: Discord and Telegram communities are notified of the interruption to contract calls on Moonbeam and Moonriver and that a hotfix will be deployed in the next few hours.

6:50 PM: Council votes on the second hotfix, runtime 1504.

7:14 PM: Democracy vote is open for the second hotfix.
https://moonriver.polkassembly.network/referendum/96Go to page https://moonriver.polkassembly.network/referendum/96
https://moonbeam.polkassembly.network/referendum/43Go to page https://moonbeam.polkassembly.network/referendum/43

7:45 PM: The referenda pass, and the upgrade is scheduled for both Moonbeam and Moonriver.

8:35 PM: A security advisory announcement is issued in a private Element group to other parachain teams. The Moonbeam team also begins direct outreach to other parachain teams using the same code to ensure the vulnerability is patched as soon as possible.

8:55 PM: Moonbeam and Moonriver are upgraded to runtime 1504 and are operating as expected.

9:56 PM: Several DeFi projects on Moonbeam confirm that the issue has been resolved and smart contracts are operating as expected.

10:04 PM: A confirmation of the upgrades is sent out on Twitter, Telegram, and Discord.

5/28/2022

9:00 AM: Team confirms that other potentially impacted parachain teams we are aware of have either released a patch or have identified that they are not vulnerable to the exploit.

5/29/2022

11:30 AM: A new trace image has been prepared and tested on both Moonbeam and Moonriver, and is ready for use. A notification is sent to impacted project teams, asking them to upgrade.

Recap

In total, the time from issue receipt to resolution and public acknowledgment was a little over twelve hours. This time window includes multiple phases of testing and coordination not only with the PureStake team, Moonbeam Foundation, and the bug reporter on Immunefi, but also with the Parity team and other parachains operating in the ecosystem. Due to the nature of open source development, these disclosures require both speed and coordination to resolve across a large number of potentially impacted parties operating in different time zones across the globe.

We’d like to thank the Moonbeam and Moonriver project teams for their diligence and speed as we worked to resolve the issue as quickly as possible.

For more information regarding the Immunefi bug bounty program and how you can get involved, please visit their website: https://immunefi.com/bounty/moonbeamnetwork/Go to page https://immunefi.com/bug-bounty/moonbeamnetwork/

For additional questions regarding this debrief and the security report we received, please contact the team via Discord: https://discord.gg/PfpUATXGo to page https://discord.com/invite/PfpUATX