Moonbeam Team Releases Urgent Security Patch for Integer Truncation bug

On Monday, June 27, 2022, Moonriver and Moonbeam each received an urgent upgrade through runtime 1606 to resolve a security issue responsibly disclosed by an independent whitehat hacker earlier that day (EDT). The issue has been resolved and the vulnerability is no longer exploitable on either network. The initial indication is that the bug was never maliciously exploited, and there is no known damage to the networks’ economies.

The details behind the security issue were not immediately disclosed to the public in order to both prevent exploitation on our own networks and others, and to verify that no active exploitation occurred during the period the networks were vulnerable.

About the Security Issue

On the afternoon of June 27, the Moonbeam team received a report concerning a potential security vulnerability within FrontierGo to page https://github.com/polkadot-evm/frontier (the Substrate pallet that provides core Ethereum compatibility features within the Polkadot ecosystem, which Moonbeam helped create). The bug was reported via direct outreach by pwning.ethGo to page https://twitter.com/PwningEth (who in May identified another security issueGo to page https://moonbeam.network/blog/urgent-security-patch-for-custom-precompiles/) to impacted teams and later filed as an ImmunefiGo to page https://immunefi.com/bug-bounty/moonbeamnetwork/ report.

The operations and development teams fielded this report and investigated the vulnerability. Other teams, including Astar, Parity, and Moonwell worked together with Moonbeam to assess the scope and severity of this vulnerability.

Once verified, a runtime fix was prepared and a deployment plan was put in place. However, the core vulnerability was inadvertently disclosed by a check-in to a public repository while the Moonbeam team was in the remediation process. Out of an abundance of caution, the Technical Committees of Moonbeam and Moonriver initiated the placement of the networks into maintenance modeGo to page https://moonbeam.network/blog/maintenance-mode/ to prevent any potential for misuse; during maintenance mode only upgrades and governance actions are permitted and the networks continue to produce blocks.

Once the runtime fix was executed and verified that evening, maintenance mode was deactivated and the networks resumed activity.

Vulnerability Scope

The root vulnerability was an integer truncation that bypassed sanity checks when a transfer was initiated through smart contracts. While the transfer would only happen with the truncated value, a malicious actor could have created a contract that initiated a transfer with a higher value (over 128 bits) to a Token Wrapper contract, and due to the bug, fooled the wrapper contract into believing that such a high amount of token was actually transferred and resulting in the receiving address incorrectly being credited with the high amount of wrapped tokens.

Remediation

Runtime 1606 addressed the issue by removing the truncation, leaving the saturating cast that was already in place, which prevents any overflow to be executed.

As a follow-up, a governance motion on Moonriver passed on Friday July 1st removing an invalid balance that had been generated during the period of time the teams were testing resolutions. Our thanks to the team at Web3Go for finding this issue and bringing it to our attention.

The code fix is available here https://github.com/PureStake/frontier/compare/652abf16…ca027df5Go to page https://github.com/moonbeam-foundation/frontier/compare/652abf16...ca027df5

Existing Security Measures

Code Audits

The Moonbeam team has an ongoing service relationship with two auditing firms (SR-Labs and NCC) that perform continuous, incremental, and full snapshot code audits. The codebases on both Moonriver and Moonbeam were in scope for this auditing process. Additionally, the team is looking into engaging a third firm to further expand audit coverage.

Bug Bounty Program

Last year, the Moonbeam project established an Immunefi bug bounty programGo to page https://immunefi.com/bug-bounty/moonbeamnetwork/ to encourage additional security testing on the Moonbeam codebase. This issue was sourced from a user of this program and is the second significant report related to Moonbeam. As is evident from the pair of disclosures, this program is providing exceptional value to the network and parachain ecosystem.

Internal Testing

Before every release, the Moonbeam team thoroughly tests the code for completeness and to identify any potential technical issues using both a comprehensive suite of automated tests and also a set of manual tests. This internal testing takes place over a period of more than one week, and happens across several internal test networks in addition to the public TestNet, Moonbase Alpha. Typically, releases are staggered between Moonriver and Moonbeam by approximately two weeks or more. Due to the urgent nature of this issue, an abbreviated testing process was completed and the upgrades were deployed to both networks at the same time in an effort to limit the exposure window.

Communications

Following on from the findings of the prior vulnerability disclosure, the Moonbeam team put into place new communication tools and best practices. This allowed for superior communication between teams and a smoother upgrade process across the board.

Communications and Response Timeline

Once the outreach was received and the relevant teams were online and involved, the Moonbeam team took the security report and began working on a fix.

The first priority was to address the immediate security vulnerability as promptly as possible on all impacted chains. When the vulnerability slipped into public view, the Moonbeam and Moonriver Technical Committees quickly moved to prevent any damage by initiating actions that placed the networks in maintenance mode.

In an effort to more thoroughly detail the events leading to the security patch, the Moonbeam team has provided a breakdown of the steps taken and communications sent regarding this event.

All times provided are in Eastern Daylight Time (EDT).

6/27/2022

• 2:13 PM – Moonbeam team is contacted by the Moonwell team, who have been directly contacted by the whitehat hacker. An Immunefi report is immediately requested by the Moonbeam team to establish authenticity and the necessary information to begin a response.
• 2:25 – PM The Moonbeam technical incident response team convenes a meeting with the Moonwell team representative to discuss the report. The issue is then escalated to the full business incident response team and the technical committee for Moonbeam and Moonriver.
• 2:36 PM – The whitehat requests introductions to the Parity and Astar team. Moonbeam establishes a Telegram group for intra-team discussions with the whitehat and contacts the requested teams and asks them to join.
• 3:10 PM – The whitehat submits Immunefi reports to Moonbeam and Astar.
• 3:21 PM – The Moonbeam team receives the report then reconvenes and decides to pursue a hotfix, but not to place the networks in maintenance mode.
• 4:00 PM – The issue and severity are confirmed by Parity, Moonbeam, and Astar. An update timeline is set by the Moonbeam development team, and work begins on the patch.
• 5:00 PM – Communications are sent to ecosystem teams.
• 6:13 PM – The Astar team publishes a fix for the issue in a public repository.
• 6:33 PM – The Moonbeam team notices the published code and asks the Technical Committees to place the networks into maintenance mode. Communication with key projects about maintenance mode begins.
• 6:46 PM – Maintenance modeGo to page https://moonbeam.network/blog/maintenance-mode/ is enacted through the technical committee on Moonriver and Moonbeam and external communications are initiated.
• 7:46 PM – The update is complete, CI/CD begins, followed by regression testing and deployment to multiple test networks.
• 8:05 PM – The Moonbeam dApps GUI is taken offline to prevent unexpected behavior.
• 9:40 PM – The upgrade is up for vote on Moonriver. Polkassembly cannot be used since EVM functionality is unavailable during maintenance mode.
  https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fwss.api.moonriver.moonbeam.network#/democracyGo to page https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fwss.api.moonriver.moonbeam.network#/democracy
• 10:31 PM – The Moonbeam upgrade is approved and queued with a 60 minute delay.
• 10:58 PM – Both networks are successfully upgraded and producing blocks. The technical committee votes for removing maintenance mode are submitted.
  Moonriver Technical Committee Proposal #92
  Moonbeam Technical Committee Proposal #50
• 11:05 PM – Maintenance mode is lifted and both networks resume full functionality.
• 11:49 PM – Moonwell confirms that all their services are fully restored.
• 11:50 PM – A confirmation of the upgrades is sent to Twitter, Telegram, Discord, and Reddit.

Recap

In total, the time from issue receipt to resolution and public acknowledgment was a little over ten hours. This time window includes multiple phases of testing and coordination not only with the PureStake team, Moonbeam Foundation, and the bug reporter on Immunefi, but also with the Parity team and Astar. Due to the nature of open source development, these disclosures require both speed and coordination to resolve across a large number of potentially impacted parties operating in different time zones across the globe.

We’d like to thank the Moonbeam and Moonriver ecosystem teams for their diligence and speed as we worked to resolve the issue as quickly as possible.

For more information regarding the Immunefi bug bounty program and how you can get involved, please visit their website: https://immunefi.com/bounty/moonbeamnetwork/Go to page https://immunefi.com/bug-bounty/moonbeamnetwork/

For additional questions regarding this debrief and the security report we received, please contact the team via Discord: https://discord.gg/PfpUATXGo to page https://discord.com/invite/PfpUATX