On Thursday, December 9, 2021, a “critical” zero-day exploit was reported in the widely used log4j2 library.
The exploit “did not affect the Hedera network,” the platform’s developers claim while noting that “out of an abundance of caution, the Hedera team addressed the issue on Friday with the highest priority, and was able to do so without any downtime or disruption to the network.”
What Is the Issue and Why Does It Matter?
The library log4j2 contains a “powerful feature called ‘lookups'” which may be used to include system information in the logs automatically.
The update from Hedera further explained that a lookup command “can download and inject remote code into the Java Virtual Machine (JVM) process, enabling an attacker to execute remote code and gain access to the system.”
The update also mentioned that any application that “takes user supplied code and logs it can be exploited if the user sends a specially formatted string that contains lookup commands within it.”
For instance, you may check out the Impact section of this update by Randori. The diagram in that post shows “how an attacker can leverage this vulnerability.”
Here Is How Hedera Responded, the company reveals:
The version of log4j2 used by the Hedera network (2.13.x) is “susceptible to this exploit, but only if user-supplied code is logged,” the company clarified.
There are “three ways to mitigate this attack (other than never logging user input in the first place),” the firm noted.
First, upgrade the library to “a version greater than or equal to 2.15.0.”
Upgrading to this library “requires thorough code analysis and a full test/debug code cycle, which takes some time and requires downtime on the network.” Hedera further noted that they “opted not to do this for Friday December 10th because it would take too long and rushing that process could expose the network to unnecessary risk.”
Second, set the `log4j2.formatMsgNoLookups` flag “to true.”
As explained in the update, this effectively “disables the feature without requiring a full code/test/debug cycle.”
Setting this flag would “mean either downtime on the network or the node operators performing a rolling upgrade.” A rolling upgrade would “mean taking down nodes in small groups so as not to disrupt consensus and restart them, waiting for them to fully reconnect,” the Hedera Hashgraph team explained.
They added that the third option, and “what was done, update the log4j2 configuration to disable lookups within the configuration file itself.”
Changing the configuration can be “done without downtime and without rolling upgrades. Configuration file changes are automatically reapplied by log4j2 every 10 minutes,” the firm noted.
For mirror nodes, the log4j2.formatMsgNoLookups flag was “set to true in all environments and a rolling upgrade was performed.” Then the version for log4j2 “was upgraded, and a new mirror node version was released and rolled out to all environments.”
The validated changes were taken up by “modifying the log configuration in a way that is visible in our log files,” the developers noted while adding that the change was “applied to all public Hedera networks (preview networks, public Testnet, and Mainnet).”
The Hedera team is now working on “reviewing log4j2 version 2.16.0 for deployment in a future release,” the update confirmed.
Who Does This Issue Affect and What Should You Do?
Mirror Node Operators: The Hedera team “backported the lo4j2 fix to previous releases of the mirror node, from v0.42 to v0.46. It is recommended that mirror node operators upgrade as soon as possible.”
You may access the updated releases at: https://github.com/hashgraph/hedera-mirror-node/releases
Swirlds SDK Users: Options two and three in the previous section “would fix the issue for users of the Swirlds SDK,” the update noted.
Option two (setting the log4j2.formatMsgNoLookups flag to true) “helps those unable to upgrade log4j2 to version 2.16.0 or greater.”
In releases 2.10 or greater, this option “can be implemented as follows when starting the JVM”:
- Code Snippet Background
- add -Dlog4j2.formatMsgNoLookups=true
- For releases older than 2.10, check out the recommendations in the Randori blog post.
Option three (updating the log4j2 configuration to disable lookups) can be “implemented according to the Apache documentation”:
Hedera Client SDK Users: These users are “not affected, so no action is needed,” the developers confirmed.