What We All Learned From Log4J2
On Thursday, December 9, a few minutes before midnight, information regarding a serious vulnerability in the Log4J2 library hit the IT world.
The preliminary information announced in the subsequent morning drew a very bleak picture. The attack vector itself is similar to the ‘format string’ (specific for C applications) popular in the 90s and the still popular stored XSS attack. The vulnerability has additional properties – it easily allows for infrastructure penetration, network mapping, and remote execution of malicious code.
Such exploits are not unknown in the IT world, and the best teams always know how to respond effectively. We wanted to use this as a good example of how – and why – that should be put into practice; what the exploit was, how you can protect yourself and how we responded…
What is Log4j2?
This library is used in java applications to log internal messages. This does not mean that environments based on .NET applications, PHP, or other programming languages are completely safe, because they can very often have components hidden deep in the infrastructure, but based on Java – e.g. log processing or reporting mechanisms.
As an example from something we were personally able to identify, we can reference an e-commerce site based on Magento. Potentially unrelated to Java, and not using Log4j2, right?
However, our rapid response team managed to remotely execute code from inside the deeply hidden infrastructure via a properly crafted order in the online store. The cause was a logstash component that used Log4j2 to process service logs, including placed orders.
The vulnerability allows for the remote execution of arbitrary code and is very easy to exploit. We can expect it to be widely exploited in a short time. Less than an hour and a half after the code exploiting the vulnerability was published on Twitter, there were already attempts by hackers to exploit the vulnerability and install cryptocurrency-mining code on the hijacked servers. We will certainly continue to hear about Log4j2 and the affected systems for weeks to come.
How Do You Protect Yourself From Events Like Log4j2?
With multiple, often multi-vendor systems under your care, how do you react in time and not suffer severe losses? There are at least a few ways to arm yourself:
Maintain a Rapid Response Team
A good IT team – at least the one responsible for maintaining systems – monitors portals and feeds that publish information about significant threats on an ongoing basis.
As a prerequisite, such a team must be competent in the technologies in which the systems are built, and must have the time to “patch” the systems in an instant.
Unfortunately, many companies lack such a time reserve, as well as efficient processes to quickly identify all the places that may be vulnerable to a given exploit. If this is the case, it is worth thinking about taking inventory of your own resources and running some scenarios like “how quickly can we find all the systems vulnerable to this?”
After all, it’s best to be on time before someone else does it for you…
Protecting Systems With a Web Application Firewall
When it comes to applications that we enable to access the internet, hiding them behind a Web Application Firewall (WAF) often helps a lot, especially when it’s coming from an efficient provider. Fortinet, AWS, F5, and Checkpoint are just a few of those that have responded quickly and many others have published rules and instructions for securing your systems using their appliances.
Even so, this often still requires keeping up to date and performing actions on the firewall itself but compared to quickly “patching” many potentially compromised systems, it’s quite an improvement.
Entrusting System Care to Professionals
On the client-side, system security is typically handled by IT teams who divide their day-to-day responsibilities between maintaining systems and implementing new projects. The same teams run the firewall, which is not always up-to-date, and consequently, their typical actions do not provide sufficient protection in case of a threat.
Therefore, to better guarantee a quick response to potential threats at a higher rate, you should consider placing IT system security issues in the hands of the technology partner’s dedicated teams. These teams have many similar systems under their care, and their daily work is to “catch” serious threats and react appropriately to their severity. Remember that the security of systems has a direct impact on business.
So… How Did We Deal With Log4j2?
In all three of the above approaches, a rapid response is essential. When teams are focused on prioritizing security and critical vulnerabilities, rather than adapting in an ad-hoc manner, your business is significantly more secure. As an example, here’s how our own rapid response team responded to the situation:
- Within an hour we managed to set up a mechanism for quickly verifying the vulnerability: a crafted DNS server with a low negative TTL zone, logging all queries and always returning the same IP address belonging to the traffic logging honeypot. A quick component verification instruction was prepared which, in simple terms, consisted of injecting a payload and verifying that it would be logged by the honeypot. The instruction was delivered to the penetration testers.
- Then the vulnerability was validated and confirmed, which allowed us to initiate a full Emergency Response.
- Development teams were deployed to patch systems and components that were known to be vulnerable.
- Additional security measures were implemented before the update was completed to extend firewalls and WAFs with rules to block payloads and log such events.
- A mechanism for scanning servers was created in order to identify components using Log4j2 and secure them in an ad-hoc mode. Running in the background, this allowed us to secure the infrastructure components that the development teams had not yet managed to reach.
As soon as it was possible, we had launched several independent response paths to the disclosed threat, and the pre-and post-tests allowed us to be sure about the effectiveness of the actions taken. Soon, on-call administrators were noticing attacks against this vulnerability, which were detected and mitigated by WAF in environments where such a component is used.
“Does my business need additional security?”
If such a question crosses your mind – we invite you to talk to our team. We will help you identify threats and recommend actions that can increase the security of the systems on which your company operates.
