ExceptionFactory

Producing content that a reasonable developer might want to read

Evaluating Log4Shell and Apache NiFi

NiFi Security Log4j Log4Shell

2021-12-14 • 12 minute read • David Handermann

Introduction

The following evaluation is an unofficial attempt to assess usage of Log4j 2 across NiFi components. It is not possible to guarantee the security of a given installation that includes vulnerable libraries. Upgrading to patched versions remains the best approach to maintaining security. Implementing official mitigation strategies that remove or replace vulnerable components is an option for deployments that may be unable to upgrade immediately.

The binary distribution of NiFi 1.15.0 available from Apache NiFi Downloads does not include Log4j 2 libraries. A small number of optional extension components include Log4j 2, and the next release of NiFi upgrades vulnerable Log4j 2 references in these optional packages.

Background

Apache Log4j 2 is the successor to the popular Log4j framework, providing a standard API with configurable output formatting and transmission methods. Known as Log4Shell in common reporting, CVE-2021-44228 is an arbitrary code execution vulnerability in Log4j 2 message pattern formatting. The log4j-core library supports message pattern lookups using the Java Naming and Directory Interface, which provides access to both DNS and LDAP services. Writing log messages containing a JNDI lookup pattern at an enabled logging level can trigger the vulnerable execution path in Log4j 2. For Log4j 2 usage in non-standard configurations, version 2.15.0 remains vulnerable to malicious JNDI lookup patterns when using Thread Context Map information in logging. CVE-2021-45046 describes the potential problem in version 2.15.0, which version 2.16.0 resolves.

The following provides an example JNDI lookup pattern that could trigger an LDAP request to the specified address:

${jndi:ldap://localhost.localdomain/a}

Applications running a version of Java 8 prior to Update 121 are vulnerable to running arbitrary code through either RMI or LDAP, and versions prior to Update 191 have LDAP vulnerabilities described in CVE-2018-3149. Current versions of Java 8 do not allow loading arbitrary code from remote locations, but JNDI lookup support in Log4j 2 still allows DNS lookups based on log message contents. Log4j 2 provides several mitigation options based on the library version, and version 2.15.0 disables message format lookup support in the default configuration. Log4j 2 version 2.16.0 includes additional security measures that disable JNDI lookups in the default configuration as described in LOG4J2-3208 and LOG4J2-3211. Upgrading Log4j 2 to version 2.15.0 provides basic protection against the arbitrary code execution, and upgrading to version 2.16.0 provides optimal protection.

Summary

Apache NiFi application code has used SLF4J for logging with Logback as the runtime implementation since project inception. The NiFi Registry and MiNiFi subprojects follow the same pattern. Based on this approach, NiFi framework modules do not have any interaction with or dependence on Log4j 2, thus avoiding exposure to potential Log4j 2 vulnerabilities in core modules.

Although the NiFi framework does not include direct references to Log4j 2, a small number of extension components include indirect dependencies on Log4j 2 libraries. The security implications of each transitive relationship depend on the usage pattern of the associated library.

Verifying the absence of vulnerable usage is difficult as it requires an exhaustive evaluation of the Log4j 2 references in library source code. Triggering the vulnerable message converter method also requires Log4j 2 pattern processing, which does not occur for disabled loggers and levels. These factors provide a very narrow path for potential issues in NiFi extension components. With this background, reviewing areas of concern in several NiFi components provides a better understanding of the theoretical attack surface. Initial review of NiFi extension component references to Log4j 2 does not provide a clear indication of vulnerability.

Log4j 2 Overview

Understanding the Log4j 2 vulnerability requires a basic knowledge of the library architecture. The framework consists of multiple libraries, including log4j-api and log4j-core. The Log4j 2 Architecture description incorporates a diagram of core components, including the central Logger interface and the LogManager, which provides access to Logger instances.

Interface Summary

The log4j-api library contains a minimal set of components, and does not include classes vulnerable to arbitrary code execution in message pattern processing. For this reason, the presence of the log4j-api library by itself is not an indication of the Log4Shell vulnerability. NiFi Registry uses Spring Boot, which includes a transitive dependency on log4j-api through the spring-boot-starter-logging library. As described on the Spring Blog, however, Spring Boot does not use Log4j 2 as the default logging system.

Core Implementation

The log4j-core library contains a large number of configuration and implementation classes, including the MessagePatternConverter , which converts the message portion of a log into a string representation prior to writing the log to an appender. Prior to LOG4J2-3198 released in Log4j 2 2.15.0, the converter supported resolving lookup patterns, including formatted JNDI references, resulting in the Log4Shell vulnerability.

The MessagePatternConverter is just one of many implementations of the PatternConverter interface, which abstracts conversion from a pattern element such as %level to a rendered string such as DEBUG. The MessagePatternConverter handles processing for the %message pattern element, which is one of the fundamental log event elements. Pattern converters implement the supporting architecture for the PatternLayout class, which provides log event layout handling for most standard Log4j 2 configurations.

Based on this implementation architecture, log events must go through the pattern conversion process in order to trigger the vulnerability. In other words, application code that has the potential to write messages containing problematic JNDI lookups will not execute the lookup command if the logging is not enabled for the associated logger and level.

NiFi Overview

The extensible nature of NiFi processing provides opportunities to introduce an array of additional libraries. The NiFi source code repository manages these dependencies and provides a custom bundling structure known as NiFi Application Resources, or NAR files. Following the pattern of other Java bundles, a NAR is a ZIP archive that contains a manifest and one or more application resources, such as JAR or WAR files. NiFi uses a custom Java ClassLoader to extract and read NAR files at runtime, using a hierarchical strategy. This loading strategy allows NiFi to enforce a common version of shared classes at a higher level while supporting different versions of classes in specific modules.

Log4j 1 Runtime Handling

As an example of the hierarchical loading strategy, NiFi uses the log4j-over-slf4j bridge library to route Log4j 1 messages through SLF4J to Logback. Log4j 1 is a legacy library that does not follow the same interface and implementation library architecture as Log4j 2. NiFi hierarchical class loading ensures that log4j-over-slf4j classes take precedence over any instances of Log4j 1 classes present in extension modules. This approach protects NiFi against potential Log4j 1 vulnerabilities, such as CVE-2021-4104, which allows for potential arbitrary code execution when the Log4j 1 configuration includes an appender configured for JMS.

Log4j 2 References

Although Log4j 1 references in NiFi are common enough to include the SLF4J bridge library in the root library path, extension component dependencies on Log4j 2 have not prompted a similar inclusion of the Log4j 2 to SLF4J Adapter. For this reason, runtime behavior and potential issues present in Log4j 2 message conversion depend on usage patterns in referencing libraries.

Each version of NiFi includes a different set of extension component dependencies, so the first step in any evaluation is identifying the NiFi release version. The Apache NiFi Downloads provides access to a binary distribution that contains the majority of extension components available for the system. It is important to note, however, that the distribution does not contain some optional components. The Maven Central Repository contains several additional NAR files that can be downloaded and installed in addition to the binary distribution.

Reviewing the status of several recent NiFi releases should provide a general understanding of the issues involved. Older versions may have different vulnerabilities, so the best starting point is the latest available version.

The presence of log4j-core does not necessarily indicate an actual vulnerability in associated components, but only the potential based on possible usage patterns.

NiFi 1.15.0

The binary distribution of NiFi 1.15.0 released on 2021-11-07 does not include any Log4j 2 libraries. NiFi installations using the binary distribution without optional NAR files do not have the log4j-core library and are not vulnerable to CVE-2021-44228.

NiFi 1.15.0 also does not include any references to Log4j 1 libraries.

Several optional NAR files available from the Maven Central Repository include log4j-core:

NiFi 1.14.0

The binary distribution of NiFi 1.14.0 released on 2021-07-14 includes log4j-core in the binary distribution within the following NAR files:

The following optional NAR files also include log4j-core:

Several standard and optional NAR files include Log4j 1, but the NiFi hierarchical loading strategy gives precedence to SLF4J bridge classes and avoids loading Log4j 1 logging components.

Log4j 2 External Usage Patterns

Based on Log4j 2 references in NiFi extension components, vulnerable usage depends on several factors. In order for NiFi to invoke vulnerable Log4j 2 message pattern processing, it is not sufficient for the NAR file containing log4j-core to be available in the library directory. NiFi must configure and trigger the extension component in order to support potential invocation of pattern layout handling.

In addition, the external library must invoke Log4j 2 with a method that could contain a problematic JNDI lookup pattern. The external library must also call a Log4j 2 Logger that will format and write a message at the specified log level. All of these conditions make it difficult to rule out the possibility of vulnerable code paths, but these conditions also illustrate the limited nature of potential threats.

NiFi Atlas NAR

The nifi-atlas-nar includes log4j-core starting in NiFi 1.15.0 due to a dependency on atlas-notification version 2.1.0. The project added this dependency to atlas-notification in version 2.1.0 as part of changes described in ATLAS-3427.

Reviewing the associated changes and current architecture, however, Atlas uses a Log4j 1 appender to write failed notification messages, as opposed to Log4j 2. Most Atlas classes use the SLF4J Logger interface for standard application logging, which routes to Logback in NiFi according to the standard project configuration. Based on this usage, NiFi Atlas components do not appear to invoke vulnerable Log4j 2 message pattern processing.

NiFi Druid Controller Service API NAR

The nifi-druid-controller-service-api-nar includes log4j-core along with several other Log4j 2 libraries. NiFi 1.6.0 included the initial version of Apache Druid components according to NIFI-4428. Druid 0.9.1 and following implement logging using a wrapper around the SLF4J Logger, with Log4j 2 as the default runtime implementation.

SLF4J uses the first logging implementation library found in the class path, and also writes warning messages when detecting multiple implementation classes. Based on the NiFI hierarchical loading strategy, Druid logging calls to SLF4J should be routed through the standard NiFi Logback configuration instead of Log4j 2.

NiFi Elasticsearch 5 NAR

The nifi-elasticsearch-5-nar includes log4j-core starting in NiFi 1.1.0 as described in NIFI-3011. The initial nifi-elasticsearch-5-processors library included a direct dependency on log4j-core together with the Elasticsearch 5.0.1 transport client library. The Elasticsearch 5.0.1 library includes optional dependencies for log4j-api and log4j-core to support debugging of requests and responses when communicating with an Elasticsearch server.

Without any direct references to Log4j 2 in NiFi Elasticsearch 5 components, and with debugging disabled by default, NiFi usage does not include Log4j 2 during standard processing. NiFi 1.12.0 changed the binary distribution to exclude the nifi-elasticsearch-5-nar, requiring a separate download from the Maven Central Repository to load Elasticsearch 5 components.

NiFi Elasticsearch Client Service NAR

The nifi-elasticsearch-client-service-nar and nifi-elasticsearch-restapi-nar files both include log4j-core in NiFi 1.14.0. Changes described in NIFI-8001 updated the version of Elasticsearch libraries from 5.6.16 to 7.10.2 and changed dependencies, which eliminated transitive references to Log4j 2 in NiFi 1.15.0. The elasticsearch-rest-high-level-client , present in NiFi 1.14.0 and earlier, depends on the core elasticsearch library and includes references to log4j-core. The elasticsearch-rest-client referenced in NiFi 1.15.0 has a much smaller dependency tree and does not include any dependencies on Log4j 2.

As indicated by the dependency change from the Elasticsearch High Level REST Client , which Elastic deprecated in version 7.15.0, NiFi components leverage the Low Level REST Client for service and processor implementation. The Low Level REST Client does not use Log4j 2 in standard runtime operations, which avoids vulnerable code paths in message formatting.

NiFi Hive 3 NAR

The nifi-hive3-nar includes log4j-core together other Log4j 2 and Log4j 1 libraries. NiFi Hive 3 components reference log4j-core through a transitive dependency in hive-exec version 3.1.2. Hive 3 uses the SLF4J Logger for logging in component classes, and includes Log4j 2 configuration utilities to provide runtime handling.

With NiFi providing Logback in the primary class loader as the logging implementation of SLF4J, logging messages in Hive 3 components should be routed through the Logback library. Although Hive 3 execution components include code to initialize a Log4j 2 configuration, NiFi usage of Hive 3 components does not appear to exercise those code paths.

NiFi Ranger NAR

The nifi-ranger-nar includes log4j-core through a transitive dependency in the Apache Ranger ranger-plugins-audit library starting with version 2.1.0. The ranger-plugins-audit library introduced the dependency on log4j-core as part of changes to support Elasticsearch 7.6.0 described in RANGER-2834. NiFi upgraded the Ranger client library to version 2.1.0 in NiFi 1.12.1 and 1.13.0 according to NIFI-7798. Prior to those versions of NiFi, the nifi-ranger-nar did not include the log4j-core library.

Ranger usage of Log4j 2 appears limited to sending audit information to Elasticsearch. Based on other analysis of Elasticsearch client libraries, NiFi usage of Ranger auditing does not appear to leverage Log4j 2 message pattern processing.

Conclusion

Some reports have already described the Log4Shell vulnerability as one of the worst issues in computing history. The importance of logging, the widespread usage of Log4j 2, and the relative simplicity of exploitation have combined to make remediation essential across all impacted products and services. Whether a product contains direct or indirect references to Log4j 2, understanding the potential problem scope is key to addressing security concerns.

The small number of optional NiFi extension components referencing Log4j 2 provide a very limited window for problematic log message processing. The SLF4J and Logback architecture in NiFi, together with hierarchical class loading, narrow the scope of potential vulnerability significantly.