ExceptionFactory

Producing content that a reasonable developer might want to read

Implementing Apache NiFi Support for Sensitive Dynamic Properties

NiFi Security Encryption

2022-08-02 • 7 minute read • David Handermann

Introduction

Apache NiFi provides an extensible approach to data processing through a wide range of components, configuration options, and extension points. In addition to predefined settings, a number of standard Processors and Controller Services support user-defined properties, allowing operators to provide configuration values not enumerated in the common list of properties. This strategy works well for components with a large or unbounded number of potential configuration options.

Earlier versions of Apache NiFi required developers to indicate whether a user-defined property should be considered sensitive, resulting in different approaches to support configurable sensitive status based on property naming. Apache NiFi 1.17.0 introduced standard framework support for sensitive user-defined properties, providing a clear approach for both component developers and flow operators.

Development Strategy

Framework support for sensitive dynamic properties began with a clear feature proposal and a summary issue description outlining multiple implementation tasks. This approach provided the opportunity to focus on various aspects of the implementation of the course of multiple pull requests. Most of the framework changes are not visible to the user, but involved careful adjustments to core components. Although it is difficult to consider the full scope of concerns when designing new capabilities, the implementation process for sensitive dynamic properties illustrates the value of initial design and incremental development.

User Interface Changes

Sensitive dynamic property support included several minor adjustments to user interface dialogs and component documentation. These changes provide an indication of whether a component supports sensitive user-defined properties, and a straightforward approach to selecting the desired status.

Add Property Changes

The Properties section of each NiFi component configuration includes a button for adding a user-defined property in the upper right corner. Pressing the button opens the Add Property dialog, which includes the Property Name field.

NiFi 1.17.0 adds a new Sensitive Value field with radio button options for Yes or No.

Components that support sensitive dynamic properties allow Yes or No to be selected for the Sensitive Value field. Components that do not support this feature have the values disabled, with No selected, indicating standard behavior.

Entering a Property Name, selecting Yes, and pressing OK adds the specified sensitive property to the component configuration. After entering a value, the Value column displays Sensitive value set, following the standard convention for sensitive property values. The framework encrypts the sensitive value and stores the encoded string in the flow configuration.

The user interface does not support changing sensitive status after the initial selection of Yes or No for the Sensitive Value field. Deleting a dynamic property allows the property to be redefined using a different sensitive status. Although deleting and recreating a dynamic property involves several steps, this follows the same user interaction strategy required to change the name of an existing dynamic property.

Parameter Context Integration

The introduction of sensitive dynamic properties enables direct references to Parameters having the Sensitive Value field set to Yes in a Parameter Context. The Parameter Context configuration requires alignment between the sensitive status of a parameter value and the sensitive status of a component property. For this reason, prior to NiFi 1.17.0, most dynamic properties could not reference sensitive parameter values. With the release of NiFi 1.17.0, components enabled for sensitive dynamic properties support complete integration with range of Parameter Context configuration options.

Documentation Adjustments

The usage documentation generated for each component includes a new label indicating support for sensitive dynamic properties. Components that do not support any dynamic properties do not have any changes.

Components that support dynamic properties have a section header named Dynamic Properties, under which there is a label reading Supports Sensitive Dynamic Properties, with a value indicating Yes or No. This label specifies expected behavior when adding a new user-defined property.

Supported Components

NiFi 1.17.0 updated multiple components to support sensitive dynamic properties, including selected Controller Services, Processors, and Reporting Tasks.

Supported Controller Services

NiFi 1.17.0 enabled sensitive dynamic properties for the following Controller Services:

Configuring a database connection can require custom properties, such as a password to read a protected keystore. Although DBCPConnectionPool supported sensitive dynamic properties through a property name prefixed with the word SENSITIVE, the framework implementation provides a standardized approach. Current configurations using prefixed property names continue to work with NiFi 1.17.0, but should be migrated.

Supported Processors

The following Processors support sensitive dynamic properties in NiFi 1.17.0:

It is important to note that selecting Yes for the Sensitive Value field enables property encryption when persisting the flow configuration, but does not guarantee secure handling in components with custom scripting. For components that support custom scripting, it is the responsibility of the flow designer to ensure that custom code handles sensitive values with sufficient security.

Supported Reporting Tasks

The following Reporting Task supports sensitive dynamic properties in NiFi 1.17.0:

Flow Configuration Details

Sensitive dynamic property support builds on existing capabilities for encrypting and persisting flow configuration settings.

The serialized flow configuration persisted in flow.json.gz and flow.xml.gz stores sensitive dynamic properties using the same encryption strategy defined for standard sensitive property values. Based on the Sensitive Properties Key and Sensitive Properties Algorithm settings in nifi.properties, the framework encrypts property values and stores the bytes using hexadecimal encoding.

Flow Definitions

Following the standard framework approach for sensitive values, an exported Flow Definition does not include direct values for sensitive dynamic properties. This applies to both Flow Definition downloads and flows controlled through NiFi Registry.

Exported Flow Definitions include descriptors for sensitive dynamic properties, with true specified in the sensitive field, which may be useful to external services evaluating all possible properties in a particular flow. Referencing a sensitive parameter value from a configured Parameter Context allows the framework to export and import Flow Definitions without losing the definition of sensitive dynamic properties. When a sensitive dynamic property does not reference a Parameter Context, however, the framework ignores the sensitive property descriptor because it is unable to provide a value for the property. This highlights the importance of using Parameter Contexts when designing Flow Definitions for version control or transmission between different deployments.

Component Implementation

Implementing support for sensitive dynamic properties in specific components involves several elements. The implementation requires both a specific method definition and a behavior annotation on the component class. The Apache NiFi Developer Guide includes a section on sensitive dynamic properties as part of the steps involved in exposing processor properties.

Required Component Method

Enabling support for dynamic properties requires implementing the getSupportedDynamicPropertyDescriptor() method defined in the AbstractConfigurableComponent class. The method must return a PropertyDescriptor based on the supplied property name, indicate dynamic status, and define minimum validation requirements.

The following code snippet provides a sample implementation of the required method:

@Override
protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String propertyName) {
    return new PropertyDescriptor.Builder()
        .name(propertyName)
        .dynamic(true)
        .addValidator(Validator.VALID)
        .build();
}

The default value of the sensitive() builder method is false, so it does not need to be specified.

Setting the value of the sensitive() builder method to true on the PropertyDescriptor instructs the framework to protect all dynamic properties. This approach is not compatible with user-defined sensitive status, and should be avoided.

Required Component Annotation

To support sensitive dynamic properties, a component class must have the SupportsSensitiveDynamicProperties annotation at the class level. When a component implements getSupportedDynamicPropertyDescriptor() and has the SupportsSensitiveDynamicProperties annotation, the framework allows operators to configure the Sensitive Value status of a user-defined property.

This implementation strategy describes a contract between the component and the user. Components annotated to support sensitive dynamic properties must avoid certain activities, such as logging dynamic property values. Existing code should be reviewed prior to enabling support for sensitive dynamic properties to ensure secure property handling.

Conclusion

Support for sensitive dynamic properties implements several features requested over multiple years. This capability provides an additional level of configuration security for various flow designs, improving integration with both internal framework features and external services. With a straightforward API and clear user interface, framework support for sensitive dynamic properties enhances a number of standard components, provides new options for custom development, and enables more secure flow design patterns.