Implementing Apache NiFi Support for Sensitive Dynamic Properties
Introduction
Apache NiFi provides an extensible approach to data processing through a wide range of components, configuration options, and extension points. In addition to predefined settings, a number of standard Processors and Controller Services support user-defined properties, allowing operators to provide configuration values not enumerated in the common list of properties. This strategy works well for components with a large or unbounded number of potential configuration options.
Earlier versions of Apache NiFi required developers to indicate whether a user-defined property should be considered sensitive, resulting in different approaches to support configurable sensitive status based on property naming. Apache NiFi 1.17.0 introduced standard framework support for sensitive user-defined properties, providing a clear approach for both component developers and flow operators.
Development Strategy
Framework support for sensitive dynamic properties began with a clear feature proposal and a summary issue description outlining multiple implementation tasks. This approach provided the opportunity to focus on various aspects of the implementation of the course of multiple pull requests. Most of the framework changes are not visible to the user, but involved careful adjustments to core components. Although it is difficult to consider the full scope of concerns when designing new capabilities, the implementation process for sensitive dynamic properties illustrates the value of initial design and incremental development.
User Interface Changes
Sensitive dynamic property support included several minor adjustments to user interface dialogs and component documentation. These changes provide an indication of whether a component supports sensitive user-defined properties, and a straightforward approach to selecting the desired status.
Add Property Changes
The Properties section of each NiFi
component configuration includes a button for adding a user-defined property in the upper right corner. Pressing the
button opens the Add Property dialog, which includes the Property Name field.
NiFi 1.17.0 adds a new Sensitive Value field with radio button options for Yes or No.
Components that support sensitive dynamic properties allow Yes or No to be selected for the Sensitive Value field.
Components that do not support this feature have the values disabled, with No selected, indicating standard behavior.
Entering a Property Name, selecting Yes, and pressing OK adds the specified sensitive property to the component
configuration. After entering a value, the Value column displays Sensitive value set, following the standard
convention for sensitive property values. The framework encrypts the sensitive value and stores the encoded string in
the flow configuration.
The user interface does not support changing sensitive status after the initial selection of Yes or No for the
Sensitive Value field. Deleting a dynamic property allows the property to be redefined using a different sensitive
status. Although deleting and recreating a dynamic property involves several steps, this follows the same user
interaction strategy required to change the name of an existing dynamic property.
Parameter Context Integration
The introduction of sensitive dynamic properties enables direct references to
Parameters having the Sensitive Value
field set to Yes in a Parameter Context. The Parameter Context configuration requires alignment between the sensitive
status of a parameter value and the sensitive status of a component property. For this reason, prior to NiFi 1.17.0,
most dynamic properties could not reference sensitive parameter values. With the release of NiFi 1.17.0, components
enabled for sensitive dynamic properties support complete integration with range of Parameter Context configuration
options.
Documentation Adjustments
The usage documentation generated for each component includes a new label indicating support for sensitive dynamic properties. Components that do not support any dynamic properties do not have any changes.
Components that support dynamic properties have a section header named Dynamic Properties, under which there is a
label reading Supports Sensitive Dynamic Properties, with a value indicating Yes or No. This label specifies
expected behavior when adding a new user-defined property.
Supported Components
NiFi 1.17.0 updated multiple components to support sensitive dynamic properties, including selected Controller Services, Processors, and Reporting Tasks.
Supported Controller Services
NiFi 1.17.0 enabled sensitive dynamic properties for the following Controller Services:
Configuring a database connection can require custom properties, such as a password to read a protected keystore.
Although DBCPConnectionPool supported sensitive dynamic properties through a property name prefixed with the word
SENSITIVE, the framework implementation provides a standardized approach. Current configurations using prefixed
property names continue to work with NiFi 1.17.0, but should be migrated.
Supported Processors
The following Processors support sensitive dynamic properties in NiFi 1.17.0:
It is important to note that selecting Yes for the Sensitive Value field enables property encryption when persisting
the flow configuration, but does not guarantee secure handling in components with custom scripting. For components that
support custom scripting, it is the responsibility of the flow designer to ensure that custom code handles sensitive
values with sufficient security.
Supported Reporting Tasks
The following Reporting Task supports sensitive dynamic properties in NiFi 1.17.0:
Flow Configuration Details
Sensitive dynamic property support builds on existing capabilities for encrypting and persisting flow configuration settings.
The serialized flow configuration persisted in flow.json.gz and flow.xml.gz stores sensitive dynamic properties
using the same encryption strategy defined for standard sensitive property values. Based on the Sensitive Properties Key
and Sensitive Properties Algorithm settings in nifi.properties, the framework encrypts property values and stores the
bytes using hexadecimal encoding.
Flow Definitions
Following the standard framework approach for sensitive values, an exported Flow Definition does not include direct values for sensitive dynamic properties. This applies to both Flow Definition downloads and flows controlled through NiFi Registry.
Exported Flow Definitions include descriptors for sensitive dynamic properties, with true specified in the sensitive
field, which may be useful to external services evaluating all possible properties in a particular flow. Referencing a
sensitive parameter value from a configured Parameter Context allows the framework to export and import Flow Definitions
without losing the definition of sensitive dynamic properties. When a sensitive dynamic property does not reference a
Parameter Context, however, the framework ignores the sensitive property descriptor because it is unable to provide a
value for the property. This highlights the importance of using Parameter Contexts when designing Flow Definitions for
version control or transmission between different deployments.
Component Implementation
Implementing support for sensitive dynamic properties in specific components involves several elements. The implementation requires both a specific method definition and a behavior annotation on the component class. The Apache NiFi Developer Guide includes a section on sensitive dynamic properties as part of the steps involved in exposing processor properties.
Required Component Method
Enabling support for dynamic properties requires implementing the
getSupportedDynamicPropertyDescriptor()
method defined in the AbstractConfigurableComponent class. The method must return a PropertyDescriptor based on the
supplied property name, indicate dynamic status, and define minimum validation requirements.
The following code snippet provides a sample implementation of the required method:
@Override
protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String propertyName) {
return new PropertyDescriptor.Builder()
.name(propertyName)
.dynamic(true)
.addValidator(Validator.VALID)
.build();
}
The default value of the sensitive() builder method is false, so it does not need to be specified.
Setting the value of the sensitive() builder method to true on the PropertyDescriptor instructs the
framework to protect all dynamic properties. This approach is not compatible with user-defined sensitive status, and
should be avoided.
Required Component Annotation
To support sensitive dynamic properties, a component class must have the
SupportsSensitiveDynamicProperties
annotation at the class level. When a component implements getSupportedDynamicPropertyDescriptor() and has the
SupportsSensitiveDynamicProperties annotation, the framework allows operators to configure the Sensitive Value
status of a user-defined property.
This implementation strategy describes a contract between the component and the user. Components annotated to support sensitive dynamic properties must avoid certain activities, such as logging dynamic property values. Existing code should be reviewed prior to enabling support for sensitive dynamic properties to ensure secure property handling.
Conclusion
Support for sensitive dynamic properties implements several features requested over multiple years. This capability provides an additional level of configuration security for various flow designs, improving integration with both internal framework features and external services. With a straightforward API and clear user interface, framework support for sensitive dynamic properties enhances a number of standard components, provides new options for custom development, and enables more secure flow design patterns.