Configuring Apache NiFi Repository Encryption
Background
Persistent repositories are a core capability of most Apache NiFi deployments. Although concepts such as stateless flow execution and volatile storage meet certain use cases, persistent access to content and metadata protects against data loss and supports handling information beyond the size of available memory. With the potential for storing sensitive information over an extended duration, some environments require additional data protection.
Recognizing that encryption at the operating system level is not always an option, NiFi introduced initial support for encrypted provenance repositories in version 1.2.0. With the initial release, NiFi Project Management Committee member Andy LoPresto provided a thorough description of Encrypted Provenance Repositories on the project blog. NiFi version 1.10.0 added an encrypted content repository implementation, 1.11.0 included an encrypted FlowFile repository, and 1.14.0 introduced an encrypted swap manager for FlowFiles. Each repository implementation required separate configuration properties, including the specification of encryption key sources. This approach followed the flexibility of general repository settings, but also involved potential duplication of encryption key properties. With the goal of simplifying the configuration, the core concepts provided a strong starting point for developing a streamlined implementation.
Introduction
Configuring repository encryption in Apache NiFi involves provisioning a source of key material and defining repository
properties. Prior to version 1.15.0, enabling encrypted persistent storage required a different set of properties for
each type of repository. Earlier versions supported multiple methods of configuring key material, including key
specification in nifi.properties, externalized key material in a custom file, and the introduction of
PKCS #12 and Bouncy Castle
FIPS keystore support in NiFi 1.14.0. This
variety of options supported flexibility during the initial development process, but created complexity for standard
deployments.
Changes in NiFi 1.15.0 standardized key material configuration using the keystore provider and introduced a single version property to enable encryption for all repositories. These improvements reduced configuration challenges while maintaining full compatibility with earlier configuration and implementation approaches.
Cipher Algorithms
Version 1 of NiFi repository encryption uses the Advanced Encryption Standard cipher for all repository implementations. The associated key size depends on the parameters used to generate secret key material. AES supports a variety of operating modes with different performance and security characters.
In order to support streaming file contents that can range from several bytes to multiple gigabytes, the encrypted content repository implementation uses AES in counter mode. AES-CTR does not support authenticated encryption, but it provides a straightforward approach to streaming reads and writes.
For the FlowFile and provenance repositories, which handle records of similar small size, NiFi uses AES in Galois/Counter Mode. AES-GCM provides data integrity checking in the form of message authentication code tags appended to the encrypted payload. The NiFi User Guide provides a listing of cipher algorithms and implementation classes for each repository. Although each repository implementation is slightly different, all implementations leverage Java object serialization to store and retrieve encryption metadata for each record.
Keystore Formats
The KeyStore class provides an abstraction around access to certificate and key information. Java applications such as NiFi often use a KeyStore to access certificate and private key information. For the purpose of repository encryption, however, a KeyStore provides access to secret key entries, also known as symmetric keys. In contrast to certificate and private key entries where an application uses the public key for encryption and the private key for decryption, secret key entries support both encryption and decryption.
PKCS12 Supported Algorithms
The historical JKS format does not support secret key entries, so PKCS12 or BCFKS must be used when configuring the keystore provider for repository encryption. The standard implementation of the PKCS12 KeyStore in Java 8 defaults to using password-based encryption with 50,000 SHA-1 iterations for hashing and Triple DES for encryption.
Although SHA-1 and Triple DES are listed as approved algorithms according to FIPS 140-2 for legacy compatibility, SHA-1 has known security issues and modern protocols use AES in place of Triple DES. Java 16 changed the default algorithms to HMAC SHA-256 for hashing and AES-256 for encryption, providing a better level of protection for PKCS12 keystores. Java 8 Update 301 and Java 11 Update 12 include support for these new algorithms, but retain the historical defaults to support backward compatibility.
BCKFS Supported Algorithms
The Bouncy Castle BCFKS format leverages PBKDF2 with SHA-512 and the AES-CCM cipher, which provides data integrity checking together with AES encryption. The PBKDF2 method uses 51,200 iterations according to the default configuration. The combination of SHA-512 and authenticated encryption using AES-CCM make BCFKS relatively better than the default Java 8 implementation of PKCS12. It should be noted, however, that strong algorithms still require a strong password to provide robust protection. For these reasons, selecting a complex passphrase is the most important part of keystore configuration.
Configuration Settings
To support a simplified configuration process, NiFi 1.15.0 brings together the initial set of encrypted repository implementations under a single protocol version number. This strategy moves the encrypted repository capability out of experimental status and also establishes a foundation of future development efforts.
Enabling repository encryption requires persistent access to key material for encryption and decryption operations. With the goal of protecting sensitive information, securing access to the key material itself is an important part of the configuration process.
The following sections describe the process for enabling repository encryption on a new installation of NiFi 1.15.0.
Protocol Configuration
NiFi 1.15.0 introduced a streamlined approach to repository encryption configuration through the addition of an encryption protocol version property. Setting the version property enables encryption for all persistent storage components and overrides individual repository implementation properties.
The nifi.properties configuration can be updated as follows to enable repository encryption:
nifi.repository.encryption.protocol.version=1
Secret Key Generation
Although the supporting technical details can be complicated, generating and storing a secret key for repository encryption is a straightforward process. The NiFi User Guide provides examples for generating a suitable secret key using either PKCS12 or BCFKS. The Java keytool command requires additional arguments for BCFKS because it is not one of the formats supported in standard Java distributions.
From the NiFi installation directory, the following keytool command generates an AES-256 secret key, with an alias
of key-1, and stores it in a BCFKS file named repository.bcfks in the conf directory:
keytool \
-genseckey \
-alias key-1 \
-keyalg AES \
-keysize 256 \
-keystore conf/repository.bcfks \
-storetype BCFKS \
-providerpath lib/bootstrap/bcprov-jdk15on-*.jar \
-providerclass org.bouncycastle.jce.provider.BouncyCastleProvider
The keytool command prompts to enter and confirm a password as follows:
Enter keystore password:
Re-enter new password:
The command provides the option to use a different key password, but NiFi requires using the same key password and keystore password. Press enter when prompted for the key password as follows:
Enter key password for <key-1>
(RETURN if same as keystore password):
Keystore Verification
The following keytool command can be used to list the contents of the new keystore:
keytool \
-list \
-keystore conf/repository.bcfks \
-storetype BCFKS \
-providerpath lib/bootstrap/bcprov-jdk15on-*.jar \
-providerclass org.bouncycastle.jce.provider.BouncyCastleProvider \
-storepass PLACEHOLDER
The store password argument value PLACEHOLDER must be replaced with the keystore password.
NiFi Properties Configuration
The keystore password must be provided in nifi.properties along with the alias of the generated secret key and the
location of the keystore file. The nifi.properties configuration should be updated as follows:
nifi.repository.encryption.key.id=key-1
nifi.repository.encryption.key.provider=KEYSTORE
nifi.repository.encryption.key.provider.keystore.location=./conf/repository.bcfks
nifi.repository.encrytpion.key.provider.keystore.password=PLACEHOLDER
The PLACEHOLDER property value must be replaced with the keystore password entered using the keytool command.
Key Rotation
NiFi is capable of changing the secret key used for new encryption operations while continuing to support access to
information encrypted with another secret key. The key rotation process involves appending a new secret key entry to an
existing keystore, and updating nifi.properties to reference the new key alias.
The following keytool command generates a new secret key with an alias of key-2 and stores it in an existing BCFKS
file:
keytool \
-genseckey \
-alias key-2 \
-keyalg AES \
-keysize 256 \
-keystore conf/repository.bcfks \
-storetype BCFKS \
-providerpath lib/bootstrap/bcprov-jdk15on-*.jar \
-providerclass org.bouncycastle.jce.provider.BouncyCastleProvider \
-storepass PLACEHOLDER
The store password argument value PLACEHOLDER must be replaced with the keystore password.
Press enter when prompted for the key password to use the same password as the keystore:
Enter key password for <key-2>
(RETURN if same as keystore password):
The nifi.properties configuration should be updated with the new key alias as follows:
nifi.repository.encryption.key.id=key-2
Restart NiFi to activate the configuration changes.
Upgrade Considerations
When upgrading from a version of NiFi prior to 1.15.0, existing settings in nifi.properties can be used without
modification. Leveraging the encryption protocol version property and shared keystore provider properties requires
changing repository configuration settings, and may also involve removing the contents of existing repositories.
Configuring a new source of key material effectively prevents access to information encrypted with a different key, so
all processing queues must be clear prior to making changes to avoid data loss.
Backward Compatibility
In versions 1.13.0 and earlier, NiFi supported configuring secret key material using values in nifi.properties or
values stored in external files. Version 1.15.0 maintains support for those historical configuration approaches, but the
external keystore provider is the preferred approach starting with version 1.14.0. Deployments running 1.14.0 and
leveraging the same keystore provider for all repositories can be migrated to use the new configuration properties
without impacting access to existing repository records.
NiFi 1.15.0 maintains compatibility with earlier configuration versions, but the NiFi System Administrator’s Guide indicates that the shared repository encryption properties supersede the deprecated key provider properties for each repository.
Conclusion
The configuration challenges and experimental status associated with repository encryption have presented obstacles to adoption, but changes in NiFi 1.15.0 have made it easier than ever to enable this capability. With internal improvements, the current implementation offers better performance in most cases. Streamlined implementation classes and simplified configuration properties also provide a foundation for future development efforts. As with other types of security measures, repository encryption is not the most visible feature or the most demonstrable capability, but it provides a useful option for deployments that require additional security measures.