Enabling Apache NiFi Support for OpenPGP Signatures
Standard OpenPGP encryption algorithms provide confidentiality, protecting information from unauthorized access, but ciphers supported in RFC 4880 do not incorporate other important cryptographic characteristics such as authentication and integrity. Although modern ciphers such as AES-GCM and ChaCha20-Poly1305 provide integrated integrity checking, official support for authenticated encryption in OpenPGP is still being drafted as of this writing. OpenPGP has supported digital signatures since the initial version of the protocol, enabling message signing as either an independent operation, or in conjunction with encryption. Message signing and signature verification are vital aspects of any cryptographic protocol, and digital signatures provide important features even when message confidentiality is not required.
Apache NiFi 1.14.0 included restructured Processors and Controller Services for encrypting and decrypting OpenPGP messages, providing performance improvements and new configuration options. The introduction of Controller Services for accessing public and private key material established a reusable approach to public-key cryptography for OpenPGP in NiFi, enabling additional development and integration options. These new components evolved OpenPGP support in NiFi, but did not provide a complete set of OpenPGP features.
Introducing Signature Processing
Apache NiFi 1.15.0 brings digital signature processing to the set of available OpenPGP capabilities. The
Processors provide enhanced security for OpenPGP handling, and also enable new processing strategies. Both components
depend on the OpenPGP Controller Services that also support the
Processors. Leveraging the same Controller Services simplifies component implementation as well as flow configuration.
Similar to the encryption and decryption processors,
EncryptContentPGP support interoperation
with other applications and services that follow the OpenPGP standard. Building on the robust OpenPGP support in the
Bouncy Castle library provided a reliable protocol implementation for creating and
testing NiFi components.
For NiFi flows integrating with external services that support OpenPGP, the ability to sign outgoing files or verify incoming files provides better security than encryption on its own. When receiving encrypted files, the ability to verify a digital signature using a trusted public key provides a means of checking content enciphered using an otherwise valid encryption method. OpenPGP digital signatures support several SHA-2 hash algorithms, providing a greater degree of integrity protection than the Modification Detection Code Packet embedded with OpenPGP encrypted messages. The MDC Packet provides the equivalent of a checksum for encrypted messages, but some have criticized its security in light of its reliance on the broken SHA-1 algorithm.
The option to generate OpenPGP digital signatures also enables use cases where content does not require strict
confidentiality. OpenPGP digital signatures not only enable content integrity verification, but also provide attribution
to the message sender. Version control systems such as Git
support signing commits, and artifact repositories such
leverage OpenPGP signing to enable information integrity and
VerifyContentPGP Processors can support similar strategies for files produced or
consumed through NiFi flows.
Signing and Encryption Considerations
Although digital signatures provide a robust strategy for verifying message integrity, the combination of encryption and signing operations raises potential concerns. Over twenty years ago, Don Davis described the problem with simple signing and encryption in multiple protocols, including OpenPGP. The fundamental problem relates to the disjointed nature of signing and encryption in OpenPGP and similar standards. Treating signing and encryption as independent operations provides flexibility for developers, but raises questions about whether the message sender performed both the signing and encryption operations.
This problem can be illustrated through the relationship between the
SignContentPGP and the
Processors. When Alice wants to send a file to Bob, she can configure
SignContentPGP to sign the file using her
private key, and configure
EncryptContentPGP to encrypt the signed file using Bob’s public key. When Bob receives the
file, he decrypts the file with
DecryptContentPGP using his private key, and then verifies the file with
VerifyContentPGP using Alice’s public key. With this flow design, Alice knows the content of the original file as
indicated through the digital signature. This flow also ensures the Bob was the intended recipient, because the file
requires access to Bob’s private key for successful decryption. What if Bob decides to send Alice’s signed file to
After Bob decrypts the file, the signed file may have no indication that Bob was the intended recipient. Bob could then
the take the file that Alice signed and encrypt it using
EncryptContentPGP with Charlie’s public key. Bob then sends
the encrypted and signed file to Charlie. When Charlie receives the file and decrypts it with
his private key, he has the same file that Alice signed. Charlie verifies the file with
VerifyContentPGP using Alice’s
public key. How can Charlie determine whether he was the intended recipient of the file?
In his paper, Don Davis outlines several strategies to solve the problem. This simplest solution is to indicate the intended recipient in the original file prior to signing. For a file formatted using a standard structure such as XML or JSON, this might be as simple as including a field indicating the intended recipient. This field can be checked following signature verification to reject files that could have been subjected to malicious routing. Other solutions involve signing the encrypted file as well as the original file, which involves additional processing. The ideal technical solution involves interrelationship between signing and encryption operations, but the current OpenPGP standard does not support such as approach. These issues highlight the challenges inherent in configuring a flow that addresses applicable security concerns. Understanding the original content, as well as the signing, encryption, and verification approach on both sides of the communication is essential to a secure configuration.
OpenPGP Processor Configuration
VerifyContentPGP Processors can be configured in various ways based on the intended use case.
DecryptContentPGP Processors also incorporate enhancements to support configuration in
conjunction with signing and verification.
Digital Signature Processing
Configuring digital signature generation and verification provides baseline message integrity without confidentiality. In other words, creating and incorporating a digital signature allows a recipient to verify the message content and the message sender, but it does not conceal the content from an intermediate third party. Using digital signatures without encryption may be appropriate for some communication patterns. Configuring signing and verification also provides a foundation for adding encryption when needed.
SignContentPGP Processor supports several configuration properties for constructing signed files. The processor
accepts any type of input FlowFile content and produces a signed OpenPGP message or detached signature depending on the
Message Compression Algorithm
Compression Algorithm property defaults to
ZIP and supports the following values:
ZIP algorithm offers the greatest level of compatibility with other OpenPGP implementations, but disabling
compression or selecting a different algorithm provide better performance or sizing characteristics.
Message File Encoding
File Enconding property defaults to
BINARY and supports the following values:
BINARY encoding is much more efficient in terms of file size, but the
ASCII encoding produces a Base64 string
suitable for transmission through protocols that may not support binary processing.
Signature Hash Algorithm
Hash Algorithm is essential to signature security. The OpenPGP standard allows a variety of signature algorithms,
Hash Algorithm property in
SignContentPGP limits the configuration to the following SHA-2 algorithms:
Hash Algorithm property defaults to
SHA512, which requires the greatest amount of computation, but also provides
a degree of additional security. The other algorithms should provide sufficient security for common use cases.
Signing Strategy property controls the content of files transferred to the
success relationship. The default
SIGNED strategy supports standard use cases where output files consist of an OpenPGP message containing the original
file and the associated signature information.
DETACHED strategy configures the processor to produce a standalone signature, dropping the original file after
successful signing. Using the
DETACHED strategy allows NiFi to send the standalone digital signature to a separate
location for external processing. In most cases, using the
DETACHED strategy requires configuring the NiFi flow to
duplicate or fork the original file for separate processing.
Private Key Service
Private Key Service property requires configuring a
with one or more private keys that
SignContentPGP can use for signature generation. The standard implementation of the private key service supports
configuring private key material using a file location or as a direct property value of the service.
Private Key ID
Private Key ID property requires specifying the hexadecimal identifier of the private key that
will use for signing. The key identifier must consist of 16 uppercase hexadecimal characters. The identifier is based on
the last 16 characters of the key fingerprint formatted as a hexadecimal string.
VerifyContentPGP Processor supports reading OpenPGP messages and verifying the attached signature against the
computed hash using trusted public keys. The processor requires the configuration of a
with one or more public keys that
VerifyContentPGP can use for signature verification.
Signed OpenPGP messages contain multiple packets describing the signature details. One-Pass Signature Packets include the original hash algorithm as well as the algorithm and identifier of the signing key. This packet precedes the literal data and allows implementations to search for the associated public key prior to reading the remaining contents. The Signature Packet follows the literal data and contains the message digest as well as additional signature details.
The OpenPGP signing format allows
VerifyContentPGP to read files without significant memory buffering. The placement
of the one-pass signature also allows the processor to indicate verification failures due to missing public keys before
reading the entire message. Verification failure logs include the associated public key identifier for troubleshooting.
Following successful signature verification,
VerifyContentPGP writes the literal data to a FlowFile routed to
success relationship. The processor also writes a number of FlowFile attributes containing signature algorithm
information, such as
pgp.signature.created indicating the time of signature creation in
Unix epoch milliseconds.
Signing then Encrypting
In addition to signing and verifying content with OpenPGP, NiFi 1.15.0 also supports combining signing and encryption.
This configuration requires a holistic approach to flow design in order to avoid the security pitfalls described
earlier. Standardized recipient indication in the original file, prior to processing through
SignContentPGP, is a
minimum security requirement for signed and encrypted data flows.
Conceptual Flow Design
The following order of operations provides a basic outline for configuring the sending side of a flow with signing and encryption:
- Prepare file with recipient information
- Sign file with sender private key using
- Encrypt file with recipient public key using
The receiving side of the flow should use the following pattern:
- Decrypt file with recipient private key using
- Verify file with sender public key using
- Validate file recipient information
Although this approach provides a general framework, there are many additional details to be considered for a particular implementation. When improperly configured, signing and encryption can provide a false sense of security. Reviewing signing and encryption in isolation can lead to ignoring other security issues. For these reasons, a flow design should be evaluated from a variety of perspectives before enabling transmission of sensitive information.
EncryptContentPGP Processor does not require custom property configuration to be used in conjunction with
SignContentPGP. NiFi 1.15.0 updates
EncryptContentPGP to detect incoming OpenPGP messages, avoiding unnecessary
wrapping of OpenPGP signature packets. This improvement allows
EncryptContentPGP to produce OpenPGP messages that
retain digital signatures, supporting interoperation with applications such as GnuPG.
NiFi 1.15.0 adds a new property to the
DecryptContentPGP Processor named
Decryption Strategy. This property value
DECRYPTED, maintaining compatibility with earlier versions. FlowFiles processed using the
strategy do not retain any OpenPGP signature information. Using the default strategy writes a new FlowFile containing
the decrypted literal data and routes the FlowFile to the
success relationship. The
DECRYPTED strategy may be
acceptable as an interim configuration for flows in the process of migrating to signing and encryption.
When configuring a flow to require decryption and verification, the
Decryption Strategy property must be set to
PACKAGED in order for
DecryptContentPGP to write a FlowFile suitable for signature verification. The
strategy configures the processor to write a FlowFile containing a signed OpenPGP message, which can be verified
Support for OpenPGP digital signatures in NiFi 1.15.0 fulfills several long-standard feature requests and creates the opportunity for new flow design patterns. Message signing and verification can improve the security posture for various communication operations, with or without confidentiality requirements.
Although the nature of the OpenPGP standard presents concerns when attempting to combine signing and encryption, a
careful flow design that considers the complete content lifecycle can mitigate potential issues. Building on the
Controller Services released in NiFi 1.14.0, the
VerifyContentPGP Processors round out support
for common OpenPGP features.