Enabling Apache NiFi Support for OpenPGP Signatures

2022-02-19 • 10 minute read • David Handermann

Background

Standard OpenPGP encryption algorithms provide confidentiality, protecting information from unauthorized access, but ciphers supported in RFC 4880 do not incorporate other important cryptographic characteristics such as authentication and integrity. Although modern ciphers such as AES-GCM and ChaCha20-Poly1305 provide integrated integrity checking, official support for authenticated encryption in OpenPGP is still being drafted as of this writing. OpenPGP has supported digital signatures since the initial version of the protocol, enabling message signing as either an independent operation, or in conjunction with encryption. Message signing and signature verification are vital aspects of any cryptographic protocol, and digital signatures provide important features even when message confidentiality is not required.

Apache NiFi 1.14.0 included restructured Processors and Controller Services for encrypting and decrypting OpenPGP messages, providing performance improvements and new configuration options. The introduction of Controller Services for accessing public and private key material established a reusable approach to public-key cryptography for OpenPGP in NiFi, enabling additional development and integration options. These new components evolved OpenPGP support in NiFi, but did not provide a complete set of OpenPGP features.

Introducing Signature Processing

Apache NiFi 1.15.0 brings digital signature processing to the set of available OpenPGP capabilities. The SignContentPGP and VerifyContentPGP Processors provide enhanced security for OpenPGP handling, and also enable new processing strategies. Both components depend on the OpenPGP Controller Services that also support the EncryptContentPGP and DecryptContentPGP Processors. Leveraging the same Controller Services simplifies component implementation as well as flow configuration. Similar to the encryption and decryption processors, SignContentPGP and EncryptContentPGP support interoperation with other applications and services that follow the OpenPGP standard. Building on the robust OpenPGP support in the Bouncy Castle library provided a reliable protocol implementation for creating and testing NiFi components.

For NiFi flows integrating with external services that support OpenPGP, the ability to sign outgoing files or verify incoming files provides better security than encryption on its own. When receiving encrypted files, the ability to verify a digital signature using a trusted public key provides a means of checking content enciphered using an otherwise valid encryption method. OpenPGP digital signatures support several SHA-2 hash algorithms, providing a greater degree of integrity protection than the Modification Detection Code Packet embedded with OpenPGP encrypted messages. The MDC Packet provides the equivalent of a checksum for encrypted messages, but some have criticized its security in light of its reliance on the broken SHA-1 algorithm.

The option to generate OpenPGP digital signatures also enables use cases where content does not require strict confidentiality. OpenPGP digital signatures not only enable content integrity verification, but also provide attribution to the message sender. Version control systems such as Git support signing commits, and artifact repositories such as Sonatype leverage OpenPGP signing to enable information integrity and attribution. The SignContentPGP and VerifyContentPGP Processors can support similar strategies for files produced or consumed through NiFi flows.

Signing and Encryption Considerations

Although digital signatures provide a robust strategy for verifying message integrity, the combination of encryption and signing operations raises potential concerns. Over twenty years ago, Don Davis described the problem with simple signing and encryption in multiple protocols, including OpenPGP. The fundamental problem relates to the disjointed nature of signing and encryption in OpenPGP and similar standards. Treating signing and encryption as independent operations provides flexibility for developers, but raises questions about whether the message sender performed both the signing and encryption operations.

This problem can be illustrated through the relationship between the SignContentPGP and the EncryptContentPGP Processors. When Alice wants to send a file to Bob, she can configure SignContentPGP to sign the file using her private key, and configure EncryptContentPGP to encrypt the signed file using Bob’s public key. When Bob receives the file, he decrypts the file with DecryptContentPGP using his private key, and then verifies the file with VerifyContentPGP using Alice’s public key. With this flow design, Alice knows the content of the original file as indicated through the digital signature. This flow also ensures the Bob was the intended recipient, because the file requires access to Bob’s private key for successful decryption. What if Bob decides to send Alice’s signed file to someone else?

After Bob decrypts the file, the signed file may have no indication that Bob was the intended recipient. Bob could then the take the file that Alice signed and encrypt it using EncryptContentPGP with Charlie’s public key. Bob then sends the encrypted and signed file to Charlie. When Charlie receives the file and decrypts it with DecryptContentPGP using his private key, he has the same file that Alice signed. Charlie verifies the file with VerifyContentPGP using Alice’s public key. How can Charlie determine whether he was the intended recipient of the file?

In his paper, Don Davis outlines several strategies to solve the problem. This simplest solution is to indicate the intended recipient in the original file prior to signing. For a file formatted using a standard structure such as XML or JSON, this might be as simple as including a field indicating the intended recipient. This field can be checked following signature verification to reject files that could have been subjected to malicious routing. Other solutions involve signing the encrypted file as well as the original file, which involves additional processing. The ideal technical solution involves interrelationship between signing and encryption operations, but the current OpenPGP standard does not support such as approach. These issues highlight the challenges inherent in configuring a flow that addresses applicable security concerns. Understanding the original content, as well as the signing, encryption, and verification approach on both sides of the communication is essential to a secure configuration.

OpenPGP Processor Configuration

The SignContentPGP and VerifyContentPGP Processors can be configured in various ways based on the intended use case. The EncryptContentPGP and DecryptContentPGP Processors also incorporate enhancements to support configuration in conjunction with signing and verification.

Digital Signature Processing

Configuring digital signature generation and verification provides baseline message integrity without confidentiality. In other words, creating and incorporating a digital signature allows a recipient to verify the message content and the message sender, but it does not conceal the content from an intermediate third party. Using digital signatures without encryption may be appropriate for some communication patterns. Configuring signing and verification also provides a foundation for adding encryption when needed.

SignContentPGP Configuration

The SignContentPGP Processor supports several configuration properties for constructing signed files. The processor accepts any type of input FlowFile content and produces a signed OpenPGP message or detached signature depending on the property configuration.

Message Compression Algorithm

The Compression Algorithm property defaults to ZIP and supports the following values:

• UNCOMPRESSED
• ZIP
• ZLIB
• BZIP2

The ZIP algorithm offers the greatest level of compatibility with other OpenPGP implementations, but disabling compression or selecting a different algorithm provide better performance or sizing characteristics.

Message File Encoding

The File Enconding property defaults to BINARY and supports the following values:

• ASCII
• BINARY

The BINARY encoding is much more efficient in terms of file size, but the ASCII encoding produces a Base64 string suitable for transmission through protocols that may not support binary processing.

Signature Hash Algorithm

The Hash Algorithm is essential to signature security. The OpenPGP standard allows a variety of signature algorithms, but the Hash Algorithm property in SignContentPGP limits the configuration to the following SHA-2 algorithms:

• SHA256
• SHA384
• SHA512

The Hash Algorithm property defaults to SHA512, which requires the greatest amount of computation, but also provides a degree of additional security. The other algorithms should provide sufficient security for common use cases.

Signing Strategy

The Signing Strategy property controls the content of files transferred to the success relationship. The default SIGNED strategy supports standard use cases where output files consist of an OpenPGP message containing the original file and the associated signature information.

The DETACHED strategy configures the processor to produce a standalone signature, dropping the original file after successful signing. Using the DETACHED strategy allows NiFi to send the standalone digital signature to a separate location for external processing. In most cases, using the DETACHED strategy requires configuring the NiFi flow to duplicate or fork the original file for separate processing.

Private Key Service

The Private Key Service property requires configuring a PGPPrivateKeyService with one or more private keys that SignContentPGP can use for signature generation. The standard implementation of the private key service supports configuring private key material using a file location or as a direct property value of the service.

Private Key ID

The Private Key ID property requires specifying the hexadecimal identifier of the private key that SignContentPGP will use for signing. The key identifier must consist of 16 uppercase hexadecimal characters. The identifier is based on the last 16 characters of the key fingerprint formatted as a hexadecimal string.

VerifyContentPGP Configuration

The VerifyContentPGP Processor supports reading OpenPGP messages and verifying the attached signature against the computed hash using trusted public keys. The processor requires the configuration of a PGPPublicKeyService with one or more public keys that VerifyContentPGP can use for signature verification.

Signed OpenPGP messages contain multiple packets describing the signature details. One-Pass Signature Packets include the original hash algorithm as well as the algorithm and identifier of the signing key. This packet precedes the literal data and allows implementations to search for the associated public key prior to reading the remaining contents. The Signature Packet follows the literal data and contains the message digest as well as additional signature details.

The OpenPGP signing format allows VerifyContentPGP to read files without significant memory buffering. The placement of the one-pass signature also allows the processor to indicate verification failures due to missing public keys before reading the entire message. Verification failure logs include the associated public key identifier for troubleshooting. Following successful signature verification, VerifyContentPGP writes the literal data to a FlowFile routed to the success relationship. The processor also writes a number of FlowFile attributes containing signature algorithm information, such as pgp.signature.created indicating the time of signature creation in Unix epoch milliseconds.

Signing then Encrypting

In addition to signing and verifying content with OpenPGP, NiFi 1.15.0 also supports combining signing and encryption. This configuration requires a holistic approach to flow design in order to avoid the security pitfalls described earlier. Standardized recipient indication in the original file, prior to processing through SignContentPGP, is a minimum security requirement for signed and encrypted data flows.

Conceptual Flow Design

The following order of operations provides a basic outline for configuring the sending side of a flow with signing and encryption:

1. Prepare file with recipient information
2. Sign file with sender private key using SignContentPGP
3. Encrypt file with recipient public key using EncryptContentPGP

The receiving side of the flow should use the following pattern:

1. Decrypt file with recipient private key using DecryptContentPGP
2. Verify file with sender public key using VerifyContentPGP
3. Validate file recipient information

Although this approach provides a general framework, there are many additional details to be considered for a particular implementation. When improperly configured, signing and encryption can provide a false sense of security. Reviewing signing and encryption in isolation can lead to ignoring other security issues. For these reasons, a flow design should be evaluated from a variety of perspectives before enabling transmission of sensitive information.

EncryptContentPGP Configuration

The EncryptContentPGP Processor does not require custom property configuration to be used in conjunction with SignContentPGP. NiFi 1.15.0 updates EncryptContentPGP to detect incoming OpenPGP messages, avoiding unnecessary wrapping of OpenPGP signature packets. This improvement allows EncryptContentPGP to produce OpenPGP messages that retain digital signatures, supporting interoperation with applications such as GnuPG.

DecryptContentPGP Configuration

NiFi 1.15.0 adds a new property to the DecryptContentPGP Processor named Decryption Strategy. This property value defaults to DECRYPTED, maintaining compatibility with earlier versions. FlowFiles processed using the DECRYPTED strategy do not retain any OpenPGP signature information. Using the default strategy writes a new FlowFile containing the decrypted literal data and routes the FlowFile to the success relationship. The DECRYPTED strategy may be acceptable as an interim configuration for flows in the process of migrating to signing and encryption.

When configuring a flow to require decryption and verification, the Decryption Strategy property must be set to PACKAGED in order for DecryptContentPGP to write a FlowFile suitable for signature verification. The PACKAGED strategy configures the processor to write a FlowFile containing a signed OpenPGP message, which can be verified using VerifyContentPGP.

Conclusion

Support for OpenPGP digital signatures in NiFi 1.15.0 fulfills several long-standard feature requests and creates the opportunity for new flow design patterns. Message signing and verification can improve the security posture for various communication operations, with or without confidentiality requirements.

Although the nature of the OpenPGP standard presents concerns when attempting to combine signing and encryption, a careful flow design that considers the complete content lifecycle can mitigate potential issues. Building on the Controller Services released in NiFi 1.14.0, the SignContentPGP and VerifyContentPGP Processors round out support for common OpenPGP features.