Posts
Elephant Below the Waterline: Hadoop Dependencies in Iceberg
Apache Iceberg is an open table format with widespread adoption, but it carries the weight of historical baggage from Apache Hadoop. With Apache Parquet as a popular storage format for Iceberg, tight coupling to Hadoop classes presents maintenance challenges for integration with the Iceberg ecosystem.
Bringing Kubernetes Clustering to Apache NiFi
Apache NiFi 2.0.0-M1 introduced extensible leader election, adding first-class support for clustering on Kubernetes using native services, without the need for Apache ZooKeeper. Based on the building blocks of Kubernetes ConfigMaps and Leases, new NiFi clustering capabilities reduce both configuration complexity and operational overhead.
Getting Started with Hatch for Python Projects
Hatch enables building, managing, and publishing Python projects using standardized configuration and straightforward commands. As a project of the Python Packaging Authority, Hatch implements modern management conventions and provides a plugin system for extensible integration with project lifecycle actions. With default project templates, Hatch supports static analysis, unit testing, code coverage, binary packaging, module publishing, and more, through an intuitive command-line interface.
Building OpenTelemetry Collection in Apache NiFi with Netty
OpenTelemetry provides an common specification that supports software observability across numerous platforms and services. With the introduction of ListenOTLP in versions 2.0.0-M1 and 1.24.0, Apache NiFi enables a number of filtering, transformation, and routing use cases. Built on the Netty framework, ListenOTLP provides complete support for the telemetry types and encoding strategies defined in OpenTelemetry Protocol 1.0.0.
Modernizing Streaming Encryption with age in Apache NiFi
Apache NiFi 2.0.0-M1 and 1.24.0 introduced new Processors supporting the age-encryption.org/v1 specification. Built on the Jagged framework implementation, the EncryptContentAge and DecryptContentAge Processors perform streaming cipher operations using the ChaCha20-Poly1305 algorithm. The age specification supports key agreement using the elliptic curve X25519 function, providing a modern solution for automated encryption and decryption with standard key pairs.
Firsthand Analysis of Apache NiFi Vulnerability CVE-2023-34468
Recent reporting on Apache NiFi vulnerability CVE-2023-34468 has highlighted significant concerns related to potential remote code execution. Although upgrading to the latest version of Apache NiFi remains the recommended solution, a closer evaluation of the vulnerability shows important details glossed over in published analysis. Exploiting H2 database connection strings requires both authentication and sufficient authorization, the importance of which is missing from recent reporting.
Introducing Jagged for age Encryption in Java
Jagged is a set of Java libraries supporting the age encryption specification. Designed as a simple and modern file encryption standard, age builds on trusted cryptographic algorithms and provides a concise structure for formatting header and payload information. Jagged provides a modular implementation to enable application integration for automated or interactive use cases.
Streamlining Apache NiFi Cluster State Migration
Cluster state tracking in Apache NiFi supports consistent and resilient flow processing across multiple nodes. Apache ZooKeeper provides a common solution for NiFi state tracking, with Redis as an alternative option. NiFi 2.0 adds Kubernetes ConfigMaps as a state tracking provider, and also introduces a simplified approach for migrating from one state provider to another with minimal configuration.
Supporting OIDC Refresh Tokens in Apache NiFi
Apache NiFi 1.21.0 introduced support for OAuth 2 Refresh Tokens as part of redesigned OpenID Connect integration. Refresh Tokens support extended application sessions while maintaining security using Access Tokens with short expirations. Redesigned OIDC integration is compatible with existing deployments and provides additional security with standardized OAuth 2 Token Revocation.
SSHJ Key Authentication Formats
SSHJ is a Java library supporting SSH and SFTP client operations. As the SSH protocol has evolved to support multiple authentication strategies, SSHJ has adapted to support a variety of formats and algorithms for public key authentication. Based on an extensible design, SSHJ is capable of loading and using keys from a number of different source formats.
Backward Compatible Content Decryption in Apache NiFi
Backward compatibility is both an important and challenging part of software engineering. Decrypting information using legacy algorithms requires additional maintenance, but it provides a migration path for better alternatives. Apache NiFi 1.20.0 introduced new content decryption processors to enable migration from weak and proprietary formats to more robust options.
Integrating Apache NiFi with Okta LDAP Groups
Lightweight Directory Access Protocol supports a number of integration strategies in Apache NiFi, including authentication and authorization. LDAP can be used in conjunction with single sign-on solutions to provide user enumeration and group membership for NiFi access policies. In addition to serving as an Identity Provider using OIDC or SAML, Okta provides an LDAP interface for centralized management and retrieval of users and groups.
Integrating Apache NiFi with Okta OIDC Authentication
Apache NiFi has supported single sign-on authentication using OpenID Connect since version 1.4.0. Building on the OAuth 2.0 specification, OIDC supports delegated authentication using standard credential processing flows. The Okta identity platform provides configurable OIDC authentication, enabling centralized identity management and access policy enforcement. Okta delivers a well-documented implementation of OpenID Connect, supporting a robust authentication strategy for NiFi deployments.
Integrating Apache NiFi with Okta SAML Authentication
Apache NiFi 1.13.0 introduced support for single sign-on authentication through a SAML identity provider. NiFi 1.17.0 included a refactored implementation based on Spring Security 5 while maintaining compatible points of integration. The Okta identity platform enables configurable SAML 2.0 authentication, supporting federated access as well as group management and single logout processing. Configuring Apache NiFi with Okta SAML provides a strong access management solution.
Introducing Apache NiFi Deprecation Logging
Apache NiFi 1.18.0 added deprecation logging to warn administrators about components and features targeted for removal in future major releases. Deprecation messages presents important software versioning information in a standard format, improving communication between project developers and community users. Reading and responding to deprecation warnings enables administrators to prepare for upgrades and avoid breaking changes.
Relaying Syslog UDP Events with Apache NiFi
The syslog protocol has provided a conventional approach to networked logging for decades. Apache NiFi has supported sending and receiving syslog messages since version 0.4.0. Apache NiFi 1.17.0 introduced the UDPEventRecordSink service, supporting record-oriented message transmission over User Datagram Protocol for syslog and other use cases.
Implementing Apache NiFi Support for Sensitive Dynamic Properties
Apache NiFi 1.17.0 introduced framework support for sensitive dynamic properties, allowing operators to protect custom properties in selected components. Sensitive dynamic properties enable component developers to support flexible configuration while maintaining system security.
Supporting HTTP/2 in Apache NiFi
The Hypertext Transfer Protocol is one of the foundational Internet protocols. Almost 20 years after the initial version, RFC 7540 codified HTTP/2, maintaining the core concepts of HTTP/1.1 and incorporating several important optimizations. Building on advances in Jetty 9 and JDK 8, recent improvements to Apache NiFi introduced server support for HTTP/2 in both framework and extension components.
Analyzing and Mitigating XML External Entity Vulnerabilities in Apache NiFi
Apache NiFi 1.16.1 resolved XML external entity vulnerabilities in multiple components, described in CVE-2022-29265. Reviewing current and previous XML vulnerabilities enables an accurate characterization of the impact on particular deployments. A summary of the resolution provides useful details for any project that performs XML processing.
Introducing Apache NiFi HTTP Request Logging
Apache NiFi 1.16.0 added configurable logging for HTTP requests, which the framework processes during user interface actions or service operations. HTTP request logging supports a number of use cases, including access auditing, communication troubleshooting, and performance monitoring.
How Not to Write Unit Tests
Writing unit tests is a common part of most software development projects. Following best practices for implementing tests is essential to maintaining robust and reliable applications. Reviewing optimal approaches to writing less than optimal tests can also inform development efforts.
Enabling Apache NiFi Support for OpenPGP Signatures
Apache NiFi 1.15.0 incorporates new processors for signing and verifying OpenPGP messages. SignContentPGP and VerifyContentPGP provide enhanced security for OpenPGP processing, supporting cryptographic signature handling as a standalone operation or in conjunction with encryption.
Introducing Socket Broker
Socket Broker is a Java library supporting SOCKS and HTTP proxy servers with authentication. Socket Broker provides TCP network connections through proxy servers with optional support for username and password credentials, without the need for configuring Java system properties.
Managing Logging Libraries in Apache NiFi
Vulnerabilities in Log4j 2 and other logging libraries have prompted increased scrutiny across many products. Apache NiFi integrates with a wide variety of services that require various dependencies, including multiple types of logging. NiFi 1.15.2 incorporated the removal and exclusion of several unnecessary libraries, highlighting the importance of managing logging dependencies.
Evaluating Log4Shell and Apache NiFi
The Apache Log4j 2 arbitrary code execution vulnerability known as Log4Shell has impacted numerous products and services. Although Apache NiFi does not use Log4j 2 directly, several extension components include library references that should be considered.
Configuring Apache NiFi Repository Encryption
Configurable information storage is a core feature of Apache NiFi. Multiple releases have expanded support for encrypting information in application repositories. Recent updates in NiFi 1.15.0 have streamlined both the implementation and the configuration associated with repository encryption.
Improving JWT Authentication in Apache NiFi
JSON Web Tokens provide authorized access to Apache NiFi for a number of authentication strategies. Recent changes to JWT handling have improved the security posture of several important elements including key generation, secret storage, signature verification, and token revocation.
Restructuring Apache NiFi Support for OpenPGP
Apache NiFi 1.14.0 includes a redesigned approach for encrypting and decrypting OpenPGP messages. The introduction of new Processors and Controller Services for OpenPGP provides additional capabilities and address a number of issues with the original implementation. These new components support a variety of potential use cases and create opportunities for additional development efforts.
Surveying Pretty Good Privacy After Three Decades
Pretty Good Privacy has provided a common standard for message encryption and digital signatures for over 30 years. The OpenPGP specification is not without its detractors, but new development efforts have continued in the face of pointed criticisms. The current landscape includes both historical artifacts and new initiatives to bring current cryptographic standards to the specification.
Deciphering Apache NiFi Component Property Encryption
Encrypting sensitive component properties is one of the foundational features of Apache NiFi. Understanding and configuring the required settings is essential to deploying a secure system.
Single User Access and HTTPS in Apache NiFi
Apache NiFi 1.14.0 builds on a foundation of configurable security and provides a better starting point for simple deployments. Single user authentication and automatic certificate generation for HTTPS access close several gaps in the default configuration.