Relaying Syslog UDP Events with Apache NiFi
Despite a number of modernized alternatives, sending and receiving logging events using the syslog protocol remains a common strategy. After several decades of differing vendor implementations, RFC 3164 described the conventional approach as the BSD syslog protocol in 2001. RFC 5424 standardized structural improvements to timestamp and message field formatting in 2009. Although RFC 5424 mandated support for Transport Layer Security with TCP, with specific details outlined in RFC 5425, transmission over User Datagram Protocol continued to be supported as described in RFC 5426. Most current syslog servers support the newer standard along with mutual TLS for authentication and encryption, but UDP transmission using RFC 3164 formatting continues to serve as a baseline option.
Apache NiFi components cover a broad range of use cases and formats, from binary records with embedded schemas to
unstructured lines of ASCII characters. To enable flexible integration patterns, Apache NiFi has supported sending and
receiving syslog messages over both TCP and UDP since version 0.4.0. The
Processor provides configurable properties for both transmission protocol and message formatting. The
Processor supports generalized UDP packet transmission, depending on other Processors to format messages in FlowFile
contents. Both Processors enable functional and flexible sending of syslog UDP events, but neither are capable of
handling high volumes of messages. Neither
record-oriented FlowFiles, limiting transmission
to one event per FlowFile. This approach burdens Apache NiFi framework resources, requiring transaction handling and
repository updates for each event processed.
Apache NiFi 1.17.0 introduced the
Controller Service to support record-oriented processing and transmission over UDP with the
UDPEventRecordSink supports sending one UDP packet per record, enabling batched processing and higher
rates of transmission for aggregated events. The syslog protocol provides one example use case for the new Record Sink,
but it is capable of supporting any type of record-oriented data with a configurable Record Writer.
Security and Reliability Considerations
As described in RFC 5425 Section 2, sending unencrypted syslog messages over an untrusted network raises a number of security concerns. Logs often contain sensitive information about system operation and user behavior, requiring protection against unauthorized disclosure and modification. For these reasons, encrypted communication is essential when transmitting logs over an untrusted network. Transporting syslog events with TLS provides communication encryption, and mutual TLS enables strong authentication for sending systems. Virtual Private Networking protocols delegate communication encryption without requiring configuration of syslog servers and clients. Either VPN or TLS should be part of a general network logging strategy.
In addition to communication security, communication reliability is another important factor to consider when deploying syslog clients and servers. UDP is not a connection-oriented protocol and does not provide any delivery guarantees. TCP provides a measure of reliability with socket connection handling and delivery acknowledgement at the protocol layer. The syslog protocol itself does not include a delivery acknowledgement strategy, which means that data loss is possible in the event of unexpected host or network outages. Although acknowledgement at the TCP layer provides a stronger confirmation of delivery, other solutions such as the Reliable Event Logging Protocol are necessary for environments where a high degree of reliability is required. On the other hand, for systems with repetitive logging or less critical information, syslog over UDP can provide a high level of throughput without the overhead of TCP socket connection tracking.
Relaying Syslog Messages
RFC 3164 Section 3 defines a machine receiving and forwarding syslog messages as a relay. NiFi can function as a relay, performing additional filtering and routing to support a number of alerting and archiving use cases. For example, in an environment with current syslog processing, NiFi can be used to support sending to an existing syslog collector while enabling alternative analysis strategies.
Receiving Syslog Records
NiFi provides several options for receiving syslog UDP records. The best solution depends on overall flow design requirements, such as subsequent processing, filtering, or intended destinations. The following Processors can be configured to receive syslog messages over UDP:
Port property is required for all Processors, indicating the UDP port number on which to receive network packets.
Although port 514 is the traditional default for syslog UDP, most operating systems restrict listening on ports below
1024 for users other than root. NiFi should never run as the root user, so a high numbered port such as 5514 should be
For optimal processing throughput, it is essential to batch multiple log records into a single FlowFile. All of these
Processors support batching, but each one uses a different processing strategy with varying levels of validation. All
three Processors include a
Max Batch Size property, with different default values.
All three Processors also have a
Max Size of Message Queue property, which defaults to
10000 for each Processor.
This property controls the maximum number of events that will be placed on a queue within the Processor for eventual
retrieval when the NiFi framework triggers the Processor. Setting this property too low can result in data loss when the
queue is full and Processor is not able to keep up with the rate of incoming messages. Setting this property too high
has a negative impact on memory consumption, leading to potential errors when the framework is not able to read message
fast enough to avoid queue saturation. Monitoring system memory usage is an important part of selecting an optimal queue
size and avoiding unexpected heap memory issues.
ListenSyslog provides a standard approach for handling one or more syslog messages. The
Protocol property defaults
UDP, but also supports
TCP, which can be configured together with an
SSL Context Service to enable TLS.
Parse Messages property enables or disables syslog message validation, setting FlowFile attributes on success and
routing messages to the
invalid relationship on parsing failures. The
Parse Messages property can be set to
when sending clients are expected to produce valid messages, which reduces processing overhead for high volume flows.
Max Batch Size property defaults to
1, which is useful for initial testing, but not optimal for production
operation. The property should be set to at least
100 for record-oriented processing. Setting a value of
higher is necessary for larger deployments. With the default
Message Delimiter setting, the
will combine multiple syslog messages together with a newline character.
ListenUDP presents a generic solution, capable of receiving arbitrary UDP packets. This Processor does not perform
any input validation, which can beneficial for deployments with high throughput requirements. For environments where
sending syslog clients provide consistent message formatting,
ListenUDP supports receiving syslog messages with
Max Batch Size property in
ListenUDP defaults to
1, which will result in creating a large number of small
Max Batch Size to at least
100 enables subsequent Processors to perform record-oriented
processing. The number of sending clients and the volume of messages may require a batch size of
1000 or higher to
provide optimal performance. Similar to
ListenUDP Processor has a
Batching Message Delimiter
property that controls how the Processor combines multiple messages, defaulting to a newline character.
ListenUDPRecord introduces record-oriented processing at the beginning of a syslog flow. This Processor uses a
Record Reader to parse received packets, providing basic record validation before further processing. The
Record Writer property controls the format of records written to output FlowFiles.
Max Batch Size property in
ListenUDPRecord defaults to
1000, providing a reasonable starting point new flow
Poll Timeout property defaults to
50 ms, after which the Processor will create a FlowFile
containing the number of records available from the internal queue. The
Poll Timeout value should be low to avoid
consuming shared resources, but could be increased to produce larger FlowFiles with more records in environments with
intermittent transmission of syslog events.
Record Reader property must be configured with the
service to read incoming syslog messages. The
SyslogReader parses messages using the conventional RFC 3164 structure,
and also supports RFC 5424 timestamps for unambiguous dates based on the
ISO 8601 standard. For relaying syslog messages, setting the
Raw message property to
true in the
SyslogReader provides a simple strategy for maintaining the original message
structure in a
_raw field when passing records to a configured
provides an alternative
Record Reader implementation capable of reading syslog events that contain structured data
conforming to the RFC 5424 standard. The regular
SyslogReader is also capable for reading RFC 5424 messages, but it
will not perform further parsing of the message section of a record. For these reasons, the
SyslogReader presents the
best option for environments with a mixture of formats, or when parsing structured elements is not necessary. The
Syslog5424Reader is a suitable solution for deployments where all sending clients use the RFC 5424 standard.
NiFi does not include a standard
Record Writer implementation for syslog messages. With the variation in syslog
message structure and the nature of syslog protocol itself, creating a reusable syslog Record Writer is not necessarily
straightforward. For the purpose of relaying syslog UDP messages, however, the
Service supports formatting messages using record fields and FlowFile attributes.
Text property of the
FreeFormTextRecordSetWriter must be configured according to the structure of incoming
records. For syslog records processed through the
SyslogReader, the writer should be configured with the following
- Character Set
Text property setting uses NiFi
to reference the raw message field, which is included in each syslog record processed through the
Raw message property enabled. The
FreeFormTextRecordSetWriter appends a newline character at the end of each
rendered message, which integrates well with subsequent syslog processing.
Queue Capacity Logging
ListenUDPRecord Processors support extended debug logging to track the capacity of the internal
event queue. Debug log messages include current capacity, space remaining, and the largest size of the queue since the
Processor was started. Capacity logging messages will be written at the debug level every 60 seconds.
Add the following lines to the NiFi
logback.xml configuration to enable capacity logging for either
<logger name="org.apache.nifi.processors.standard.ListenUDP" level="DEBUG" /> <logger name="org.apache.nifi.processors.standard.ListenUDPRecord" level="DEBUG" />
Debug messages will be logged as follows:
Event Queue Capacity  Remaining  Size  Largest Size 
Largest Size is a key indicator of adequate capacity. If the
Largest Size equals the
Event Queue Capacity, the
Max Size of Message Queue property should be increased to accommodate incoming events. Having the
Event Queue Capacity indicates that the NiFi framework is not processing events fast enough to retrieve them
from the internal queue. Larger message queues place a greater burden on Java Virtual Machine heap memory. General
system resource utilization should be evaluated when considering queue size increases.
The logger command
The logger command on Linux and Unix systems enables simple testing of listening syslog
Processors. The command is capable of sending one message, specified through a command argument, or multiple messages
provided using a file argument. The
logger command supports both the conventional RFC 3164 BSD syslog protocol and the
RFC 5424 standard.
The following command can be used to send a message with the word
running to a Processor listening on UDP port 5514:
logger -n 127.0.0.1 -P 5514 --rfc3164 running
The command will send a UDP syslog event containing the timestamp, local hostname, and local username along with the message specified, using the RFC 3164 format. The syslog event reads as follows:
<13>Sep 26 12:30:45 hostname username: running
The bracketed number
13 indicates the event Priority as defined in
RFC 3164 Section 4.1.1. Following the formula described,
8 leaves a remainder of
5. The number
5 indicates the
Notice level for event Severity, and
user-level messages as the event Facility.
Sending Syslog Records
Regardless of the Processor selected for receiving syslog messages, the optimal approach for sending records over UDP in
NiFi 1.17.0 is the
Processor configured with the
SyslogReader for the
Record Reader property, and the
Record Destination Service property.
Flows designed using the
ListenUDPRecord Processor can use the same
SyslogReader Service for the
Processor. Both the
ListenUDP Processors combine multiple syslog messages using a newline
character, which the
SyslogReader is also capable of handling.
Include Zero Record Results property in
PutRecord defaults to
false, which should be retained to avoid
UDPEventRecordSink has standard
Port properties to configure the destination for UDP records.
Both of these properties are required as part of the standard component configuration.
Record Writer property supports standard Controller Services that implement the
interface. As described for
FreeFormTextRecordSetWriter provides a straightforward option for
formatting syslog records parsed using the
SyslogReader. For flows using
ListenUDPRecord, the same instance of
FreeFormTextRecordSetWriter can be configured as the
Record Writer in
UDPEventRecordSink. For flows using other
Processors, the same property values can be applied to a new instance of
UDPEventRecordSink also includes a
Sender Threads property, defaulting to
2, which controls the maximum size of
the thread pool for handling UDP packet processing.
UDPEventRecordSink leverages the Netty
framework for network protocol handling, and passes the
Sender Threads property value to the Netty
NioEventLoopGroup class, which is responsible
for managing socket operations. More is not necessarily better when it comes to thread configuration, but in general,
the number of
Sender Threads should match the number of
Concurrent Tasks configured for the
The number of
Sender Threads should not exceed the number of CPU cores.
It is important to note that the
UDPEventRecordSink will not necessarily report communication failures, and some
network errors may be logged without causing
PutRecord to route related FlowFiles to the
failure relationship. As
mentioned earlier, the nature of UDP means that receiving systems do not provide delivery acknowledgment. Based on host
configuration or network firewall settings, the Netty framework may receive an
ICMP packet indicating a
UDPEventRecordSink will log as an error. This error serves as a courtesy notification. The NiFi Bulletin
Board should be monitored for this type of communication problem, but it is best to verify transmission on the receiving
system to confirm delivery.
Record-oriented processing offers high throughput for content that follows a standard structure, including the
conventional BSD syslog protocol. Building on existing capabilities, the
UDPEventRecordSink extends record-oriented
transmission to UDP packets. Relaying syslog UDP messages provides a simple use case, but more advanced flows can be
designed using other record-oriented Processors and Controller Services. Leveraging record batching for initial
processing is essential for optimal throughput, and selecting the appropriate amount of input validation is important
design decision. Taking these concepts together, it is possible to build advanced flows that support a number of system
logging use cases.