Meinberg, with the great help of Mr Weber, has implemented “syslog over TLS” in the LTOS version 7.06. The following report describes the general advantages of “syslog over TLS” and the implementation of it in the LTOS.
Why “Syslog over TLS” is a Good Idea
Syslog messages are a very effective means of monitoring the state of an IT system. Events reveal login attempts, error states, configuration changes, and a great deal of other security-critical information. There are also many who seek to have this data stored and analyzed centrally, for example, to provide “System Information and Event Management” systems (SIEM systems) with a common data source. Under ideal conditions, SIEM systems can process, collate, and build upon this information to provide system administrators with very useful reports that allow attacks on systems and networks to be detected early. However, it is important for syslog messages to be secured so that the transmission of syslog messages does not in its own right become a security issue, potentially allowing hackers to exploit the sensitive information in transit. Thankfully, the program “syslog-ng” used by the LANTIME Operating System (LTOS) supports a TLS connection. This TLS connection preserves, in particular, the integrity and confidentiality of the transmitted data. “syslog-ng” has been in use in LTOS for a long time for managing syslog messages, but the LTOS configuration possibilities always lacked the support of “syslog over TLS” in the past.
In theory, only two additional things are required to make use of the TLS functionality of “syslog-ng”. First, you need a valid root CA certificate, and second, the configuration parameter transport (“tls”) must be added to the “syslog-ng” configuration. In practice, as we’ll see below, it turns out to be not quite so simple, given that there are a couple of other things to consider. But let’s start by addressing in general terms how this functionality is implemented in LTOS.
Syslog over TLS in LTOS
Because “syslog-ng” is already installed and the Web Interface already offers the ability to configure external syslog servers, all that’s needed to complete the implementation is to set the new configuration parameters. These are “Transport Protocol = TLS”, the default “syslog over TLS” port 6514, and the switch “peer verify” to ensure that the server certificate’s validity is verified. Figure 1 shows the new configuration form provided from LTOS Version 7.06.001 onwards.
The function for uploading a root CA certificate was already in place before for LDAPS support and is located in the Web Interface under “Security -> Certificates -> CA Certificates”. It allows the certificates of the CAs (including optionally those of the intermediate CAs) that have signed the certificate of the syslog server. Public certificate authorities are already registered in the firmware, and an updated list is provided with each new version of the LTOS firmware.
This means that it is possible to implement the features broadly without any major adjustments or difficulties and that a connection can be established to a syslog server over TLS quickly.
As chance would have it, Mr Weber was already performing comprehensive tests in relation to “syslog over TLS” using products from other vendors at that time. Given this opportunity, we asked if he’d be willing to test a beta version of LTOS for Meinberg, which he said he’d be happy to do! So, many thanks once again for your support!
Implementation Pitfalls
Mr Weber’s test results with different IP protocols and Fully Qualified Domain Names did reveal a number of flaws, however.
Legacy IP is preferred over IPv6
As soon as an FQDN was entered as a server address, “syslog-ng” would use IPv4 rather than IPv6. If the DNS server returned an AAAA record for an IPv6 address and an A record for an IPv4 address, the IPv6 record would not be used. If only an IPv6 address was returned, the connection would not be established at all. A brief study of the “syslog-ng” documentation revealed that “syslog-ng” assumes a default value of IPv4 unless the desired protocol is explicitly declared, which is not really how name resolution is meant to work. Digging a little further revealed that this behaviour had already been recognized as a bug or a very valuable feature, depending on your point of view. The open issue at https://github.com/syslog-ng/syslog-ng/issues/3386 addresses the request to adjust this behaviour.
Therefore, in order to ensure that “syslog-ng” correctly implements name resolution, additional logic needs to be added. Based on a quick analysis of the code, patching “syslog-ng” itself would not have been a useful investment of time. The much simpler alternative was to draft a small script to perform the name resolution and to adapt the “syslog-ng” configuration to these specific circumstances. This enables a name resolution strategy to be formulated. The parameter “IP PROTOCOL” in /etc/mbg/ntf.cfg can be used to select one of the following options:
- “4” (only IPv4 resolution)
- “6” (only IPv6 resolution),
- “4,6” (IPv4 resolution before IPv6 resolution)
- “6,4” (IPv6 resolution before IPv4 resolution)
The following pseudocode describes the decision-making process. The function “check_if_valid_IPvX” tests if there is a syntactically valid IPv4 address. In addition to the process described by the pseudocode, the IP address can also be “pinged” to determine if it is reachable. The function “resolve_FQDN” triggers a DNS query based on the passed name and returns all recorded IP addresses.
# Pseudocode #IP_CONF = [4],[6],[4,6] or [6,4] #HOST = input of address field of external syslog servers #Return values: ((The (FQDN or IP) and the IP-Protocol) or Error) check_host(IP_CONF, HOST) for IPv in IP_CONF: if IPv == 4: if check_if_valid_IPv4(HOST): return HOST, 4 else resIPs = resolve_FQDN(HOST) for resIP in resIPs: if check_if_valid_IPv4(resIP): return HOST, 4 elif IPv == 6: if check_if_valid_IPv6(HOST): return HOST, 6 else resIPs = resolve_FQDN(HOST) for resIP in resIPs: if check_if_valid_IPv6(resIP): return HOST, 6 return "ERROR no valid IP or FQDN"
An example of a configuration line for an external syslog server with an FQDN reachable via IPv6 in /etc/syslog-ng/syslog-ng.conf would be:
destination syslog_server_1 { network( "syslog.test.example.fqdn" port(6514) ip-protocol(6) time-zone("Etc/GMT-5") transport("tls") tls( ca-file("/etc/ssl/certs/rootca_bundle.pem") peer-verify(yes) ) ); };
Check of Subject Alternative Names
Another problem found during testing was that, despite the “peer-verify” option being activated, the configured syslog server address did not need to match any of the “Subject Alternative Name” (SAN) entries in the certificate if you were using self-signed certificates. While browsers also do not check for a correct “Common Name” (CN) or SAN when a self-signed certificate is in play, the user still at least needs to provide explicit consent to grant such an exception for the certificate. “syslog-ng”, on the other hand, establishes the connection silently without any message output, despite the option “peer-verify(yes)” being explicitly declared. This would be a major security risk, albeit one that is thankfully not a problem with a two-tier PKI, which allows you to have a valid certificate issued by an official PKI to enable it to be recognized as valid by other domains.
Figure 2 shows examples of parameters that would normally be compared whenever the “TLS peer verify” option is enabled. The screenshot shows the “External Syslog Server” configuration form in the LTOS Web Interface at the top and the contents of an X.509 certificate below. The syslog server address entered here must match one of the SAN entries in the syslog server certificate. If there is no SAN entry, the deprecated CN field will be checked instead. A connection can only be established if the syslog server address matches one of the SANs or the CN precisely. Where wildcards are supported, the strings can vary within the scope afforded by the wildcards.
This means that in the case of self-signed certificates, because the server certificate is identical to the certificate on the client, the CN and SAN entries cease to have any real purpose – the ability to move the certificate with the key to a different server provides in its own right the means to get up to all kinds of no good.
For all those who use self-signed certificates despite the above, it’s worth noting that this behaviour deviates significantly from how it should operate according to the documentation (https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide/73\#TOPIC-1829193), whereby “peer-verify = yes” at the very least should compel “syslog-ng” not to trust a certificate if the client does not match a registered SAN or the CN.
Photo by Fabien Bazanegue on Unsplash.