Some years ago I wrote a blog post called “Basic syslog-ng Installation“. While I used it myself quite often in my labs or at the customers’ sites, it shows only basic UDP transport which is both unreliable and insecure. So, let’s have a look at a fresh installation of syslog-ng with TLS support for security reasons. However, TCP and UDP as transport are covered as well for the support of legacy systems.

Prerequisite: X.509 Certificate

The secure transport of log messages relies on a well-known TLS connection. Therefore, the server needs a valid X.509 certificate, commonly known as a TLS certificate or the like. Of course, the server needs the certificate plus its corresponding private key. ;) If you are an enterprise, you can generate/sign a certificate as always within your PKI. If you don’t have your own PKI but still want to use a trusted certificate, you can use Let’s Encrypt, e.g., with the acme.sh script. (Out of scope for this blog post.)

The easiest way is to generate a self-signed certificate for this use case:

• use the FQDN of the syslog server as the common name
• the subject alternative names (SAN) should contain the FQDN as well, and additionally the IP addresses of the server (if your syslog clients use the IP address of the server rather than the FQDN, which is likely)
• though modern browsers only accept certificates that have a lifetime < 1 year, I don’t see this requirement here. Hence I am using 3 years as the lifetime.

TL;DR: Use the following OpenSSL command to generate your certificate. Replace the FQDN and the IP addresses according to your needs:

openssl req -x509 -newkey rsa:2048 -sha256 -days 1096 -nodes -keyout syslog.key -out syslog.crt -subj "/CN=syslog.weberlab.de/C=DE/O=Webernetz.net" -addext "subjectAltName=DNS:syslog.weberlab.de,IP:194.247.5.27,IP:2001:470:1f0b:16b0:20c:29ff:fea8:26f7"

You’ll have two files: syslog.crt and syslog.key. Optionally, you can verify that the .crt file starts with —–BEGIN CERTIFICATE—– while the .key file starts with —–BEGIN PRIVATE KEY—–.

Installation

I am using a Ubuntu 20.04.2 LTS version for this guide. It all starts with:

sudo apt update
sudo apt install syslog-ng

On my fresh Ubuntu, this installed 33 new packages. After the installation, the syslog-ng daemon is already running, though not listening on any interfaces:

weberjoh@vm32-test2:~$ sudo systemctl status syslog-ng.service
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-06-16 10:55:32 UTC; 2min 59s ago
       Docs: man:syslog-ng(8)
   Main PID: 19666 (syslog-ng)
      Tasks: 3 (limit: 38436)
     Memory: 11.9M
     CGroup: /system.slice/syslog-ng.service
             └─19666 /usr/sbin/syslog-ng -F

Jun 16 10:55:32 vm32-test2 systemd[1]: Starting System Logger Daemon...
Jun 16 10:55:32 vm32-test2 syslog-ng[19666]: DIGEST-MD5 common mech free
Jun 16 10:55:32 vm32-test2 systemd[1]: Started System Logger Daemon.

Use  syslog-ng -V to see the installed version. In my case, it’s 3.25.1.

Configuration

Concerning the two certificate files: Change the ownership to root:root, make a folder for them within the syslog-ng folder and move them to that place:

sudo chown root:root syslog.*
sudo mkdir /etc/syslog-ng/cert.d
sudo mv syslog.* /etc/syslog-ng/cert.d/

For the following TLS related config snippets, I used the “syslog-ng Open Source Edition 3.30 – Administration Guide“.

I am using a single configuration file within the “conf.d” subfolder:

sudo nano /etc/syslog-ng/conf.d/generic.conf

Using the following content, the service will listen for IPv6 and IPv4 on UDP (514), TCP (514) and TLS (6514), while creating a new folder for each logging source, year & month, writing into a new file every day. Note that I am using the “syslog” driver for receiving IETF/RFC5424 messages rather than the old “network” driver for merely BSD/RFC3164:

##################################################
options {
        create_dirs(yes);
        owner(root);
        group(root);
        perm(0644);
        dir_owner(root);
        dir_group(root);
        dir_perm(0755);
};

##################################################
source s_network {
        syslog (
                ip-protocol(6)
                transport("udp")
                port(514)
        );
        syslog (
                ip-protocol(6)
                transport("tcp")
                port(514)
        );
        syslog (
                ip-protocol(6)
                port(6514)
                transport("tls")
                tls (
                        cert-file("/etc/syslog-ng/cert.d/syslog.crt")
                        key-file("/etc/syslog-ng/cert.d/syslog.key")
                        peer-verify(optional-untrusted)
                )
        );
};

##################################################
destination d_host-specific {
        file("/var/log/syslog-ng/$HOST/$YEAR/$MONTH/$HOST-$YEAR-$MONTH-$DAY.log");
};

##################################################
log {
        source(s_network);
        destination(d_host-specific);
};

Restarting the server:

sudo systemctl restart syslog-ng.service

Verifying that the service is listening on those three ports:

weberjoh@vm32-test2:~$ sudo ss -tulpen | grep syslog
udp    UNCONN  0       0                          *:514                 *:*      users:(("syslog-ng",pid=249051,fd=12)) ino:825039 sk:6 v6only:0 <->
tcp    LISTEN  0       255                        *:6514                *:*      users:(("syslog-ng",pid=249051,fd=25)) ino:825041 sk:7 v6only:0 <->
tcp    LISTEN  0       255                        *:514                 *:*      users:(("syslog-ng",pid=249051,fd=24)) ino:825040 sk:8 v6only:0 <->

Basically, that’s it. ;) You can now point your clients to this syslog-ng server. When the very first syslog messages are received, the /var/log/syslog-ng and the corresponding subfolders will be created. For example, my syslog client with IP address 2001:470:7250::1 sending logs in June 2021 logs into: /var/log/syslog-ng/2001:470:7250::1/2021/06.

Please note that logs coming through legacy IP connections are stored in the following way: ::ffff:<IPv4-address>. Have a look at this blog post for more details. Example:

weberjoh@vm32-test2:/var/log/syslog-ng$ ls -ahl
total 20K
drwxr-xr-x  5 root root   4.0K Jun 23 09:08 .
drwxr-xr-x 10 root syslog 4.0K Jun 23 07:37 ..
drwxr-xr-x  3 root root   4.0K May 20 08:06 ::ffff:192.168.3.53
drwxr-xr-x  3 root root   4.0K May 20 08:05 ::ffff:192.168.3.87
drwxr-xr-x  3 root root   4.0K May 20 08:05 ::ffff:194.247.5.1
drwxr-xr-x  3 root root   4.0K May 20 08:05 ::ffff:194.247.5.15
drwxr-xr-x  3 root root   4.0K May 20 08:05 ::ffff:194.247.5.16
drwxr-xr-x  3 root root   4.0K May 22 02:05 ::ffff:194.247.5.17
drwxr-xr-x  3 root root   4.0K May 20 12:36 ::ffff:194.247.5.26

TCP Port?!?

One question that took me some time: Which port is used for syslog-over-TCP? Normally, the same port as for UDP should fit, that is: 514. But the TCP port 514 is *not* registered for “syslog” but for “shell”, ref: IANA. This is unlike other common protocols such as DNS, where port 53 is registered for UDP and TCP. Uhm. RFC 1395 lists TCP port 601 for reliable syslog connections, which is listed at the IANA as well. However, RFC 6587 tells this:

TL;DR: Decide whatever you want, but maybe 514 is a good idea because everyone recognizes it as syslog. ;) A Twitter poll from me gave the same result:

Hey network admins: Which port are you using for Syslog over TCP? (There's no well-known port for it…)

— Johannes Weber 🎸 (@webernetz) June 7, 2021

Verification/Troubleshooting

You have multiple ways to troubleshoot the server and especially its TLS configuration. For example, you can use tcpdump to check if connections to ports 514 or 6514 are coming in at all (replace “ens160” with the name of your NIC):

sudo tcpdump -i ens160 port 6514 or port 514

Example connection on TCP 514:

weberjoh@vm32-test2:~$ sudo tcpdump -i ens160 port 6514 or port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
07:40:14.079021 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 2469074641:2469075198, ack 2549261998, win 113, length 557
07:40:14.079077 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 557, win 501, length 0
07:40:16.082117 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 557:1099, ack 1, win 113, length 542
07:40:16.082154 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 1099, win 501, length 0
07:40:19.889419 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 1099:1657, ack 1, win 113, length 558
07:40:19.889466 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 1657, win 501, length 0
07:40:22.079816 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 1657:2190, ack 1, win 113, length 533
07:40:22.079857 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 2190, win 501, length 0
07:40:22.904173 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 2190:2794, ack 1, win 113, length 604
07:40:22.904234 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 2794, win 501, length 0
07:40:24.907202 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 2794:3361, ack 1, win 113, length 567
07:40:24.907286 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 3361, win 501, length 0

Example connection on TLS via TCP 6514:

weberjoh@vm32-test2:~$ sudo tcpdump -i ens160 port 6514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
07:57:10.756095 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [S], seq 3652917499, win 28800, options [mss 1440,sackOK,TS val 721716289 ecr 0,nop,wscale 8], length 0
07:57:10.756171 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [S.], seq 1815308753, ack 3652917500, win 64260, options [mss 1440,sackOK,TS val 527693389 ecr 721716289,nop,wscale 7], length 0
07:57:10.756420 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [.], ack 1, win 113, options [nop,nop,TS val 721716289 ecr 527693389], length 0
07:57:10.756801 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [P.], seq 1:160, ack 1, win 113, options [nop,nop,TS val 721716289 ecr 527693389], length 159
07:57:10.756825 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [.], ack 160, win 501, options [nop,nop,TS val 527693389 ecr 721716289], length 0
07:57:10.762989 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [P.], seq 1:1359, ack 160, win 501, options [nop,nop,TS val 527693395 ecr 721716289], length 1358
07:57:10.763184 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [.], ack 1359, win 124, options [nop,nop,TS val 721716290 ecr 527693395], length 0
07:57:10.764690 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [P.], seq 160:286, ack 1359, win 124, options [nop,nop,TS val 721716290 ecr 527693395], length 126
07:57:10.764713 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [.], ack 286, win 501, options [nop,nop,TS val 527693397 ecr 721716290], length 0
07:57:10.765486 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [P.], seq 1359:1585, ack 286, win 501, options [nop,nop,TS val 527693398 ecr 721716290], length 226
07:57:10.765764 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [P.], seq 286:408, ack 1585, win 135, options [nop,nop,TS val 721716290 ecr 527693398], length 122
07:57:10.765791 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [.], ack 408, win 501, options [nop,nop,TS val 527693398 ecr 721716290], length 0
07:57:10.765871 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [P.], seq 408:620, ack 1585, win 135, options [nop,nop,TS val 721716290 ecr 527693398], length 212
07:57:10.765891 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [.], ack 620, win 500, options [nop,nop,TS val 527693398 ecr 721716290], length 0
07:57:13.768417 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [P.], seq 620:750, ack 1585, win 135, options [nop,nop,TS val 721716591 ecr 527693398], length 130
07:57:13.768475 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [.], ack 750, win 501, options [nop,nop,TS val 527696401 ecr 721716591], length 0
07:57:13.768527 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [FP.], seq 750:874, ack 1585, win 135, options [nop,nop,TS val 721716591 ecr 527693398], length 124
07:57:13.768954 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [F.], seq 1585, ack 875, win 501, options [nop,nop,TS val 527696401 ecr 721716591], length 0
07:57:13.769085 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [.], ack 1586, win 135, options [nop,nop,TS val 721716591 ecr 527696401], length 0

To check for the open ports, use Nmap from a different machine:

sudo nmap -Pn -sS -sU -p 514,6514 <FQDN-of-the-syslog-server>

Examples for IPv6 and legacy IP:

weberjoh@nb17-lx2:~$ sudo nmap -6 -Pn -sS -sU -p 514,6514 test2.weberlab.de

Starting Nmap 7.60 ( https://nmap.org ) at 2021-06-23 07:48 UTC
Nmap scan report for test2.weberlab.de (2001:470:1f0b:16b0:20c:29ff:fea8:26f7)
Host is up (0.00055s latency).
Other addresses for test2.weberlab.de (not scanned): 194.247.5.27

PORT     STATE         SERVICE
514/tcp  open          shell
6514/tcp open          syslog-tls
514/udp  open|filtered syslog
6514/udp closed        syslog-tls
MAC Address: 00:0C:29:A8:26:F7 (VMware)

Nmap done: 1 IP address (1 host up) scanned in 2.19 seconds
weberjoh@nb17-lx2:~$
weberjoh@nb17-lx2:~$
weberjoh@nb17-lx2:~$ sudo nmap -Pn -sS -sU -p 514,6514 test2.weberlab.de

Starting Nmap 7.60 ( https://nmap.org ) at 2021-06-23 07:48 UTC
Nmap scan report for test2.weberlab.de (194.247.5.27)
Host is up (0.00051s latency).
Other addresses for test2.weberlab.de (not scanned): 2001:470:1f0b:16b0:20c:29ff:fea8:26f7

PORT     STATE         SERVICE
514/tcp  open          shell
6514/tcp open          syslog-tls
514/udp  open|filtered syslog
6514/udp closed        syslog-tls
MAC Address: 00:0C:29:A8:26:F7 (VMware)

Nmap done: 1 IP address (1 host up) scanned in 1.85 seconds

It’s correct that UDP 514 is “open|filtered” and UDP 6514 is closed.

To verify the X.509 certificate you can use “openssl s_client”:

openssl s_client -connect <FQDN-of-the-syslog-server>:6514 -showcerts

In my case: (It’s using TLSv1.3 as you can see.)

weberjoh@nb17-lx2:~$ openssl s_client -connect test2.weberlab.de:6514 -showcerts
CONNECTED(00000005)
depth=0 CN = syslog.weberlab.de, C = DE, O = Webernetz.net
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = syslog.weberlab.de, C = DE, O = Webernetz.net
verify return:1
---
Certificate chain
 0 s:CN = syslog.weberlab.de, C = DE, O = Webernetz.net
   i:CN = syslog.weberlab.de, C = DE, O = Webernetz.net
-----BEGIN CERTIFICATE-----
MIIDnjCCAoagAwIBAgIUO/WKKIVLZg/QTR2tFnV8z2pHgk0wDQYJKoZIhvcNAQEL
BQAwQjEbMBkGA1UEAwwSc3lzbG9nLndlYmVybGFiLmRlMQswCQYDVQQGEwJERTEW
MBQGA1UECgwNV2ViZXJuZXR6Lm5ldDAeFw0yMTA2MTYxMDU2NTVaFw0yNDA2MTYx
MDU2NTVaMEIxGzAZBgNVBAMMEnN5c2xvZy53ZWJlcmxhYi5kZTELMAkGA1UEBhMC
REUxFjAUBgNVBAoMDVdlYmVybmV0ei5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDlS5BVOWOFRamFS8sAE7xIX0QBeFTJpckDJfjPCD5YBpbEZFV1
hvx5TTurroZhWuFTOJ2zDcVOAVr14Dd1WmB7nQZdy3w2Em/J5ZUfsUUnuHiOCtOE
TmEhiRivvZ8KspZnyi7Xl0X67y2YB1cConz+SwO3H3vdRg77jYut4y+01X0DyQ5B
evdjTQjm6s/SWa7kAG438uMSeskW+FfRMDuNOlfeXM6UgJ5gjykc+Ps0ENd77kJD
Ois5/HQP9DkHosqJDF90gTSEz+0JV0rEiLGP5X5dBrSfrrPW3pDnVMRKZXvAV4Ju
x5Aky6ULtctTFmcFwsVP8HWL4BZ8CpJMGNKrAgMBAAGjgYswgYgwHQYDVR0OBBYE
FBBVs9EKxGgbqlV+5xyiEuw3ZsudMB8GA1UdIwQYMBaAFBBVs9EKxGgbqlV+5xyi
Euw3ZsudMA8GA1UdEwEB/wQFMAMBAf8wNQYDVR0RBC4wLIISc3lzbG9nLndlYmVy
bGFiLmRlhwTC9wUbhxAgAQRwHwsWsAIMKf/+qCb3MA0GCSqGSIb3DQEBCwUAA4IB
AQB0poPFrYlu5uH9KdapqDQIJ/QPjpblneZBZ3tvguOvBKz/+5rPE/dNYiQ6L8b2
wxP8ks3YyjVIr8mhrRPwsZxg0rLe2DDeVqrp6JsxklPGTR0YL6q8x2JGFtLBzECy
R6VLR5EHWywTBnOzsIj3+9JiMO5y8LfjXhvLt4ljvevwNi50Ce+Oc8c30qdkAb8S
ALhRSaBhdmSZBpxp59yeD8RFgAH/pB9knuaLbt1MFc7AJ8Mwh0NJBlBllSBq49ft
Gw9YPU5WskcRQCfic0spgC8etpXdQ9doUvOpmAa3IJjQxn8PbCIV3Owx79TL/2gn
lTjvgGTf4XyktwwR8JkgCpj8
-----END CERTIFICATE-----
---
Server certificate
subject=CN = syslog.weberlab.de, C = DE, O = Webernetz.net

issuer=CN = syslog.weberlab.de, C = DE, O = Webernetz.net

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1486 bytes and written 399 bytes
Verification error: self signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self signed certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: C86198747196918F79D8672891201203F2C9C477EC6862A2DB3BAF9EB3E6D1C1
    Session-ID-ctx:
    Resumption PSK: E6F64A2948EF5D1071A2101A47ABAEAFB4CA7668B4D86441A2F992DF9C26EB3F50149A64DFD8B03321C42EB5D762C506
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 11 a1 ff 4f e4 8c 05 98-4f 35 7e a4 33 f9 f1 58   ...O....O5~.3..X
    0010 - fb 6f 18 1a 5e 99 cd fa-bb 9a e6 63 f2 86 d6 62   .o..^......c...b
    0020 - 04 8a 03 7a 4f 14 58 ee-2c e7 62 94 07 a8 46 7a   ...zO.X.,.b...Fz
    0030 - 5a f4 fd ec 33 55 95 b1-f7 54 d6 3b e6 3e d3 bc   Z...3U...T.;.>..
    0040 - a0 8c c5 4b da c7 81 4f-1f ce c4 dd 32 3f 3c f5   ...K...O....2?<.
    0050 - 35 ef 00 aa 00 af a9 f8-60 c8 a0 a7 a4 61 df 37   5.......`....a.7
    0060 - 85 d8 0e 6b dd 78 8d 82-dc e8 63 23 ee 61 9d 49   ...k.x....c#.a.I
    0070 - a1 a6 a5 05 8c 5b 14 63-a7 88 09 80 f8 3c 66 50   .....[.c.....<fP
    0080 - 6f 4c 16 b4 44 c6 12 cb-93 d5 38 1c 7c 0c 28 7d   oL..D.....8.|.(}
    0090 - 18 6c 01 c4 66 8e 7d e5-f3 74 8e bb fe f5 77 35   .l..f.}..t....w5
    00a0 - 4b 8f 0d 84 b9 2a 45 86-31 eb 9a a5 a6 ff 4b 4f   K....*E.1.....KO
    00b0 - fa 20 68 55 1a 68 56 15-ef 30 e1 ed 05 90 0f 34   . hU.hV..0.....4
    00c0 - ec 6c ad 9c 48 39 e7 13-9d 5c c5 ea eb e6 ab b2   .l..H9...\......

    Start Time: 1624434726
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
^C

And to enumerate the provided TLS ciphers, use Nmap again. However, please note that Nmap does not support TLSv1.3 at the moment (Nmap 7.91):

nmap --script ssl-enum-ciphers -Pn -p 6514 <FQDN-of-the-syslog-server>

My example:

weberjoh@nb17-lx2:~$ nmap --script ssl-enum-ciphers -Pn -p 6514 test2.weberlab.de
Host discovery disabled (-Pn). All addresses will be marked 'up' and scan times will be slower.
Starting Nmap 7.91 ( https://nmap.org ) at 2021-06-23 08:31 UTC
Nmap scan report for test2.weberlab.de (194.247.5.27)
Host is up (0.00049s latency).
Other addresses for test2.weberlab.de (not scanned): 2001:470:1f0b:16b0:20c:29ff:fea8:26f7

PORT     STATE SERVICE
6514/tcp open  syslog-tls
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) - A
|       TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

Nmap done: 1 IP address (1 host up) scanned in 0.82 seconds

A Little Capture

And by the way: This is what it looks like in Wireshark. Standard TLS handshake, that is: client hello, server hello, certificate, server key exchange, client key exchange, and so on, finally followed by (unreadable since encrypted) application data:

You can find these captured packets in my most recent Ultimate PCAP.

All the best! God bless you.

Featured image “journals” by Meagan is licensed under CC BY-NC 2.0.

As we have just set up a TLS capable syslog server, let’s configure a Palo Alto Networks firewall to send syslog messages via an encrypted channel. While it was quite straightforward to configure I ran into a couple of (unresolved) problems as I added and deleted some syslog servers and their certificates. Uhm.

Syslog & Certificate Configuration

I am using a PA-220 with PAN-OS 10.0.7 for this setup. My syslog-ng server with version 3.25.1 is running on a Ubuntu 20.04.3 LTS.

Basically, you just have to set the transport of a syslog server to “SSL”. (For whatever reason it’s still named SSL. It should read TLS since a couple of years.) I am using the “IETF” format nowadays. This seems to be the more modern standard, refer to RFC 5424 Appendix A.1. Though I don’t see any obvious differences between the BSD and the IETF type.

If you’re doing an immediate commit you will see these system logs:

Syslog  SSL error while writing stream; tls_error='SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed'

Screenshot:

–> This is because I am using a self-signed x.509 certificate on my syslog-ng server which is not trusted indeed. Good so far!

The same errors are appearing at the /var/log/syslog log file on the syslog-ng server itself:

Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Syslog connection accepted; fd='42', client='AF_INET6([::ffff:94.31.107.114]:43087)', local='AF_INET6([::]:6514)'
Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca', location='/etc/syslog-ng/conf.d/generic.conf:28:17'
Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Error reading RFC6587 style framed data; fd='42', error='Connection reset by peer (104)'
Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Syslog connection closed; fd='42', client='AF_INET6([::ffff:94.31.107.114]:43087)', local='AF_INET6([::]:6514)'

You have to import the certificate of the syslog-ng server into the Palo as well:

Now it’s working. ;)

Note that for some reason the Palo does NOT use IPv6 for this outgoing syslog connection, though my FQDN had an AAAA record at the time of writing and the syslog server itself was accessible.

weberjoh@nb15-lx:~$ host test2.weberlab.de
test2.weberlab.de has address 194.247.5.27
test2.weberlab.de has IPv6 address 2001:470:1f0b:16b0:20c:29ff:fea8:26f7

This is what the legacy IP TLS connection looks like with Wireshark:

Some maloperations?!?

Since I wanted to use IPv6, I configured the IPv6 address of the syslog-ng server manually:

The TCP handshake to the syslog server worked, but no TLS connection was established. I got the same errors in the system log:

And this is it in Wireshark:

After trying that I decided to go back to the FQDN of my syslog-ng server to let the Palo use it with legacy IP. However, I was not able to get it working again at all! I tried the following:

1. without certificate -> not working
2. imported certificate -> working
3. deleted certificate -> still working (!!!???)
4. change of the syslog server to a dummy
5. change back to my real one -> not working (ok)
6. imported certificate again -> still not working (not ok!)
7. reboot of the whole device -> still not working

Hence I suppose some kind of bugs here. :(

Anyway, lessons learned:

Photo by Claudio Testa on Unsplash.

As we have just set up a TLS capable syslog server, let’s configure a Fortinet FortiGate firewall to send syslog messages via an encrypted channel (TLS). Let’s go:

I am using a Fortinet FortiGate (FortiWiFi) FWF-61E with FortiOS v6.4.7 build1911 (GA) for this tutorial. My syslog-ng server with version 3.13.2 is running on Ubuntu 18.04.6 LTS.

You cannot configure any syslog server details (rather than the address itself) via the GUI on this so-called “Next Generation Firewall”. Log & Report -> Log Settings -> Remote Logging and Archiving:

Hence we have to use the CLI:

config log syslogd setting
    set status enable
    set server "syslog2.weberlab.de"
    set mode reliable
    set port 6514
    set enc-algorithm high
end

Some notes:

• You can’t “set enc-algorithm high” unless you have “set mode reliable”.
• You don’t have to “set port 6514” manually because it is set automatically by doing the “set enc-algorithm high” command.
• Using an FQDN as the server (as I did in the listing), the FortiGate will use legacy IP though an AAAA record is present. If you want to use IPv6 you must use an IPv6 address here.
• ;)
• Official Fortinet CLI reference

If your syslog server uses a self-signed or untrusted certificate it won’t work right now. This is good from a security point of view. You’ll see the following syslog messages on your syslog server: “tlsv1 alert unknown ca”:

weberjoh@nb17-lx2:~$ tail -f /var/log/syslog
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection accepted; fd='96', client='AF_INET6([2001:470:1f0b:16b0::1]:15831)', local='AF_INET6([::]:6514)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection accepted; fd='99', client='AF_INET6([2001:470:1f0b:16b0::1]:15830)', local='AF_INET6([::]:6514)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: I/O error occurred while reading; fd='96', error='Connection reset by peer (104)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection closed; fd='96', client='AF_INET6([2001:470:1f0b:16b0::1]:15831)', local='AF_INET6([::]:6514)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: I/O error occurred while reading; fd='99', error='Connection reset by peer (104)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection closed; fd='99', client='AF_INET6([2001:470:1f0b:16b0::1]:15830)', local='AF_INET6([::]:6514)'

Or looking at it with Wireshark you’ll see this “Alert Message”:

Import the syslog x.509 certificate at System -> Certificates -> Import -> CA Certificate:

Logging via TLS will immediately start after that. No further configuration is needed.

Confirmed with Wireshark again: “Change Cipher Spec” followed by Application Data. Nice!

Very good. Keep in mind that you should not use unencrypted management protocols at all. Hence this one here is a step in the right direction.

Photo by Robin Spielmann on Unsplash.

From time to time I stumble upon Tweets about counting the number of IPv6 addresses (1 2 3). While I think it is ok to do it that way when you’re new to IPv6 and you want to get an idea of it, it does not make sense at all because the mere number of IPv6 addresses is ridiculously high and only theoretically, but has no relevance for the real-world at all. Let me state why:

1. First of all, the basic concept of IPv6 is not about the number of addresses such as “how to choose the subnet size appropriate to the number of clients I’m expecting”, but about the number of prefixes such as “how to subdivide my overall network into logical areas like security zone or business units”. It’s not about concrete IPv6 addresses at all. For each of your /64 subnets, which is the one and only subnet size you should ever use, you have 2^64 host addresses, which is de facto indefinitely.
2. The underlying network infrastructure must somehow deal with the number of addresses. Most commonly Ethernet is used, which relies on MAC addresses. Switches have to take care of their MAC address tables, while routers are using the neighbor cache (the equivalent to the ARP cache for legacy IP) to know where to forward the Ethernet frame. Obviously, every single IPv6 address must be in the neighbor cache of a router. And those tables are limited in space, which is at most something like 512 k entries nowadays. You can argue that you can route an entire /64 to a single machine, using those addresses for vhosts or containers or something like this. Though, in the end, we won’t see many more than a couple of thousands of unique IPv6 addresses per subnet as they still require a state to be maintained.

If you want to get an idea about the advantages of IPv6 addresses, have a look at this poster. For more information about developing an IPv6 address plan for your enterprise, refer to this book, to this podcast, to this post, or similar. 👍

But please don’t use IPv6 exhaustion counters or debate about whether or not we have enough IPv6 addresses for every atom on the surface of the Earth. 🤦‍♂️

Photo by Towfiqu barbhuiya on Unsplash.

Fortunately, there was a SharkFest – the “Wireshark Developer and User Conference” – this year in Europe again. I was there and gave an IPv6 Crash Course likewise. Yeah! It’s my favourite topic, you know. 75 minutes full of content, hence the name crash course.

Here are my slides as well as the video recording. If you want a crash course for IPv6, here we go:

(The camera was moved to a better position after a few  minutes.)

Download of the slides:

Topics covered

• legacy IP problems which IPv6 solves
• IPv6 addresses: representation, abbreviations, link-local, GUA, basic subnet sizes
• IPv6 address assignment: static, stateful DHCPv6, SLAAC (RS, RA, DAD) with or without stateless DHCPv6, RDNSS, privacy extensions
• link-layer address resolution: NS, NA, solicited-node multicast address
• Wireshark display filters & coloring rules, ICMPv6 everywhere
• routing: static, OSPFv3, MP-BGP
• security: differences/challenges compared to legacy IP
• tips about where to start
• transition period: DNS64 & NAT64
• FAQs: disabling IPv6? ULAs?
• literature, links, podcasts, blogs, mailing lists
• stickers 😉

Blogposts referenced

• I Love IPv6 Addressing!
• Basic IPv6 Messages: Wireshark Capture
• IPv6 Upper Layer Protocol Samples
• The Ultimate PCAP
• OSPFv3 for IPv6 Lab: Cisco, Fortinet, Juniper, Palo Alto, Quagga
• Basic MP-BGP Lab: Cisco Router, Palo Alto, Fortinet

Photo by Gerald Schömbs on Unsplash.

For some reason, I came across a blog post by Gian Paolo called Small servers. This reminded me of some fairly old network protocols (that no one uses as far as I know) that are not in my Ultimate PCAP yet. Hence I took some minutes, captured them, and took some Wireshark screenshots. They are: echo, discard, daytime, chargen, and time. Mostly via TCP and UDP, and, as you would have expected, IPv6 and legacy IP.

I’m aware that this is not of interest to most of you. :) But for the sake of completeness, and because I love adding new protocols to the Ultimate PCAP, I added them though.

I used an old Cisco 2811 router with IOS version 15.1(4)M12a for this:

service udp-small-servers
service tcp-small-servers

The registered transport protocol port numbers and the transport layer protocols implemented on Cisco routers are:

For the daytime to work with UDP, I queried my Meinberg LANTIME M200 at ntp3.weberlab.de (AAAA) and ntp3-legacy-ip.weberlab.de (A) respectively. Furthermore, I also queried the “time” protocol, port number 37, against the Meinberg since it is not implemented by Ciscos small-servers.

Some Nmap scans, just for reference:

weberjoh@h2877111:~$ nmap -6 router1.weberlab.de
Starting Nmap 7.80 ( https://nmap.org ) at 2022-10-20 13:08 CEST
Nmap scan report for router1.weberlab.de (2001:470:1f0a:319::2)
Host is up (0.033s latency).
Other addresses for router1.weberlab.de (not scanned): 37.24.166.89
rDNS record for 2001:470:1f0a:319::2: tunnel643592-pt.tunnel.tserv6.fra1.ipv6.he.net
Not shown: 996 closed ports
PORT   STATE SERVICE
7/tcp  open  echo
9/tcp  open  discard
13/tcp open  daytime
19/tcp open  chargen

Nmap done: 1 IP address (1 host up) scanned in 8.79 seconds
weberjoh@h2877111:~$
weberjoh@h2877111:~$
weberjoh@h2877111:~$ nmap router1.weberlab.de
Starting Nmap 7.80 ( https://nmap.org ) at 2022-10-20 13:07 CEST
Nmap scan report for router1.weberlab.de (37.24.166.89)
Host is up (0.047s latency).
Other addresses for router1.weberlab.de (not scanned): 2001:470:1f0a:319::2
rDNS record for 37.24.166.89: ip-037-024-166-089.um08.pools.vodafone-ip.de
Not shown: 996 closed ports
PORT   STATE SERVICE
7/tcp  open  echo
9/tcp  open  discard
13/tcp open  daytime
19/tcp open  chargen

Nmap done: 1 IP address (1 host up) scanned in 5.82 seconds
weberjoh@h2877111:~$
weberjoh@h2877111:~$
weberjoh@h2877111:~$ sudo nmap -6 -sU -p 7,9,13,19 router1.weberlab.de
Starting Nmap 7.80 ( https://nmap.org ) at 2022-10-20 13:19 CEST
Nmap scan report for router1.weberlab.de (2001:470:1f0a:319::2)
Host is up (0.10s latency).
Other addresses for router1.weberlab.de (not scanned): 37.24.166.89
rDNS record for 2001:470:1f0a:319::2: tunnel643592-pt.tunnel.tserv6.fra1.ipv6.he.net

PORT   STATE  SERVICE
7/udp  open   echo
9/udp  closed discard
13/udp closed daytime
19/udp closed chargen

Nmap done: 1 IP address (1 host up) scanned in 2.15 seconds
weberjoh@h2877111:~$
weberjoh@h2877111:~$
weberjoh@h2877111:~$ sudo nmap -sU -p 7,9,13,19 router1.weberlab.de
Starting Nmap 7.80 ( https://nmap.org ) at 2022-10-20 13:18 CEST
Nmap scan report for router1.weberlab.de (37.24.166.89)
Host is up (0.030s latency).
Other addresses for router1.weberlab.de (not scanned): 2001:470:1f0a:319::2
rDNS record for 37.24.166.89: ip-037-024-166-089.um08.pools.vodafone-ip.de

PORT   STATE         SERVICE
7/udp  open          echo
9/udp  open|filtered discard
13/udp closed        daytime
19/udp open|filtered chargen

Nmap done: 1 IP address (1 host up) scanned in 10.70 seconds
weberjoh@h2877111:~$
weberjoh@h2877111:~$
weberjoh@h2877111:~$ sudo nmap -6 -sU -p 13 ntp3.weberlab.de
Starting Nmap 7.80 ( https://nmap.org ) at 2022-11-18 08:55 CET
Nmap scan report for ntp3.weberlab.de (2001:470:1f0b:16b0::dcfb:123)
Host is up (0.022s latency).

PORT   STATE SERVICE
13/udp open  daytime

Nmap done: 1 IP address (1 host up) scanned in 4.21 seconds
weberjoh@h2877111:~$
pi@pi05-random:~ $
pi@pi05-random:~ $ sudo nmap -sU -p 13 ntp3-legacy-ip.weberlab.de

Starting Nmap 7.40 ( https://nmap.org ) at 2022-11-18 11:55 CET
Nmap scan report for ntp3-legacy-ip.weberlab.de (194.247.5.12)
Host is up (0.00048s latency).
PORT   STATE SERVICE
13/udp open  daytime
MAC Address: 00:13:95:24:34:04 (congatec AG)

Nmap done: 1 IP address (1 host up) scanned in 1.33 seconds

Calling Them: Basically Telnet

To be honest, all of those protocols basically rely on mere TCP or UDP, just like HTTP or SMTP or all the other plaintext protocols out there. That is: telnet and netcat for these scenarios here. Now for all those 5 protocols, I did 4x calls each, namely TCP for IPv6 and legacy IP, as well as UDP for IPv6 and legacy IP. I referenced the protocols by either their name or their numerical number: (Reminder: Exiting telnet with ^] which is “Strg +” on a german keyboard.)

###echo
#exiting telnet with "Strg +" and "quit"
telnet router1.weberlab.de echo
telnet -4 router1.weberlab.de echo
netcat -u router1.weberlab.de 7
netcat -4 -u router1.weberlab.de 7

###discard
#exiting telnet with "Strg +" and "quit"
telnet router1.weberlab.de discard
telnet -4 router1.weberlab.de discard
netcat -u router1.weberlab.de 9
netcat -4 -u router1.weberlab.de 9

###daytime
#exiting telnet with "Strg +" and "quit"
telnet router1.weberlab.de daytime
telnet -4 router1.weberlab.de daytime
telnet ntp3.weberlab.de 13
telnet ntp3-legacy-ip.weberlab.de 13
#UDP needs an enter key to respond
netcat -u ntp3.weberlab.de 13
netcat -u ntp3-legacy-ip.weberlab.de 13

###chargen
#exiting telnet with "Strg +" and "quit"
telnet router1.weberlab.de chargen
telnet -4 router1.weberlab.de chargen
netcat -u router1.weberlab.de 19
netcat -4 -u router1.weberlab.de 19

###time
#exiting telnet with "Strg +" and quit
telnet ntp3.weberlab.de time
telnet ntp3-legacy-ip.weberlab.de time
#UDP needs an enter key to respond
netcat -u ntp3.weberlab.de 37
netcat -u ntp3-legacy-ip.weberlab.de 37

The complete story is this:

weberjoh@h2877111:~$ telnet router1.weberlab.de echo
Trying 2001:470:1f0a:319::2...
Connected to router1.weberlab.de.
Escape character is '^]'.
hello world
hello world
yup
yup
here we go
here we go
^]
telnet> quit
Connection closed.
weberjoh@h2877111:~$ telnet -4 router1.weberlab.de echo
Trying 37.24.166.89...
Connected to router1.weberlab.de.
Escape character is '^]'.
same here
same here
but with legacy IP ;D
but with legacy IP ;D
^]
telnet> quit
Connection closed.
weberjoh@h2877111:~$ netcat -u router1.weberlab.de 7
what about UDP
what about UDP
yeah
yeah
^C
weberjoh@h2877111:~$ netcat -4 -u router1.weberlab.de 7
as well as for legacy IP here
as well as for legacy IP here
ciao
ciao
^C
weberjoh@h2877111:~$ telnet router1.weberlab.de discard
Trying 2001:470:1f0a:319::2...
Connected to router1.weberlab.de.
Escape character is '^]'.
discarding everything



^]
telnet> quit
Connection closed.
weberjoh@h2877111:~$ telnet -4 router1.weberlab.de discard
Trying 37.24.166.89...
Connected to router1.weberlab.de.
Escape character is '^]'.
foo bar
^]
telnet> quit
Connection closed.
weberjoh@h2877111:~$ netcat -u router1.weberlab.de 9
was sollen wir trinken?
weberjoh@h2877111:~$ netcat -4 -u router1.weberlab.de 9
sieben Tage lang
was sollen wir trinken?
so ein Durst!
^C
weberjoh@h2877111:~$ telnet router1.weberlab.de daytime
Trying 2001:470:1f0a:319::2...
Connected to router1.weberlab.de.
Escape character is '^]'.
Friday, November 18, 2022 13:03:03-CET
Connection closed by foreign host.
weberjoh@h2877111:~$ telnet -4 router1.weberlab.de daytime
Trying 37.24.166.89...
Connected to router1.weberlab.de.
Escape character is '^]'.
Friday, November 18, 2022 13:03:12-CET
Connection closed by foreign host.
weberjoh@h2877111:~$ telnet ntp3.weberlab.de 13
Trying 2001:470:1f0b:16b0::dcfb:123...
Connected to ntp3.weberlab.de.
Escape character is '^]'.
18 NOV 2022 12:03:18 UTC
Connection closed by foreign host.
weberjoh@h2877111:~$ telnet ntp3-legacy-ip.weberlab.de 13
Trying 194.247.5.12...
Connected to ntp3-legacy-ip.weberlab.de.
Escape character is '^]'.
18 NOV 2022 12:03:21 UTC
Connection closed by foreign host.
weberjoh@h2877111:~$ netcat -u ntp3.weberlab.de 13

18 NOV 2022 12:03:26 UTC

18 NOV 2022 12:03:27 UTC

18 NOV 2022 12:03:29 UTC

18 NOV 2022 12:03:30 UTC

18 NOV 2022 12:03:30 UTC

18 NOV 2022 12:03:30 UTC

18 NOV 2022 12:03:30 UTC

18 NOV 2022 12:03:30 UTC
^C
weberjoh@h2877111:~$ netcat -u ntp3-legacy-ip.weberlab.de 13






^C
weberjoh@h2877111:~$ telnet router1.weberlab.de chargen
Trying 2001:470:1f0a:319::2...
Connected to router1.weberlab.de.
Escape character is '^]'.
 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefg
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefgh
"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghi
#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghij
$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijk
%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijkl
&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklm
'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmn
()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmno
)*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnop
*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopq
+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
^]
telnet> quit
Connection closed.
weberjoh@h2877111:~$ telnet -4 router1.weberlab.de chargen
Trying 37.24.166.89...
Connected to router1.weberlab.de.
Escape character is '^]'.
 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefg
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefgh
"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghi
#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghij
$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijk
%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijkl
&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklm
'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmn
()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmno
)*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnop
*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopq
+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
/0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
^]
telnet> quit
Connection closed.
weberjoh@h2877111:~$ netcat -u router1.weberlab.de 19

weberjoh@h2877111:~$ netcat -4 -u router1.weberlab.de 19

asdf


^C
weberjoh@h2877111:~$ telnet ntp3.weberlab.de time
Trying 2001:470:1f0b:16b0::dcfb:123...
Connected to ntp3.weberlab.de.
Escape character is '^]'.
▒!▒<Connection closed by foreign host.
weberjoh@h2877111:~$ telnet ntp3-legacy-ip.weberlab.de time
Trying 194.247.5.12...
Connected to ntp3-legacy-ip.weberlab.de.
Escape character is '^]'.
▒!▒@Connection closed by foreign host.
weberjoh@h2877111:~$ netcat -u ntp3.weberlab.de 37

▒!▒D
▒!▒E
▒!▒F^C
weberjoh@h2877111:~$ netcat -u ntp3-legacy-ip.weberlab.de 37



^C
weberjoh@h2877111:~$

Note that the “time” protocol is the only one which does not display ASCII characters in the telnet session, but kind of junk. ;) This is due to its format: “The server then sends the time as a 32-bit unsigned integer in binary format and in network byte order, representing the number of seconds since 00:00 (midnight) 1 January, 1900 GMT.”

Under the Magnifying Glass: Wireshark

Let’s have a brief look at those protocols with Wireshark.

Echo

The Echo protocol simply sends back everything it has received. Specified in RFC 862. Fun fact: The whole RFC is only ONE PAGE!!!

As you can see in the screenshot, I’ve done it via TCP and UDP, both for IPv6 and legacy IP. A good time to point to the “Follow TCP Stream” or “Follow UDP Stream” again:

Discard

Uh, as of now (November 2022), Wireshark does not yet recognize the “Discard” protocol at port 9. I added a feature request here. Use this display filter to find it nevertheless: tcp.port eq 9 or udp.port eq 9

The Discard service, RFC 863, simply discards everything it has received. For TCP, everything is ACKed at least (no data is sent back, though), while for UDP you won’t see any packets from the server. However, seems like my router did not listen on the Discard port 9 for UDP on IPv6, since an ICMPv6 destination unreachable -> port unreachable came back:

Daytime

“A daytime service simply sends the current date and time as a character string without regard to the input”, RFC 867. Different implementations send different structures of pure ASCII letters:

Chargen

The Character Generator Protocol “simply sends data without regard to the input”, RFC 864. My Cisco router only replied by TCP though. Good example to “Follow TCP Stream” again:

Time

Finally, the Time protocol on port 37, RFC 868, returns the number of seconds since 01.01.1900. Wireshark decodes it:

The End

And yes, at the very end, I disabled those small-servers again on the Cisco router:

no service udp-small-servers
no service tcp-small-servers

Photo by Kris-Mikael Krister on Unsplash.

The other day I just wanted to capture some basic Linux traceroutes but ended up troubleshooting different traceroute commands and Wireshark display anomalies. Sigh. Anyway, I just added a few Linux traceroute captures – legacy and IPv6 – to the Ultimate PCAP. Here are some details:

Yep, I was shocked myself, but while I had some Windows traceroutes (based on ICMP echo-requests) and even some layer 4 traceroutes (sending TCP SYNs, e.g.) in the Ultimate PCAP, I was missing a generic Linux traceroute that uses UDP datagrams to destination ports ≥ 33434 for it. So, here we are.

Traceroute with Legacy IP

I used an Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-234-generic x86_64):

weberjoh@nb15-lx:~$ traceroute -V
Modern traceroute for Linux, version 2.0.21
Copyright (c) 2008  Dmitry Butskoy,   License: GPL v2 or any later
weberjoh@nb15-lx:~$
weberjoh@nb15-lx:~$
weberjoh@nb15-lx:~$ traceroute netsec.blog
traceroute to netsec.blog (5.35.226.136), 30 hops max, 60 byte packets
 1  192.168.3.1 (192.168.3.1)  14.003 ms  13.960 ms  13.932 ms
 2  ip-037-024-166-081.um08.pools.vodafone-ip.de (37.24.166.81)  15.423 ms  16.794 ms  15.892 ms
 3  ip-178-200-118-001.um45.pools.vodafone-ip.de (178.200.118.1)  29.912 ms  33.328 ms  33.087 ms
 4  ip-081-210-129-132.um21.pools.vodafone-ip.de (81.210.129.132)  32.901 ms  33.709 ms  33.813 ms
 5  de-fra04d-rc1-ae-7-0.aorta.net (84.116.197.245)  41.520 ms  41.025 ms  40.964 ms
 6  de-fra04c-ri1-ae15-101.aorta.net (84.116.191.6)  37.400 ms  18.572 ms  18.346 ms
 7  et-7-0-0-u100.fra1-cr-polaris.bb.gdinf.net (80.81.192.239)  20.243 ms  20.314 ms  20.112 ms
 8  ae0.fra10-cr-antares.bb.gdinf.net (87.230.115.1)  20.941 ms  20.509 ms  24.838 ms
 9  ae2.cgn1-cr-nashira.bb.gdinf.net (87.230.114.4)  25.985 ms  26.216 ms  26.038 ms
10  ae0.100.sr-jake.cgn1.dcnet-emea.godaddy.com (87.230.114.222)  38.922 ms  38.912 ms  24.242 ms
11  * * *
12  wp367.webpack.hosteurope.de (5.35.226.136)  23.472 ms  23.407 ms  19.353 ms

Pro-tip for Wireshark: Use a custom column which shows the hop limit (IPv6) and TTL (IPv4) in one column:

By default, traceroute sends three packets with the same hop limit/TTL starting by 1. This results in various UDP streams. At the time of writing with Wireshark version 4.0.1, the coloring of those packets is this: the first 12 UDP streams are marked red (TTL 1-4), while the other ones aren’t. (We are currently discussing the default coloring rules here at the Wireshark issues pages.) Right after it, the ICMP “time-to-live exceeded” error messages came in, while traceroute itself sent out various reverse DNS queries (omitted in the following screenshot):

This was the moment when I discovered a small bug within Wireshark (from my point of view), namely that the connecting line for ICMP errors is not working in both directions.

traceroute6 ≠ traceroute -6

Then it turned a little strange. I was always confused about which CLI command I should use for ping and/or traceroute for IPv6: traceroute6 or traceroute -6. The man page from traceroute states:

While the man page from traceroute6 states:

This is a traceroute6 call:

weberjoh@nb15-lx:~$ traceroute6 -V
traceroute6 utility, iputils-s20121221
weberjoh@nb15-lx:~$
weberjoh@nb15-lx:~$
weberjoh@nb15-lx:~$ traceroute6 netsec.blog
traceroute to netsec.blog (2a01:488:42:1000:50ed:8588:8a:c570) from 2001:470:7250::b15:22, 30 hops max, 24 byte packets
 1  pa-dmz.weberlab.de (2001:470:7250::1)  1.058 ms  0.287 ms  0.181 ms
 2  router1-fa0-1.weberlab.de (2001:470:1f0b:319::1)  0.835 ms  1.234 ms  1.036 ms
 3  tunnel643592.tunnel.tserv6.fra1.ipv6.he.net (2001:470:1f0a:319::1)  14.686 ms  16.142 ms  13.943 ms
 4  * * *
 5  et-7-0-0-u100.fra1-cr-polaris.bb.gdinf.net (2001:7f8::5125:f1:1)  12.482 ms  10.207 ms  11.824 ms
 6  ae0.fra10-cr-antares.bb.gdinf.net (2a01:488:bb03:100::3)  19.943 ms  12.124 ms  11.887 ms
 7  ae2.cgn1-cr-nashira.bb.gdinf.net (2a01:488:bb02:101::2)  11.712 ms  29.159 ms  17.87 ms
 8  ae0.sr-jake.cgn1.dcn.heg.com (2a01:488:bb::223)  28.083 ms  16.166 ms  16.063 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

(Don’t worry about the count to 30 hops for this. The destination or a firewall somewhere there is not responding, hence it counts to infinity which is 30 for traceroute.)

But within Wireshark, it looked like this (omitting the ICMPv6 messages here). That is: Only one static UDP destination port, hence only one UDP stream as seen by Wireshark. Furthermore, a different data portion:

(Don’t worry that no packets at all are marked red (compared to the legacy IP ones), this is part of this discussion as well. It might change with Wireshark versions > 4.0.)

That’s why I did a traceroute -6, this time from another machine: Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0 x86_64):

weberjoh@h2877111:~$ traceroute -V
Modern traceroute for Linux, version 2.1.0
Copyright (c) 2016  Dmitry Butskoy,   License: GPL v2 or any later
weberjoh@h2877111:~$
weberjoh@h2877111:~$
weberjoh@h2877111:~$ traceroute -6 netsec.blog
traceroute to netsec.blog (2a01:488:42:1000:50ed:8588:8a:c570), 30 hops max, 80 byte packets
 1  * * *
 2  491.ae12.core-b1.as6724.net (2a01:238:b:105::1)  0.285 ms  0.271 ms  0.210 ms
 3  110.ae14.bb-rt1-2.h20.r22.rs.ber.de.as6724.net (2a01:238:b:3901::1)  0.400 ms  0.343 ms 110.ae13.bb-rt1-1.e18.r23.rs.ber.de.as6724.net (2a01:238:b:3801::1)  0.537 ms
 4  ae-0-88.bb-a.ak.ber.de.oneandone.net (2001:8d8:0:3::99)  0.852 ms  0.817 ms ae-1-88.bb-a.ak.ber.de.oneandone.net (2001:8d8:0:3::a1)  0.799 ms
 5  ae-13.bb-a.tp.kae.de.net.ionos.com (2001:8d8:0:2::1d2)  14.510 ms  14.451 ms  14.365 ms
 6  ae-2.bb-b.bs.kae.de.net.ionos.com (2001:8d8:0:5::4)  16.837 ms  16.863 ms  16.871 ms
 7  ae-9.bb-b.fr7.fra.de.net.ionos.com (2001:8d8:0:2::11e)  16.884 ms  16.905 ms  16.964 ms
 8  et-7-0-0-u100.fra1-cr-polaris.bb.gdinf.net (2001:7f8::5125:f1:1)  16.417 ms  16.441 ms  16.403 ms
 9  ae0.fra10-cr-antares.bb.gdinf.net (2a01:488:bb03:100::3)  17.188 ms  17.259 ms  17.183 ms
10  ae2.cgn1-cr-nashira.bb.gdinf.net (2a01:488:bb02:101::2)  19.992 ms  19.405 ms  19.340 ms
11  ae0.sr-jake.cgn1.dcn.heg.com (2a01:488:bb::223)  59.756 ms  59.678 ms  59.631 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Which now looks more familiar compared to the traceroute for legacy IP:

By the way: This is what the traffic log from a Palo Alto Networks NGFW looks like for those two tools:

Anyway, those traceroutes are now within the Ultimate PCAP along with the ICMPv6/ICMP errors coming back and the DNS queries made by those traceroute tools. Thus you can investigate them by yourself. Note that those Linux traceroutes are not a protocol by themselves, hence there is no direct display filter. You can use something like this:

udp.dstport in { 33434 .. 33534 } and ( ip.ttl < 31 || ipv6.hlim < 31 )

Want to join the discussion?

Some tweets about those investigations during the last weeks:

In #DualStack environments, traceroute will use legacy IP primarily. AHHH. Why?!? #IPv6 pic.twitter.com/shLuS1n1aZ

— Johannes Weber 🎸 (@webernetz) October 21, 2022

Ouch! The man page states "traceroute6 is equivalent to traceroute -6" and it's NOT! While "traceroute6" sends all datagrams to the same dst.port (same stream), "traceroute -6" sends each datagram to an increased dst.port (different streams). Different tools! #IPv6 #details pic.twitter.com/2k3qD6KpQj

— Johannes Weber 🎸 (@webernetz) November 14, 2022

Yeah, that happens to me when I “just want to have a short look at something”. 🤪

Photo by Nick Seagrave on Unsplash.

In general, Network Address Translation (NAT) solves some problems but should be avoided wherever possible. It has nothing to do with security and is only a short-term solution on the way to IPv6. (Yes, I know, the last 20 years have proven that NAT is used everywhere every time. 😉) This applies to all kinds of NATs for IPv4 (SNAT, DNAT, PAT) as well as for NPTv6 and NAT66.

However, there are two types of NATs that do not only change the network addresses but do a translation between the two Internet Protocols, that is IPv4 <-> IPv6 and vice versa. Let’s focus on NAT46 this time. In which situations is it used and why? Supplemented by a configuration guide for the FortiGates, a downloadable PCAP and Wireshark screenshots.

NAT46 Concept

Basically, if you have IPv6-only servers (to avoid the unnecessary dual-stack burden for at least some of your infrastructure), but still want to have those servers accessible for IPv4-only clients, you have to use some kind of protocol translation somewhere. Either through reverse proxies or load balancers or through a NAT46 gateway like this:

Note that the NAT46 proxy, pronounced NAT-four-six by the way, does not necessarily have to be in the direct path between the client and server – it only has to be accessible by appropriate routes. Nevertheless, having one central firewall in place which does the job fits perfectly for small installations.

Furthermore, note that you need a hostname in the public DNS with an A record for the IPv4 address on your NAT46 gateway. But you don’t need a special DNS device such as a DNS64 box for NAT64 to work.

The basic concept of translating IP and ICMP between IPv4 and IPv6 aka “Stateless IP/ICMP Translation Algorithm (SIIT)” is described in RFC 7915. Funnily enough, the keyword “NAT46” is not said in the document at all.

Since I have at least one “server” (it’s a Raspi) running IPv6-only, I was able to test this NAT46 gateway. My True Random PSK Generator at https://random.weberlab.de/ has only an AAAA record, while I used https://random46.weberlab.de/ to access it via legacy IP:

NAT46 on a FortiGate

I’m not a big fan of FortiGate firewalls because they are neither reliable nor sound in many situations. However, they offer many cool and new features that other vendors don’t have. So let’s configure NAT46. I’m using a FortiWiFi-61E with FortiOS v7.0.9 for this setup.

At first, it requires a NAT object aka “Virtual IP” of type IPv4 which maps the (public) IPv4 address to the (internal) IPv6 address:

Second, you need an IPv6 Pool for the IPv6 source from which the firewall will initiate the internal IPv6 connections. Note the NAT46 checkbox. Also note that the “pool” is not really a pool but only capable of one or two IPv6 addresses. It took me a while to figure it out. All ranges I tested weren’t valid until I reduced the pool to one single IPv6 address. As you can see, I chose a very special-looking one: an IPv6 address with 4646 at the very end to spot it easily:

Finally, you need an appropriate Firewall Policy. Note that you must select the “NAT46” feature before you can select the destination which is the virtual IP object:

I’m a little scared when looking at this policy in the overview since the destination IPv6 address object says “any” though nothing was selected when creating the policy. AHH. It looks like this policy now allows all IPv6 destinations. Hopefully, it doesn’t. (That’s what I mean when stating that FortiGate firewalls are not that sound.)

Having a look at the Forward Traffic Log you can indeed see both Internet Protocols:

The appropriate CLI commands for this feature are: (You have to adjust some of them according to your needs/setup, e.g., the profile-group, the inspection-mode, and the like.)

config firewall vip
    edit "random46"
        set comment "Test NAT46"
        set extip 194.247.5.23
        set nat44 disable
        set nat46 enable
        set extintf "wan1"
        set ipv6-mappedip 2001:470:1f0b:16b0:6986:b8d4:3649:9cbe
    next
end
config firewall ippool6
    edit "SNAT46"
        set startip 2001:470:1f0b:16b0::4646
        set endip 2001:470:1f0b:16b0::4646
        set nat46 enable
    next
end
config firewall policy
    edit 41
        set name "random46"
        set srcintf "wan1"
        set dstintf "internal"
        set action accept
        set nat46 enable
        set srcaddr "all"
        set dstaddr "random46"
        set srcaddr6 "all"
        set dstaddr6 "all"
        set schedule "always"
        set service "HTTP" "HTTPS" "PING" "PING6"
        set utm-status enable
        set inspection-mode proxy
        set profile-type group
        set profile-group "app-only"
        set logtraffic all
        set ippool enable
        set poolname6 "SNAT46"
    next
end

Having a look at the sessions via CLI, you can see both ones, legacy IP and IPv6. Note the “peer” line for each IP in which the other IP is referenced. Nice! You have to use two different commands to show those sessions, though. (That’s what I mean when stating that FortiGate firewalls are not that sound.)

fg2 # diagnose sys session list

session info: proto=6 proto_state=05 duration=1 expire=0 timeout=3600 flags=00000000 socktype=0 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/255
state=log may_dirty npu f00
statistic(bytes/packets/allow_err): org=1343/11/1 reply=7375/9/1 tuples=2
tx speed(Bps/kbps): 1316/10 rx speed(Bps/kbps): 7230/57
orgin->sink: org pre->post, reply pre->post dev=6->20/20->6 gwy=194.247.5.23/194.247.4.1
hook=pre dir=org act=noop 85.215.94.29:53928->194.247.5.23:443(0.0.0.0:0)
hook=post dir=reply act=noop 194.247.5.23:443->85.215.94.29:53928(0.0.0.0:0)
peer=2001:470:1f0b:16b0::4646:53928->2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443 naf=1
hook=pre dir=org act=noop 2001:470:1f0b:16b0::4646:53928->2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443(:::0)
hook=post dir=reply act=noop 2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443->2001:470:1f0b:16b0::4646:53928(:::0)
pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 policy_id=41 pol_uuid_idx=606 auth_info=0 chk_client_info=0 vd=0
serial=018cd679 tos=ff/ff app_list=0 app=0 url_cat=0
rpdb_link_id=00000000 ngfwid=n/a
npu_state=0x4040400 ofld-O
npu info: flag=0x00/0x00, offload=0/0, ips_offload=0/0, epid=0/0, ipid=0/0, vlan=0x0000/0x0000
vlifid=0/0, vtag_in=0x0000/0x0000 in_npu=0/0, out_npu=0/0, fwd_en=0/0, qid=0/0
no_ofld_reason:
ofld_fail_reason(kernel, drv): none/not-established, none(0)/none(0)
npu_state_err=00/04
total session 1


fg2 # diagnose sys session6 list

session6 info: proto=6 proto_state=15 duration=1 expire=0 timeout=3600 flags=00000000 sockport=0 socktype=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/0
state=log may_dirty npu app_valid
statistic(bytes/packets/allow_err): org=1563/11/0 reply=7802/12/0 tuples=2
tx speed(Bps/kbps): 893/7 rx speed(Bps/kbps): 4458/35
orgin->sink: org pre->post, reply pre->post dev=20->23/23->20
hook=pre dir=org act=noop 2001:470:1f0b:16b0::4646:53928->2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443(:::0)
hook=post dir=reply act=noop 2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443->2001:470:1f0b:16b0::4646:53928(:::0)
peer=194.247.5.23:443->85.215.94.29:53928 naf=2
dst_mac=b8:27:eb:03:a0:ac
misc=0 policy_id=41 auth_info=0 chk_client_info=0 vd=0
serial=00312c35 tos=ff/ff ips_view=0 app_list=2000 app=40568 url_cat=0
rpdb_link_id = 00000000 ngfwid=n/a
npu_state=0x4041808 ofld-R
npu info: flag=0x00/0x81, offload=0/0, ips_offload=0/0, epid=0/64, ipid=0/76, vlan=0x0000/0x0000
vlifid=0/76, vtag_in=0x0000/0x0000 in_npu=0/1, out_npu=0/1, fwd_en=0/0, qid=0/3
no_ofld_reason:
ofld_fail_reason(kernel, drv): none/not-established, none(0)/none(0)
npu_state_err=00/04

Finally, note that this setup required several other things around the mere network config which I have not shown here. That is:

• a hostname for random46.weberlab.de with only at least an A record
• ServerAliases on the apache2 config for the virtual host
• an adjusted rewrite condition to forward HTTP -> HTTPS for this ServerAlias
• running certbot again to have a valid X.509 certificate with this hostname in the subject alternative name field

Deeper Look on the Wire

I’ve captured some basic runs, that is: doing an HTTP request, getting redirected to HTTPS, as well as a ping aka echo-request. I captured on the client as well as on the server simultaneously and merged them later on. This capture is within the Ultimate PCAP already, but you can download it solely as well:

You can easily filter for ip or ipv6 to see only one of those Internet Protocols. Here they are side-by-side, looking at the SYN for the HTTP session:

As expected, the upper-layer protocol stuff is exactly the same after the NAT46 proxy, such as the TLS handshake with its ECDH client key exchange:

However, looking at ICMPv4 vs. ICMPv6 messages for echo-requests/-replies, the data portion looks a little different. ICMPv4 in this example used a timestamp which should be silently dropped according to RFC 7915, section 4.2. Looks like the FortiGate isn’t doing it that way but keeps the timestamp information within the data portion:

However, it’s working quite good. Nice! If you can spot any other differences between those translated protocols, please write a comment!

Fun Fact: NAT646

The other day I was on a german train using my T-Mobile tethering on my iPhone which gives perfect IPv6-native access, incl. DNS64/NAT64. Now, when surfing to this IPv4-only NAT46 domain, it eventually does a 646 translation. ;)

Of course, that would not have happened if the hostname would have had an AAAA record as well, which would be the case for real-world purposes in which your server hostname has an AAAA record (since it is IPv6-only) *and* the additional A record for the NAT46 translation.

Photo by Joshua Sortino on Unsplash.

At SharkFest’22 EU, the Annual Wireshark User and Developer Conference, I attended a beginners’ course called “Network Troubleshooting from Scratch”, taught by the great Jasper Bongertz. In the end, we had some high-level discussions concerning various things, one of them was the insight that TCP RSTs are not only sent from a server in case the port is closed, but are also commonly sent (aka spoofed) from firewalls in case a security policy denies the connection. Key question: Can you distinguish between those spoofed vs. real TCP RSTs? Initially, I thought: no, you can’t, cause the firewalls out there do a great job.

It turned out: you can!

Closed Port vs. Denying Firewall

I did a little lab for this. In a first step, I connected to one of my servers on port 21 – old-fashioned FTP. Since there was no FTP service on the server running, every connection attempt was immediately responded by a TCP RST – just as a normal Linux server behaves. The following screenshot shows five connection attempts (FileZilla) for IPv6 and five more for legacy IP, all with a RST answer from the real server:

In a second step, I configured my Palo Alto Networks firewall to block outgoing FTP connections and to respond with a RST to the client. This is common in internal environments, where you want your clients to have some sort of immediate feedback in case of blocked/closed connections. A silent drop from the firewall would result in a client application waiting for a timeout.

This screenshot shows the TCP RST that was perfectly spoofed and sent by the firewall:

If you swap between those two screenshots, you won’t see any obvious differences, don’t you? In fact, the source IP addresses (regardless of IPv6 or legacy IP) are spoofed by the firewall as if they were the real server. Good job.

How to distinguish between them???

We just proved that the packets look exactly the same, right? This is where Jasper gave a little hint at his session:

A picture is worth a thousand words:

Let’s have a look at some pings via IPv6 and legacy IP, especially on the hop limit / TTL of the returning packets which reveal the hop count of the returning route. For this, I added a custom column to Wireshark which displays the ipv6.hlim or ip.ttl:

That is: the returning IPv6 route is probably 10 hops away (since the remaining hop limit is 54, assuming the starting hop limit was 64), and the returning legacy IP route is probably 11 hops long.

Now let’s look at the RSTs again, focussing on the hop limit / TTL. This screenshot shows the 2x RSTs from the real server and 2x RSTs from the firewall:

Ha, I got you!

Q.E.D.

If you want to have a look at the captures by yourself, here we go: (They are in the Ultimate PCAP as well.)

Excursion: Palo Alto NGFW Policy Actions

I used a PA-220 with PAN-OS 10.2.1 for these tests. It was a little tricky to set the policies right for this. At first, blocking the application “ftp” did not work since the application detection of PAN will almost always let the first packets pass until it really detects the application. This is normal behaviour if you have other policies (further down) that are merely application based. My “FTP app deny” rule did not hit because the first SYN packet will flow through anyway, while the server is not responding (aka sending a RST), hence the PAN will log those connection attempts as “incomplete”:

That is: I used a port-based policy with TCP destination port 21 to block those connection attempts instantly. Now it was about selecting the correct type of “block” to have the firewall send TCP RSTs. The commonly used “Deny” did indeed block the connections, but did not respond with RSTs. (You will find those attempts in the pcap as well.) A “Drop“, even with an enabled “Send ICMP Unreachable” did not send back any RSTs as well, though the documentation states this. Finally, the “Reset client” did the trick. I also selected the “Send ICMP Unreachable” here, but I think it would have worked without it.

Here are the traffic logs during my tests:

1. allowing FTP but the server sent a TCP RST
2. action “deny”, that is: silently drop in this case
3. action “drop” with sending ICMP unreach: silently dropping as well
4. action “reset client” with sending ICMP unreach again. For whatever reason Palo does not highlight this policy action as red, which is a GUI bug in my point of view.

Appendix: What about UDP? –> ICMP

When it comes to UDP connections, there is no concept like a RST from the server within the same UDP session. (Remember: UDP is stateless.) Instead, the server is sending an ICMP port unreachable messages back to the client. Arrows 1 and 2 on the following screenshot. Note that the custom column in Wireshark displays two hop limits since the received ICMP packets (with their hop limits) embed the original packets that forced the error (with their own hop limits as seen by the server).

Talking about the Palo Alto Networks NGFW, it does *not* spoof the ICMP port unreachables, but sends ICMP administratively prohibited messages, clearly telling the client that there is a firewall in between. Those packets are sourced from the actual firewall IP addresses, arrows 3 and 4. Consistently, those ICMP packets have the hop limit / TTL from the egress interface, arrows 5 and 6:

BUT a BUG: Did you spot it? The Palo has an obvious bug, as it sends the ICMPv6 packets from the unspecified IPv6 of “::”. This definitely violates the standards. Nobody is perfect. ;) I opened a ticket and reported it, though I’m not expecting that they’ll fix it.

Photo by Kelly Sikkema on Unsplash.

Again two more commonly used network protocols for the Ultimate PCAP: the Remote Authentication Dial-In User Service (RADIUS) and the Terminal Access Controller Access-Control System Plus (TACACS+) protocols. Captured with quite some details:

You can either download the Ultimate PCAP (recommended ;)) or merely these PCAPs:

RADIUS

Quoting Wikipedia: “Remote Authentication Dial-In User Service (RADIUS) is a networking protocol that provides centralized authentication, authorization, and accounting (AAA) management for users who connect and use a network service. […] The RADIUS server checks that the information is correct using authentication schemes such as PAP, CHAP or EAP.”

For these tests, I installed FreeRADIUS version 3.0.20 on a Ubuntu 20.04.5 LTS (5.4.0-135-generic x86_64).

At first, I used the “radtest” tool for some very basic query-responses aka request-accept messages. IPv6 and legacy IP, each time with: PAP, CHAP, and MS-CHAP:

weberjoh@vm30-test1:~$ radtest -t pap -6 bob ThisIsThePassword test2-v6.weberlab.de 10 iNJ72r0uPXP5qhAX Sent Access-Request Id 238 from [::]:51580 to [2001:470:1f0b:16b0:20c:29ff:fea8:26f7]:1812 length 101
        User-Name = "bob"
        User-Password = "ThisIsThePassword"
        NAS-IPv6-Address = ::1
        NAS-Port = 10
        Message-Authenticator = 0x00
        Cleartext-Password = "ThisIsThePassword"
Received Access-Accept Id 238 from [2001:470:1f0b:16b0:20c:29ff:fea8:26f7]:1812 to [::]:0 length 32
        Reply-Message = "Hello, bob"
weberjoh@vm30-test1:~$ radtest -t chap -6 bob ThisIsThePassword test2-v6.weberlab.de 10 iNJ72r0uPXP5qhAX
Sent Access-Request Id 23 from [::]:46192 to [2001:470:1f0b:16b0:20c:29ff:fea8:26f7]:1812 length 86
        User-Name = "bob"
        CHAP-Password = 0xea8d36b4906c71784c75e17983e36cab66
        NAS-IPv6-Address = ::1
        NAS-Port = 10
        Message-Authenticator = 0x00
        Cleartext-Password = "ThisIsThePassword"
Received Access-Accept Id 23 from [2001:470:1f0b:16b0:20c:29ff:fea8:26f7]:1812 to [::]:0 length 32
        Reply-Message = "Hello, bob"
weberjoh@vm30-test1:~$ radtest -t mschap -6 bob ThisIsThePassword test2-v6.weberlab.de 10 iNJ72r0uPXP5qhAX
Sent Access-Request Id 22 from [::]:60664 to [2001:470:1f0b:16b0:20c:29ff:fea8:26f7]:1812 length 141
        User-Name = "bob"
        MS-CHAP-Password = "ThisIsThePassword"
        NAS-IPv6-Address = ::1
        NAS-Port = 10
        Message-Authenticator = 0x00
        Cleartext-Password = "ThisIsThePassword"
        MS-CHAP-Challenge = 0x9cfe6e183debe56d
        MS-CHAP-Response = 0x0001000000000000000000000000000000000000000000000000517e32e81fc99c2d05fc072ce33afe6eb5a01a782532c359
Received Access-Accept Id 22 from [2001:470:1f0b:16b0:20c:29ff:fea8:26f7]:1812 to [::]:0 length 96
        Reply-Message = "Hello, bob"
        MS-CHAP-MPPE-Keys = 0x00000000000000001c2e158974b065d412b4456d6ebf7574
        MS-MPPE-Encryption-Policy = Encryption-Allowed
        MS-MPPE-Encryption-Types = RC4-40or128-bit-Allowed
weberjoh@vm30-test1:~$
weberjoh@vm30-test1:~$
weberjoh@vm30-test1:~$ radtest -t pap bob ThisIsThePassword test2-v4.weberlab.de 10 iNJ72r0uPXP5qhAX    Sent Access-Request Id 19 from 0.0.0.0:35337 to 194.247.5.27:1812 length 89
        User-Name = "bob"
        User-Password = "ThisIsThePassword"
        NAS-IP-Address = 127.0.0.1
        NAS-Port = 10
        Message-Authenticator = 0x00
        Cleartext-Password = "ThisIsThePassword"
Received Access-Accept Id 19 from 194.247.5.27:1812 to 0.0.0.0:0 length 32
        Reply-Message = "Hello, bob"
weberjoh@vm30-test1:~$ radtest -t chap bob ThisIsThePassword test2-v4.weberlab.de 10 iNJ72r0uPXP5qhAX
Sent Access-Request Id 52 from 0.0.0.0:41381 to 194.247.5.27:1812 length 74
        User-Name = "bob"
        CHAP-Password = 0xe58f51cf7e8b048d3d8ebbf5837378e277
        NAS-IP-Address = 127.0.0.1
        NAS-Port = 10
        Message-Authenticator = 0x00
        Cleartext-Password = "ThisIsThePassword"
Received Access-Accept Id 52 from 194.247.5.27:1812 to 0.0.0.0:0 length 32
        Reply-Message = "Hello, bob"
weberjoh@vm30-test1:~$ radtest -t mschap bob ThisIsThePassword test2-v4.weberlab.de 10 iNJ72r0uPXP5qhAX
Sent Access-Request Id 102 from 0.0.0.0:43642 to 194.247.5.27:1812 length 129
        User-Name = "bob"
        MS-CHAP-Password = "ThisIsThePassword"
        NAS-IP-Address = 127.0.0.1
        NAS-Port = 10
        Message-Authenticator = 0x00
        Cleartext-Password = "ThisIsThePassword"
        MS-CHAP-Challenge = 0x336bbd41fe6c4c2d
        MS-CHAP-Response = 0x00010000000000000000000000000000000000000000000000004ec1a471e0992133be0bed6a714beffd62802495e0e34a5f
Received Access-Accept Id 102 from 194.247.5.27:1812 to 0.0.0.0:0 length 96
        Reply-Message = "Hello, bob"
        MS-CHAP-MPPE-Keys = 0x00000000000000001c2e158974b065d412b4456d6ebf7574
        MS-MPPE-Encryption-Policy = Encryption-Allowed
        MS-MPPE-Encryption-Types = RC4-40or128-bit-Allowed

Secondly, I did some “Test User Credentials” on a Fortinet FortiWiFi FWF-61E with FortiOS 7.0.9 within the RADIUS Servers profile. I did the following methods in that order: PAP, CHAP, MS-CHAP, and MS-CHAP-v2. The FortiGate is not capable of any secure authentication schemes nor communicating via IPv6. (Why is this called a next-gen firewall?)

Thirdly I did some authentication tests on a Palo Alto Networks PA-220 with PAN-OS 10.2.3. I wanted to test some advanced authentication variants of RADIUS that are secured by TLS. Therefore, I ran the “/etc/freeradius/3.0/certs/bootstrap” script on the FreeRADIUS server to get some certificates in place, edited some config files (I don’t remember which one exactly, but in the end, it worked – hahaha), exported the snakeoil root CA, imported it into the Palo Alto NGFW, marked it as “Trusted Root CA”, created an appropriate Certificate Profile, and selected this profile within the RADIUS server profile. (This would be an own blog post just about using secure RADIUS with the PAN. ;)) In the end, I was able to test the following auth protocols: PEAP-MSCHAPv2, PEAP with GTC, and EAP-TTLS with PAP. I did this by selecting the respective method, followed by the useful “test authentication […]” CLI commands. No commit needed to test all those stuff, though freshly configured within the GUI. Great.

This was the output during the tests on the PAN CLI:

weberjoh@pa> test authentication authentication-profile test2 username bob password
Enter password :

Target vsys is not specified, user "bob" is assumed to be configured with a shared auth profile.

Do allow list check before sending out authentication request...
name "bob" is in group "all"

Egress: No service source route is set, might use destination source route if configured
Test authentication to RADIUS server 2001:470:1f0b:16b0:20c:29ff:fea8:26f7:1812 for user: "bob" (with anonymous outer id) using protocol: PEAP with MSCHAPv2
Successful EAPOL auth.
Authentication succeeded against RADIUS server at 2001:470:1f0b:16b0:20c:29ff:fea8:26f7:1812 for user "bob"


Authentication succeeded for user "bob"

weberjoh@pa>
weberjoh@pa>
weberjoh@pa> test authentication authentication-profile test2 username bob password
Enter password :

Target vsys is not specified, user "bob" is assumed to be configured with a shared auth profile.

Do allow list check before sending out authentication request...
name "bob" is in group "all"

Egress: No service source route is set, might use destination source route if configured
Test authentication to RADIUS server 2001:470:1f0b:16b0:20c:29ff:fea8:26f7:1812 for user: "bob" (with anonymous outer id) using protocol: PEAP with GTC
Successful EAPOL auth.
Authentication succeeded against RADIUS server at 2001:470:1f0b:16b0:20c:29ff:fea8:26f7:1812 for user "bob"


Authentication succeeded for user "bob"

weberjoh@pa>
weberjoh@pa>
weberjoh@pa> test authentication authentication-profile test2 username bob password
Enter password :

Target vsys is not specified, user "bob" is assumed to be configured with a shared auth profile.

Do allow list check before sending out authentication request...
name "bob" is in group "all"

Egress: No service source route is set, might use destination source route if configured
Test authentication to RADIUS server 2001:470:1f0b:16b0:20c:29ff:fea8:26f7:1812 for user: "bob" (with anonymous outer id) using protocol: EAP-TTLS with PAP
Successful EAPOL auth.
Authentication succeeded against RADIUS server at 2001:470:1f0b:16b0:20c:29ff:fea8:26f7:1812 for user "bob"


Authentication succeeded for user "bob"

weberjoh@pa>

And since I ran the FreeRADIUS server in debug mode, I’ll hand out those debug logs as well, just in case you’re interested. 3320 lines for those 3x auth tests. Wow. ;) Click here to download it.

Wiresharking

For all these tests I used the same RADIUS shared secret of iNJ72r0uPXP5qhAX. Paste it into the Edit -> Preferences -> Protocols -> RADIUS section to have Wireshark decrypt some stuff:

And now, some Wireshark screenshots, while I strongly encourage you to download the Ultimate PCAP and click around it by yourself. Use the display filter of radius.

I’ve only used some basic AVPs here since I did not use RADIUS in production with several different vendors and stuff. However, you get the idea. And there are already enough fields to dig into. ;) Furthermore, I only used RADIUS with UDP (not sure whether TCP is used at all for RADIUS?) and only for authentication on port 1812, not accounting on port 1813. I also missed mistyping the user password to have a reject. Yeah, that’s the way it is.

TACACS+

To simply state the TACACS article on Wikipedia again: “TACACS+ is a Cisco designed extension to TACACS that encrypts the full content of each packet. Moreover, it provides granular control in the form of command-by-command authorization. […] TACACS+ encrypts all the information mentioned above and therefore does not have the vulnerabilities present in the RADIUS protocol.”

TACACS+ uses TCP as transport and has its well-known port of 49. For my lab, I used an Aruba ClearPass Policy Manager version 6.9.10.134806 as the server and a Cisco ASR1001-X with IOS XE Version 17.03.04a as the client aka NAS. (I apologise for being IPv4-only this time…)

Fortunately, Wireshark is able to decrypt all TACACS+ messages in case the shared secret is provided, which is true for my lab: John3.16. Edit -> Preferences -> Protocols -> TACACS+:

As always, if you want to see the whole TCP sessions aka streams (incl. the three-way handshakes), you have to use a display filter like tcp.port eq 49. The display filter for only the payload of TACACS+ (with the plus) is tacplus. (For the predecessor, which is TACACS without the plus, it is tacacs. But this is not used here.) I did the following steps during the capturing:

• 2x login with a wrong password -> authentication failed
• correct login -> authentication passed
• some CLI commands on the router to have a –> authorization
• and out of the box for TACACS+ –> accounting

That’s it for now. Merry Christmas! 🎄

Photo by CardMapr.nl on Unsplash.

For administrative purposes, SSH is used quite often. Almost everyone in IT knows it. Keywords: OpenSSH, simply using “ssh <hostname>” on your machine, PuTTY for Windows, username + password or public key authentication, TCP port 22, simple firewall rules, ignoring the fingerprints 🤦‍♂️, SCP and SFTP. That’s it – basically.

However, it gets much more complicated if you look into the details. You have to deal with many different types and representations of fingerprints, as well as crypto algorithms. Troubleshooting specific connection problems is challenging.

To get an overview of your SSH server’s configuration is to scan them with appropriate tools. I’m showing two of them here: ssh_scan and the Nmap script “ssh2-enum-algos“.

I’m using an Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-139-generic x86_64) for the following.

mozilla/ssh_scan

“An SSH configuration and policy scanner“. Unfortunately, the GitHub repository has been marked as deprecated. However, you can still install it:

sudo apt install ruby ruby-dev gcc gem
sudo gem install ssh_scan

Usage is simple: ssh_scan -t <hostname|ip>. The output gives you many insights about the keys (along with their fingerprints), the encryption-, mac-, and key-algorithms, SSHFP dns_keys, and so forth. IPv6 is preferred, as it should be.

This is a test run against an Ubuntu 18.04.6 LTS:

weberjoh@vm32-test2:~$ ssh_scan -t test1.weberlab.de
[
  {
    "ssh_scan_version": "0.0.44",
    "ip": "194.247.5.25",
    "hostname": "test1.weberlab.de",
    "port": 22,
    "server_banner": "SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7",
    "ssh_version": 2.0,
    "os": "ubuntu",
    "os_cpe": "o:canonical:ubuntu",
    "ssh_lib": "openssh",
    "ssh_lib_cpe": "a:openssh:openssh:7.6p1",
    "key_algorithms": [
      "curve25519-sha256",
      "curve25519-sha256@libssh.org",
      "ecdh-sha2-nistp256",
      "ecdh-sha2-nistp384",
      "ecdh-sha2-nistp521",
      "diffie-hellman-group-exchange-sha256",
      "diffie-hellman-group16-sha512",
      "diffie-hellman-group18-sha512",
      "diffie-hellman-group14-sha256",
      "diffie-hellman-group14-sha1"
    ],
    "encryption_algorithms_client_to_server": [
      "chacha20-poly1305@openssh.com",
      "aes128-ctr",
      "aes192-ctr",
      "aes256-ctr",
      "aes128-gcm@openssh.com",
      "aes256-gcm@openssh.com"
    ],
    "encryption_algorithms_server_to_client": [
      "chacha20-poly1305@openssh.com",
      "aes128-ctr",
      "aes192-ctr",
      "aes256-ctr",
      "aes128-gcm@openssh.com",
      "aes256-gcm@openssh.com"
    ],
    "mac_algorithms_client_to_server": [
      "umac-64-etm@openssh.com",
      "umac-128-etm@openssh.com",
      "hmac-sha2-256-etm@openssh.com",
      "hmac-sha2-512-etm@openssh.com",
      "hmac-sha1-etm@openssh.com",
      "umac-64@openssh.com",
      "umac-128@openssh.com",
      "hmac-sha2-256",
      "hmac-sha2-512",
      "hmac-sha1"
    ],
    "mac_algorithms_server_to_client": [
      "umac-64-etm@openssh.com",
      "umac-128-etm@openssh.com",
      "hmac-sha2-256-etm@openssh.com",
      "hmac-sha2-512-etm@openssh.com",
      "hmac-sha1-etm@openssh.com",
      "umac-64@openssh.com",
      "umac-128@openssh.com",
      "hmac-sha2-256",
      "hmac-sha2-512",
      "hmac-sha1"
    ],
    "compression_algorithms_client_to_server": [
      "none",
      "zlib@openssh.com"
    ],
    "compression_algorithms_server_to_client": [
      "none",
      "zlib@openssh.com"
    ],
    "languages_client_to_server": [

    ],
    "languages_server_to_client": [

    ],
    "auth_methods": [
      "publickey",
      "password"
    ],
    "keys": {
      "rsa": {
        "raw": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDhLLYj+v5tChro3/H0EYUj1sePBEh89Z1IDyQpaxsMNmbXxmt3Q/H26jDlCCIoVUrnsdcYSsa8zRnumRDgb1mGUUs18ri7bQhT+NOfwox/c29VQj7tYCF1tw8LMehn2bhAZgVyUJohsXoAe6Aa8N47aQ8ddCu09DSCBMpbZJgKONqVkc/lYZ+yBhxHUVmCrTjfuX2z3ycM8cJ6pGCsBvzZx2aJXJFmvsNiwjaXJz6rhm0UwP/9haKiC3PCvHwNNMMQfM9yUWm0eRcxtitLw/B51m4aoLpEkomomcwPSBbpPzUWrPtNXWStfFajjr2GFO7NY9AGylV25scjb+YIz2x7",
        "length": 2048,
        "fingerprints": {
          "md5": "42:5a:52:28:d3:f1:6d:bf:22:ba:26:26:2b:65:29:24",
          "sha1": "48:62:51:71:25:04:1d:d1:ea:7f:14:7e:ce:91:1c:0a:21:92:3d:cb",
          "sha256": "ee:70:d2:6e:e3:a5:db:2c:45:e8:6b:43:01:3d:85:8c:83:2b:95:94:ba:93:23:c2:db:19:3b:d4:09:f4:cf:1a"
        }
      },
      "ecdsa-sha2-nistp256": {
        "raw": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPfD/EKxAETdEr8oP1JEuDNj2MZUPNmqakQrzgsqOSirgbDCVDZrjU5Au1aM4k0KzMFRVjTT4jvs64ea45F5SN8=",
        "length": 520,
        "fingerprints": {
          "md5": "08:cc:6a:15:be:a6:0c:6a:ae:0c:0d:87:f2:cb:a4:90",
          "sha1": "5c:db:59:67:1c:30:1e:74:96:a6:87:8d:fe:6c:3f:d0:6c:9a:4f:f5",
          "sha256": "fa:f3:d7:2b:51:69:7e:62:cd:1b:36:f9:d3:86:5b:dc:52:62:da:9f:3a:db:ef:5f:3b:a8:81:7b:2d:15:e8:00"
        }
      },
      "ed25519": {
        "raw": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICg8/9MH7OjCSDevCKVZY5XBKMInGWb8Jb2Sm7driEI",
        "length": 256,
        "fingerprints": {
          "md5": "97:38:e8:8e:81:83:56:6e:0f:bc:77:06:88:75:37:d3",
          "sha1": "41:09:8f:a4:0e:be:90:f7:38:83:62:ec:40:7e:b2:2c:49:bf:c2:d1",
          "sha256": "76:f2:b1:40:5b:8b:ea:0e:0c:97:6b:20:ce:0a:0b:5d:19:f9:f3:ce:99:bb:24:6d:93:8b:4d:3c:e7:28:ac:44"
        }
      }
    },
    "dns_keys": [
      {
        "fptype": "sha256",
        "algo": "ed25519",
        "hex": "76:f2:b1:40:5b:8b:ea:0e:0c:97:6b:20:ce:0a:0b:5d:19:f9:f3:ce:99:bb:24:6d:93:8b:4d:3c:e7:28:ac:44"
      },
      {
        "fptype": "sha256",
        "algo": "rsa",
        "hex": "ee:70:d2:6e:e3:a5:db:2c:45:e8:6b:43:01:3d:85:8c:83:2b:95:94:ba:93:23:c2:db:19:3b:d4:09:f4:cf:1a"
      },
      {
        "fptype": "sha256",
        "algo": "ecdsa",
        "hex": "fa:f3:d7:2b:51:69:7e:62:cd:1b:36:f9:d3:86:5b:dc:52:62:da:9f:3a:db:ef:5f:3b:a8:81:7b:2d:15:e8:00"
      }
    ],
    "duplicate_host_key_ips": [

    ],
    "compliance": {
      "policy": "Mozilla Modern",
      "compliant": false,
      "recommendations": [
        "Remove these key exchange algorithms: diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256, diffie-hellman-group14-sha1",
        "Remove these MAC algorithms: umac-64-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, hmac-sha1",
        "Remove these authentication methods: password"
      ],
      "references": [
        "https://wiki.mozilla.org/Security/Guidelines/OpenSSH"
      ],
      "grade": "D"
    },
    "start_time": "2023-02-22 15:33:47 +0000",
    "end_time": "2023-02-22 15:33:48 +0000",
    "scan_duration_seconds": 0.806951794
  }
]

And this is against an old Cisco 2811 router with IOS version 12.3(8r)T7:

weberjoh@vm32-test2:~$ ssh_scan -t router1.weberlab.de
[
  {
    "ssh_scan_version": "0.0.44",
    "ip": "2001:470:1f0a:319::2",
    "hostname": "router1.weberlab.de",
    "port": 22,
    "server_banner": "SSH-2.0-Cisco-1.25",
    "ssh_version": 2.0,
    "os": "cisco",
    "os_cpe": "o:cisco:cisco",
    "ssh_lib": "ciscossh",
    "ssh_lib_cpe": "a:cisco:ciscossh",
    "key_algorithms": [
      "diffie-hellman-group-exchange-sha1",
      "diffie-hellman-group14-sha1",
      "diffie-hellman-group1-sha1"
    ],
    "encryption_algorithms_client_to_server": [
      "aes128-cbc",
      "3des-cbc",
      "aes192-cbc",
      "aes256-cbc"
    ],
    "encryption_algorithms_server_to_client": [
      "aes128-cbc",
      "3des-cbc",
      "aes192-cbc",
      "aes256-cbc"
    ],
    "mac_algorithms_client_to_server": [
      "hmac-sha1",
      "hmac-sha1-96",
      "hmac-md5",
      "hmac-md5-96"
    ],
    "mac_algorithms_server_to_client": [
      "hmac-sha1",
      "hmac-sha1-96",
      "hmac-md5",
      "hmac-md5-96"
    ],
    "compression_algorithms_client_to_server": [
      "none"
    ],
    "compression_algorithms_server_to_client": [
      "none"
    ],
    "languages_client_to_server": [

    ],
    "languages_server_to_client": [

    ],
    "auth_methods": [

    ],
    "keys": {
    },
    "dns_keys": [
      {
        "fptype": "sha256",
        "algo": "rsa",
        "hex": "cc:88:fd:4a:a1:cb:8e:40:8f:13:01:ee:a5:16:1e:77:51:54:42:0a:d1:56:bf:39:0b:b3:13:6b:1e:2f:bf:cb"
      }
    ],
    "duplicate_host_key_ips": [

    ],
    "compliance": {
      "policy": "Mozilla Modern",
      "compliant": false,
      "recommendations": [
        "Add these key exchange algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256",
        "Add these MAC algorithms: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com",
        "Add these encryption ciphers: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr",
        "Remove these key exchange algorithms: diffie-hellman-group-exchange-sha1, diffie-hellman-group14-sha1, diffie-hellman-group1-sha1",
        "Remove these MAC algorithms: hmac-sha1, hmac-sha1-96, hmac-md5, hmac-md5-96",
        "Remove these encryption ciphers: aes128-cbc, 3des-cbc, aes192-cbc, aes256-cbc"
      ],
      "references": [
        "https://wiki.mozilla.org/Security/Guidelines/OpenSSH"
      ],
      "grade": "F"
    },
    "start_time": "2023-02-22 15:42:52 +0000",
    "end_time": "2023-02-22 15:42:52 +0000",
    "scan_duration_seconds": 0.499461599,
    "error": "could not settle on encryption_client algorithm\nServer encryption_client preferences: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc\nClient encryption_client preferences: aes256-ctr,aes192-ctr,aes128-ctr"
  }
]

Note the recommendations and the grade.

Nmap Script ssh2-enum-algos

Another way is to use Nmap along with an NSE (Nmap Scripting Engine) script: ssh2-enum-algos. The call is simple as well: nmap --script ssh2-enum-algos <hostname|ip>. The output shows the algorithms only, as the name of the script suggests. As always with Nmap: If you want to scan via IPv6, you have to specify it with “-6” explicitly.

Again, this is a test run against the Ubuntu 18.04.6 LTS:

weberjoh@vm32-test2:~$ nmap --script ssh2-enum-algos test1.weberlab.de
Starting Nmap 7.93 ( https://nmap.org ) at 2023-02-23 09:36 UTC
Nmap scan report for test1.weberlab.de (194.247.5.25)
Host is up (0.00040s latency).
Not shown: 999 closed tcp ports (conn-refused)
PORT   STATE SERVICE
22/tcp open  ssh
| ssh2-enum-algos:
|   kex_algorithms: (10)
|       curve25519-sha256
|       curve25519-sha256@libssh.org
|       ecdh-sha2-nistp256
|       ecdh-sha2-nistp384
|       ecdh-sha2-nistp521
|       diffie-hellman-group-exchange-sha256
|       diffie-hellman-group16-sha512
|       diffie-hellman-group18-sha512
|       diffie-hellman-group14-sha256
|       diffie-hellman-group14-sha1
|   server_host_key_algorithms: (5)
|       ssh-rsa
|       rsa-sha2-512
|       rsa-sha2-256
|       ecdsa-sha2-nistp256
|       ssh-ed25519
|   encryption_algorithms: (6)
|       chacha20-poly1305@openssh.com
|       aes128-ctr
|       aes192-ctr
|       aes256-ctr
|       aes128-gcm@openssh.com
|       aes256-gcm@openssh.com
|   mac_algorithms: (10)
|       umac-64-etm@openssh.com
|       umac-128-etm@openssh.com
|       hmac-sha2-256-etm@openssh.com
|       hmac-sha2-512-etm@openssh.com
|       hmac-sha1-etm@openssh.com
|       umac-64@openssh.com
|       umac-128@openssh.com
|       hmac-sha2-256
|       hmac-sha2-512
|       hmac-sha1
|   compression_algorithms: (2)
|       none
|_      zlib@openssh.com

Nmap done: 1 IP address (1 host up) scanned in 0.62 seconds

And this is against the old Cisco 2811 router with IOS version 12.3(8r)T7:

weberjoh@vm32-test2:~$ nmap -6 --script ssh2-enum-algos router1.weberlab.de
Starting Nmap 7.93 ( https://nmap.org ) at 2023-02-23 09:38 UTC
Nmap scan report for router1.weberlab.de (2001:470:1f0a:319::2)
Host is up (0.028s latency).
Other addresses for router1.weberlab.de (not scanned): 37.24.166.89
rDNS record for 2001:470:1f0a:319::2: tunnel643592-pt.tunnel.tserv6.fra1.ipv6.he.net
Not shown: 997 closed tcp ports (conn-refused)
PORT     STATE SERVICE
22/tcp   open  ssh
| ssh2-enum-algos:
|   kex_algorithms: (3)
|       diffie-hellman-group-exchange-sha1
|       diffie-hellman-group14-sha1
|       diffie-hellman-group1-sha1
|   server_host_key_algorithms: (1)
|       ssh-rsa
|   encryption_algorithms: (4)
|       aes128-cbc
|       3des-cbc
|       aes192-cbc
|       aes256-cbc
|   mac_algorithms: (4)
|       hmac-sha1
|       hmac-sha1-96
|       hmac-md5
|       hmac-md5-96
|   compression_algorithms: (1)
|_      none
5060/tcp open  sip
8008/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 13.29 seconds

Conclusion

It’s just the very first step to merely look at your SSH servers. If you have to troubleshoot connection errors, you have to capture and analyse them in-depth. But if you want to know which protocols, keys, and so forth are possible, those SSH scanners do a great job.

If you want to improve your SSH server configuration, have a look at this guide.

Photo by Snowscat on Unsplash.

For my IPv6 training classes, I was missing a capture of a stateful DHCPv6 address assignment. That is: M-flag within the RA, followed by DHCPv6 messages handing out an IPv6 address among others. Therefore, I set up a DHCPv6 server on an Infoblox grid and furthermore used a Palo Alto NGFW as a DHCPv6 relay to it. I captured on two points: from the client’s point of view (getting to the relay) and from the server’s point of view (unicast messages from the relay). And since I was already there anyway, I additionally captured the same process for DHCPv4. So, here we go:

Basic Setup

A picture is worth a thousand words:

Yep, it’s getting a little complicated with all these addresses. ;)

Capture Details

First, you can download these captures here. I did not join those two capture files on purpose, since it’s much easier to open them side by side this way.

However, it’s in the Ultimate PCAP as well, of course. :)

On the client side, I captured with my ProfiShark. (VLAN 69 tagged since I captured right at the firewall but before the ESXi.) On the server side, I used the built-in “Traffic Capture” feature of the Infoblox Grid. Hence a “Linux cooked capture” encapsulation rather than a classical Ethernet frame. Simply ignore it. More details concerning legacy IP DHCPv4 sequences can be found here.

Here are some Wireshark screenshots. On the left-hand side the client’s perspective and on the right-hand side the server’s perspective. Please note the packet comments that I added to all those messages (and displayed them as a column as well). Likewise, note the differences in the Info column. First, IPv6:

Second, legacy IP:

One major difference between DHCP and DHCPv6: While the default gateway is sent for legacy IP within the DHCP messages, it is NOT sent for IPv6 via DHCPv6, since the IPv6 host already knows its default gateway via the router advertisement.

Another difference: While DHCP for legacy IP always embeds the client’s MAC address, DHCPv6 does not. It uses the concept of a “DHCP Unique Identifier (DUID)” which can be made out of different options.

My Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-144-generic x86_64) VM did release the IPv4 address upon shutdown, but not the IPv6 address. That’s why you can only see this single v4 release, sent by the client *directly* to the DHCP server, not via the relay. Hence this captured message is exactly the same in both capture files.

Detailed Setup

DHCPv6 Server on the Infoblox

Using NIOS version 9.0.0. DHCPv4 is as always. Let’s have a look at DHCPv6.

At first some Grid DHCP Properties. Note that the “Domain Name” is not working as expected with regards to DHCPv4 (at least with my Ubuntu clients). It is used for DDNS within the Infoblox DDI. To have a DNS search list on your client, you have to use the custom option 24 “dhcp6.domain-search”. While the SNTP servers option is working, I was *not* able to get the NTP option running. This would be option 56, which is not yet implemented in NIOS. (Why?!? Some discussion here.) Finally, note that you do NOT have a “default router” option with DHCPv6 at all, since the default router always propagates itself using router advertisements on the local link!

I added one IPv6 DHCP Range within a /64 network:

Don’t forget to activate IPv6 on the appropriate members. (So did I… ;))

After some IPv6 addresses were handed out by the (stateful) DHCPv6 server, you can find them on the Active Leases page. Note the DUID for DHCPv6 rather than the mere MAC address for DHCPv4:

Relaying on the Palo

I’m using a PA-220 with PAN-OS 10.2.3 here in my lab. Quite simple: Network -> DHCP -> DHCP Relay -> adding the interface, activating the Internet Procols you want to use (in my case: both) and add the actual DHCP(v6) server addresses:

And don’t forget an appropriate policy allowing DHCP(v6) from this originating zone to the destination zone with the actual servers, such as:

Some, but not all (!) DHCP and DHCPv6 sessions are seen by the traffic log. The multicast (v6) respectively broadcast (v4) messages are not logged, but for v6 the unicast (relay) and link-local packets, while for v4 the unicast relay and the final release are:

That’s it. Happy troubleshooting. :D

Photo by Arno Senoner on Unsplash.

More and more security-related devices are capable of sending syslog messages via an encrypted TLS channel. So does the well-known Meinberg LANTIME NTP server with its “LTOS” operating system. (And all the other LTOS-based NTP servers like the IMS or SyncFire series.) Here’s how you can configure it:

If you don’t have a TLS-capable syslog server: here’s my tutorial with syslog-ng. I’m using a LANTIME M200 with Firmware-Build 7.06.013 for the following tests. The syslog-ng server version 3.25.1-3 runs on a Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-148-generic x86_64).

Basic Configuration

The configuration is quite simple: Head to Notification -> External Syslog Server and select the “Transport Protocol” as TLS rather than UDP or TCP. Please note that the port changes to the TLS default of 6514 and that the “TLS Peer Verify” is checked by default as well:

You will get into trouble if you don’t have imported the (root) certificate of your TLS-capable syslog server yet. ;) If so, nothing will be logged but you will see errors like this when capturing the traffic:

Please note that this is a good thing! This is how security should always work: with untrusted connections, no data should be transferred.

Hence, you should upload the root or self-signed certificate at Security -> Certificates -> CA Certificates like this:

After that, you’ll see successful TLS connections, that is: change cipher spec, followed by randomly looking application data:

Some Notes

One thing I immediately noticed is, that syslog messages are sent over IPv4 though the configured hostname is dual-stacked with both, an A and AAAA record. This is in contrast to any modern OS, on which IPv6 is preferred over legacy IP. (Fun fact: This seems to be a syslog-ng thing because I recognized the same behaviour on a Palo Alto Networks firewall.) At least this is not security related.

Another little caveat I noticed is the following: There is no check whether the certificate’s name or any of the subject alternative names correlate to the hostname which is specified for the syslog server. E.g.: If the hostname is set to “foo.something.com” (which resolves to the actual IP address of the syslog server), while the uploaded certificate has its name and SAN set to “bar.something.com”. The “TLS peer verify” succeeds, though the certificate is not valid for this specified hostname. However, in my mind, this is not a big problem, since the crucial point is to validate the public key, rather than the name. Though I would have expected that the whole certificate, incl. the name, would be checked.

Luckily, Meinberg took care of those issues and dealt with them. Have a look at the next blog post. ;)

Further Reading

• Meinberg manuals for NTP servers

Photo by Max Harlynking on Unsplash.

Meinberg, with the great help of Mr Weber, has implemented “syslog over TLS” in the LTOS version 7.06. The following report describes the general advantages of “syslog over TLS” and the implementation of it in the LTOS.

Why “Syslog over TLS” is a Good Idea

Syslog messages are a very effective means of monitoring the state of an IT system. Events reveal login attempts, error states, configuration changes, and a great deal of other security-critical information. There are also many who seek to have this data stored and analyzed centrally, for example, to provide “System Information and Event Management” systems (SIEM systems) with a common data source. Under ideal conditions, SIEM systems can process, collate, and build upon this information to provide system administrators with very useful reports that allow attacks on systems and networks to be detected early. However, it is important for syslog messages to be secured so that the transmission of syslog messages does not in its own right become a security issue, potentially allowing hackers to exploit the sensitive information in transit. Thankfully, the program “syslog-ng” used by the LANTIME Operating System (LTOS) supports a TLS connection. This TLS connection preserves, in particular, the integrity and confidentiality of the transmitted data. “syslog-ng” has been in use in LTOS for a long time for managing syslog messages, but the LTOS configuration possibilities always lacked the support of “syslog over TLS” in the past.

In theory, only two additional things are required to make use of the TLS functionality of “syslog-ng”. First, you need a valid root CA certificate, and second, the configuration parameter transport (“tls”) must be added to the “syslog-ng” configuration. In practice, as we’ll see below, it turns out to be not quite so simple, given that there are a couple of other things to consider. But let’s start by addressing in general terms how this functionality is implemented in LTOS.

Syslog over TLS in LTOS

Because “syslog-ng” is already installed and the Web Interface already offers the ability to configure external syslog servers, all that’s needed to complete the implementation is to set the new configuration parameters. These are “Transport Protocol = TLS”, the default “syslog over TLS” port 6514, and the switch “peer verify” to ensure that the server certificate’s validity is verified. Figure 1 shows the new configuration form provided from LTOS Version 7.06.001 onwards.

The function for uploading a root CA certificate was already in place before for LDAPS support and is located in the Web Interface under “Security -> Certificates -> CA Certificates”. It allows the certificates of the CAs (including optionally those of the intermediate CAs) that have signed the certificate of the syslog server. Public certificate authorities are already registered in the firmware, and an updated list is provided with each new version of the LTOS firmware.

This means that it is possible to implement the features broadly without any major adjustments or difficulties and that a connection can be established to a syslog server over TLS quickly.

As chance would have it, Mr Weber was already performing comprehensive tests in relation to “syslog over TLS” using products from other vendors at that time. Given this opportunity, we asked if he’d be willing to test a beta version of LTOS for Meinberg, which he said he’d be happy to do! So, many thanks once again for your support!

Implementation Pitfalls

Mr Weber’s test results with different IP protocols and Fully Qualified Domain Names did reveal a number of flaws, however.

Legacy IP is preferred over IPv6

As soon as an FQDN was entered as a server address, “syslog-ng” would use IPv4 rather than IPv6. If the DNS server returned an AAAA record for an IPv6 address and an A record for an IPv4 address, the IPv6 record would not be used. If only an IPv6 address was returned, the connection would not be established at all. A brief study of the “syslog-ng” documentation revealed that “syslog-ng” assumes a default value of IPv4 unless the desired protocol is explicitly declared, which is not really how name resolution is meant to work. Digging a little further revealed that this behaviour had already been recognized as a bug or a very valuable feature, depending on your point of view. The open issue at https://github.com/syslog-ng/syslog-ng/issues/3386 addresses the request to adjust this behaviour.

Therefore, in order to ensure that “syslog-ng” correctly implements name resolution, additional logic needs to be added. Based on a quick analysis of the code, patching “syslog-ng” itself would not have been a useful investment of time. The much simpler alternative was to draft a small script to perform the name resolution and to adapt the “syslog-ng” configuration to these specific circumstances. This enables a name resolution strategy to be formulated. The parameter “IP PROTOCOL” in /etc/mbg/ntf.cfg can be used to select one of the following options:

• “4” (only IPv4 resolution)
• “6” (only IPv6 resolution),
• “4,6” (IPv4 resolution before IPv6 resolution)
• “6,4” (IPv6 resolution before IPv4 resolution)

The following pseudocode describes the decision-making process. The function “check_if_valid_IPvX” tests if there is a syntactically valid IPv4 address. In addition to the process described by the pseudocode, the IP address can also be “pinged” to determine if it is reachable. The function “resolve_FQDN” triggers a DNS query based on the passed name and returns all recorded IP addresses.

# Pseudocode

#IP_CONF = [4],[6],[4,6] or [6,4]
#HOST = input of address field of external syslog servers
#Return values: ((The (FQDN or IP) and the IP-Protocol) or Error)

check_host(IP_CONF, HOST)
    for IPv in IP_CONF:
        if IPv == 4:
            if check_if_valid_IPv4(HOST):
                return HOST, 4
            else
                resIPs = resolve_FQDN(HOST)
                for resIP in resIPs:
                    if check_if_valid_IPv4(resIP):
                        return HOST, 4
        elif IPv == 6:
            if check_if_valid_IPv6(HOST):
                return HOST, 6
            else
                resIPs = resolve_FQDN(HOST)
                for resIP in resIPs:
                    if check_if_valid_IPv6(resIP):
                        return HOST, 6
    return "ERROR no valid IP or FQDN"

An example of a configuration line for an external syslog server with an FQDN reachable via IPv6 in /etc/syslog-ng/syslog-ng.conf would be:

destination syslog_server_1 { network( "syslog.test.example.fqdn" port(6514) ip-protocol(6) time-zone("Etc/GMT-5") transport("tls") tls( ca-file("/etc/ssl/certs/rootca_bundle.pem") peer-verify(yes) ) ); };

Check of Subject Alternative Names

Another problem found during testing was that, despite the “peer-verify” option being activated, the configured syslog server address did not need to match any of the “Subject Alternative Name” (SAN) entries in the certificate if you were using self-signed certificates. While browsers also do not check for a correct “Common Name” (CN) or SAN when a self-signed certificate is in play, the user still at least needs to provide explicit consent to grant such an exception for the certificate. “syslog-ng”, on the other hand, establishes the connection silently without any message output, despite the option “peer-verify(yes)” being explicitly declared. This would be a major security risk, albeit one that is thankfully not a problem with a two-tier PKI, which allows you to have a valid certificate issued by an official PKI to enable it to be recognized as valid by other domains.

Figure 2 shows examples of parameters that would normally be compared whenever the “TLS peer verify” option is enabled. The screenshot shows the “External Syslog Server” configuration form in the LTOS Web Interface at the top and the contents of an X.509 certificate below. The syslog server address entered here must match one of the SAN entries in the syslog server certificate. If there is no SAN entry, the deprecated CN field will be checked instead. A connection can only be established if the syslog server address matches one of the SANs or the CN precisely. Where wildcards are supported, the strings can vary within the scope afforded by the wildcards.

This means that in the case of self-signed certificates, because the server certificate is identical to the certificate on the client, the CN and SAN entries cease to have any real purpose – the ability to move the certificate with the key to a different server provides in its own right the means to get up to all kinds of no good.

For all those who use self-signed certificates despite the above, it’s worth noting that this behaviour deviates significantly from how it should operate according to the documentation (https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide/73\#TOPIC-1829193), whereby “peer-verify = yes” at the very least should compel “syslog-ng” not to trust a certificate if the client does not match a registered SAN or the CN.

Photo by Fabien Bazanegue on Unsplash.

For the last few years, I have been confused about Palo Alto NGFWs’ various options for configuring an IPv6 address on a layer 3 interface. Let’s have a look at some details:

I’m using a PA-220 with PAN-OS 10.2.4-h2.

My Defaults

Normally, I’m enabling IPv6 on the interface (of course), leaving the Interface ID as “EUI-64”, adding a single GUI IPv6 address along with “Send RA” with its default values, while leaving the Duplicate Address Detection (DAD) unchecked (since it’s a firewall and you definitely want its address to be online) but enabling the NDP monitoring. I “Enable Router Advertisement” in general for this interface and include DNS information within the RAs. This looks like follows:

A show interface <interface name> shows the interface IPv6 addresses (link-local based on EUI-64 as well as the just configured global unicast) along with the advertised prefix:

weberjoh@pa> show interface ethernet1/8.21

--------------------------------------------------------------------------------
Name: ethernet1/8.21, ID: 257, 802.1q tag: 21
Operation mode: layer3
Virtual router default
Interface MTU 1500
Interface IP address: 192.168.21.1/24
Interface IPv6 address: fe80::286:9cff:fee7:5517/64
  2a00:6020:5004:2621::1/64
DAD: disabled
NDP Monitoring: enabled
IPv6 Client Mode: disabled
Router Advertisement: enabled
  Advertised IPv6 prefix:
    2a00:6020:5004:2621::1/64
DNS Support: enabled
  DNS Server(s):
    2a00:6020:5004:2600:1e69:7aff:fe0f:cc5e
  DNS Suffix(es):
    weberlab.de
[...]

Use interface ID as host portion

So far, so good. But I was confused about this “Use interface ID as host portion” option, which I enabled on another (sub)interface on the Palo for testing purposes:

For whatever reason, this option is named “Prefix” in the overview, which actually confused me even more:

In the end, this simply ignored the manually configured host portion of the IPv6 address and used the “Interface ID” option from the (sub)interface (which defaults to EUI-64) as the host portion. Hence the name. ;) The advertised prefix (line 16) still uses the manually configured IPv6 address rather than the one with the interface ID option:

weberjoh@pa> show interface ethernet1/8.22

--------------------------------------------------------------------------------
Name: ethernet1/8.22, ID: 259, 802.1q tag: 22
Operation mode: layer3
Virtual router default
Interface MTU 1500
Interface IP address: 192.168.22.1/24
Interface IPv6 address: fe80::286:9cff:fee7:5517/64
  2a00:6020:5004:2622:286:9cff:fee7:5517/64
DAD: disabled
NDP Monitoring: enabled
IPv6 Client Mode: disabled
Router Advertisement: enabled
  Advertised IPv6 prefix:
    2a00:6020:5004:2622::1/64
DNS Support: enabled
  DNS Server(s):
    2a00:6020:5004:2600:1e69:7aff:fe0f:cc5e
  DNS Suffix(es):
    weberlab.de

I have no clue why someone explicitly wants the host portion to be the EUI-64 one. Do you have any ideas, other than cosmetic? Maybe to have the same host portion for the link-local and the global unicast addresses? While the router advertisements are sent from the link-local address anyway. 🤷

With #IPv6, Palo Alto NGFWs have the option to "use interface ID as host portion" which replaces the statically configured one with the EUI-64 one. Any idea what the use case is for this? The RAs are sent from the LL anyway. @ehorley @Enno_Insinuator @PacketJay pic.twitter.com/kZ4L7UdWhC

— Johannes Weber 🎸 (@webernetz) June 1, 2023

Anycast?!?

To be honest, I don’t have any idea what this “Anycast” checkmark is all about at the IPv6 address configuration here. The doc states “Select to include routing through the nearest node.” Eh? Will this add this router address into some routing protocols as an additional host route? Any ideas someone?

Prefix Information

One more thing I noticed: The prefix information options within the ICMPv6 Router Advertisements are sent with the manually configured IPv6 address in both scenarios. That is: It is sent with the actual *address* of the router’s interface rather than the mere /64 prefix. While this is perfectly valid as of RFC 4861 (“An IP address or a prefix of an IP address.”), I personally prefer having the /64 prefix without a filled-in host portion in the RAs. Here are screenshots with the RAs from both interfaces I tested with. Wireshark decodes it as “Prefix”, which is not 100 % correct at this time (though unambiguous to my mind; no change required):

Photo by Thiago Palia on Unsplash.

Als netzwerktechnisches Spielkind beschäftige ich mich nicht nur mit den Netzwerken großer Firmenumgebungen, sondern auch mit meinem eigenen Anschluss daheim. Vor vielen Jahren habe ich dem echten Dual-Stack Anschluss der Deutschen Telekom mal auf die Finger geguckt – heute ist die Variante der Deutschen Glasfaser an der Reihe, welches zwar ein Dual Stack, aber ohne eigene öffentliche IPv4 Adresse ist. Quasi ein halbes DS-Lite. Kernfrage für mich war: Kann ich die Fritzbox (mit ihren mitgelieferten Presets für verschiedene ISPs) durch eine echte Enterprise-Firewall ersetzen, die ja leider nicht unbedingt alle Sprecharten wie PPPoE im Subinterface oder PPP IPv6CP unterstützen.

TL;DR: DHCP, DHCPv6-PD, RA.

Testaufbau

Bei einem Privatkundenanschluss der Deutschen Glasfaser (DG) habe ich einen echten TAP (in Form meines ProfiSharks) zwischen das Glasfaser-Modem und der Fritzbox gepackt und entsprechend alle Pakete originalgetreu mitgeschnitten:

Das Glasfaser-Modem (NT) ist von der DG gestellt und ein Nokia G-010G-Q Gerät. Dort ist auch eine “Lower MAC ID” aufgedruckt, bei der ich aber nicht weiß, an welcher Stelle sie auftaucht. Es ist ja ein Modem auf Layer 1 und wandelt somit nur die Glasfaser auf ein Kupferkabel um. Ethernet (Layer 2) wird auf beiden Seiten gesprochen. (Oder? Zumindest gehe ich davon aus.) Als Router habe ich eine FRITZ!Box 7590 mit FRITZ!OS 7.56. Der Internetanbieter ist auf “Deutsche Glasfaser” gesetzt, die IPv6-Anbindung auf “Native IPv4-Anbindung verwenden”. (Hier könnte ich mal etwas mehr technische Details in der GUI gebrauchen, um zu verstehen, was genau diese Option im Gegensatz zu den anderen macht.)

Als TAP kam ein Profitap ProfiShark 1G zum Einsatz, welcher praktischerweise zwischen dem Modem und dem WAN-Port der Fritzbox eingeschleust werden konnte. Das Modem ging an Port A des TAPs, die Fritzbox an Port B. (Kann man in Wireshark bei Frame sehen, “on interface …_A bzw. …_B.) Modem und Fritzbox waren ausgeschalten. Ich hatte dann zuerst das Modem eingeschaltet und gewartet, bis sowohl “PON” als auch “LAN” grün leuchteten. Danach habe ich die Fritzbox eingeschaltet.

Ich dachte immer, dass man diese Anschlussart bereits als DS-Lite bezeichnet. Technisch korrekt ist ein CGNAT aber nur die eine Hälfte von DS-Lite. Die andere wäre das Tunneln von IPv4 über eine IPv6-only Infrastruktur im Backbone des ISPs. Dies ist bei der DG nicht der Fall, was immerhin die typischen Probleme von DS-Lite Anschlüssen wie die verringerte MTU-Size umgeht. (Danke an Thomas für den Hinweis!)

Analyse

Zunächst mal kamen noch vor der IP-Adressvergabe bereits IPv6 und IPv4 Päckchen vom Modem an. Das waren aber alles vorherige Sessions von vor dem Ausschalten, oder eingehende Verbindungen zu meinen gehosteten Servern Raspis. Deutet auf jeden Fall darauf hin, dass dem Glasfaser-Anschluss ein dedizierter Port am vorgelagerten Switch/Router zugewiesen ist bzw. der ARP-/Neighbour-Cache noch am Leben war.

Legacy IP

Die öffentliche CGNAT IPv4 Adresse bekommt der eigene Router ganz klassisch per einfachem DHCPv4. Kein PPP oder PPPoE oder sonstwas und auch keine Authentifizierung. Einfach nur DHCP. Wie angenehm. Hier zwei Screenshots, welche das Discover der Fritzbox und das Offer der DG zeigen. ARP fand erst nach der DHCP Session statt. Und anscheinend ist der ARP Timeout der Fritzbox 300 Sekunden, weil immer dann (Delta Time) ein neuer ARP Request kam:

IPv6

Zunächst sendet die Fritzbox ihre Duplicate Address Detection (DAD, Punkt 1 im Screenshot) für ihre link-local Adresse, gefolgt von ein paar Router Solicitations (RS, 2). Diese RSs werden an dieser Stelle aber nicht durch ein entsprechendes Router Advertisement (RA) beantwortet! Das ist schon mal die erste interessante Erkenntnis. Für das DHCPv6 wartet die Fritzbox aber zum Glück nicht auf das M-Flag im RA, sondern fängt kurzerhand selbst mit einem SOLICIT (3) an. Der DUID Type der Fritzbox ist vom Typ “link-layer address” (Pfeil 4) und anfragen tut sie neben einer non-temporary address (5) auch einen Prefix ohne Längenangabe (6, DHCPv6-PD, Prefix Delegation):

Im DHCPv6 ADVERTISE sehen wir den Server Identifier der Deutschen Glasfaser vom Typ “link-layer address plus time” und eine MAC-Adresse, die von der OUI auf “VMware, Inc.” hinweist (1). (Fun fact: Die MAC-Adresse an dieser Stelle wird von Wireshark trotz der eingeschalteten “Resolve Physical Addresses” Option nicht aufgelöst. Einen Feature Request habe ich entsprechend gestellt, der keine 2 Tage später bereits implementiert war!) Dann die non-temporary address für das WAN-Interface das Fritzbox selbst (2), die IPv6 Adressen der rekursiven DNS Server (3), und schließlich der ersehnte Präfix mit einer /56 Länge (4). DANKE noch mal an dieser Stelle an die Deutsche Glasfaser, dass sie sich an sinnvolle Vorgaben hält und nicht etwa ein /60, /62 oder gar ein einziges /64 liefert. Nächste Erkenntnis: Die Fritzbox macht keine Duplicate Address Detection für die soeben erhaltene global unicast Adresse, obgleich der Standard (RFC 4862) das explizit vorschreibt. (Mehr bzgl. DAD hier.)

Interessant ist weiterhin, dass zu diesem Zeitpunkt noch nicht die Default Route bekannt ist. Denn:

Eben dieses notwendige RA (1) kam aber erst mit einer Verzögerung von knapp 600 Sekunden (2) = 10 Minuten (!!!) bei der Fritzbox an. Es deutet also darauf hin, dass die RAs seitens der Deutschen Glasfaser hier ausschließlich periodisch, nicht jedoch in Anfrage eines RS, verschickt werden. Die link-local Adresse des DG Routers sieht auch lustig aus: fe80::ff:fe:01:101 (3). Das M-flag ist korrekt gesetzt (4) und unmittelbar danach fragt die Fritzbox per Neighbour Solicitation auch nach der MAC-Adresse eben dieses Routers (5). Dieses NS wiederholt sich dann alle knapp 30 Sekunden. Ob der Neighbour Cache Timeout der Fritzbox auf nur 30 Sekunden steht?

Genau dieses Fehlen der Default Route sorgt leider für ein verzögertes IPv6. In der Tat hat ein Ping über IPv4 wenige Augenblicke nach Einschalten der Fritzbox wieder funktioniert, während IPv6 erst einige Minuten später lief. Entsprechend hatte bis dahin die Fritzbox korrekterweise mit einem “No route” von ihrer WAN-Interface GUA Adresse geantwortet:

From 2a00:6020:1000:40::436 icmp_seq=312 Destination unreachable: No route
From 2a00:6020:1000:40::436 icmp_seq=313 Destination unreachable: No route
From 2a00:6020:1000:40::436 icmp_seq=314 Destination unreachable: No route
From 2a00:6020:1000:40::436 icmp_seq=315 Destination unreachable: No route
From 2a00:6020:1000:40::436 icmp_seq=316 Destination unreachable: No route
From 2a00:6020:1000:40::436 icmp_seq=317 Destination unreachable: No route
From 2a00:6020:1000:40::436 icmp_seq=318 Destination unreachable: No route
From 2a00:6020:1000:40::436 icmp_seq=319 Destination unreachable: No route
64 bytes from redirector.heise.de (2a02:2e0:3fe:1001:302::): icmp_seq=320 ttl=57 time=6.98 ms
64 bytes from redirector.heise.de (2a02:2e0:3fe:1001:302::): icmp_seq=321 ttl=57 time=4.65 ms
64 bytes from redirector.heise.de (2a02:2e0:3fe:1001:302::): icmp_seq=322 ttl=57 time=2.73 ms
64 bytes from redirector.heise.de (2a02:2e0:3fe:1001:302::): icmp_seq=323 ttl=57 time=4.20 ms
64 bytes from redirector.heise.de (2a02:2e0:3fe:1001:302::): icmp_seq=324 ttl=57 time=2.92 ms
64 bytes from redirector.heise.de (2a02:2e0:3fe:1001:302::): icmp_seq=325 ttl=57 time=4.94 ms
64 bytes from redirector.heise.de (2a02:2e0:3fe:1001:302::): icmp_seq=326 ttl=57 time=3.94 ms
64 bytes from redirector.heise.de (2a02:2e0:3fe:1001:302::): icmp_seq=327 ttl=57 time=5.84 ms

Nunja, es gibt schlimmeres. Aber ganz sauber ist es halt auch noch nicht. Und es sorgt natürlich nicht für ein “IPv6 ist genau so gut umgesetzt wie IPv4” bei den Endnutzern. :( Ich werde sowohl die Deutsche Glasfaser als auch AVM mal anschreiben/-twittern.

(Die oben gezeigten Pakete werden ihren Weg auch ins Ultimate PCAP finden.)

Fritzbox durch Firewall ersetzen

Bei meinem damaligen Versuch, an einen Glasfaseranschluss der Telekom eine Enterprise-Firewall ans Rennen zu bekommen, bin ich ziemlich gescheitert. Zu viele Untervarianten der PPP Protokolle konnten die Firewalls damals nicht richtig umsetzen. Hier sollte das jetzt anders aussehen. DHCP für IPv4 ist absoluter Standard, und auch DHCPv6 und DHCPv6-PD beherrschen die gängigen Betriebssysteme von Palo Alto Networks oder Fortinet. Ich bin gespannt. Und es bleibt noch offen, ob VoIP aka SIP mit der Deutschen Glasfaser funktioniert, wenn der SIP-Endpunkt nicht das direkt angeschlossene Gerät ist. Das ließ sich vor Jahren mit einer Juniper ScreenOS Firewall ebenso schlecht aufbauen.

Photo by Andrik Langfield on Unsplash.

During my IPv6 classes, I discovered a (minor) bug at the NGFW from Palo Alto Networks: ICMPv6 error messages, such as “time exceeded” (type 3) as a reply of traceroute, or “destination unreachable” (type 1) as a reply of a drop policy, are not correctly sourced from the IPv6 address of the data interface itself, but from the unspecified address “::”. Here are some details:

Tested environments: PA-220 with PAN-OS 10.2.4-h2 and 10.2.5, as well as PA-440 with PAN-OS 11.0.2-h2.

The Bug

I basically stumbled upon this problem as I played around with some traceroutes:

weberjoh@nb15-lx:~$ traceroute -6 lx2.weberlab.de
traceroute to lx2.weberlab.de (2001:470:1f0b:16b0::a36:22), 30 hops max, 80 byte packets
 1  :: (::)  1.095 ms  0.763 ms  0.755 ms
 2  * * *
 3  * * *
 4  * * *
 5  port-channel12.core2.fra1.he.net (2001:470:0:63d::1)  3.155 ms  3.590 ms *
 6  tserv1.fra1.he.net (2001:470:0:69::2)  46.521 ms  49.557 ms  52.595 ms
 7  tunnel525322-pt.tunnel.tserv6.fra1.ipv6.he.net (2001:470:1f0a:16b0::2)  124.070 ms  122.339 ms  121.901 ms
 8  2001:470:1f0b:16b0::a36:22 (2001:470:1f0b:16b0::a36:22)  80.968 ms  77.495 ms  74.469 ms

In Wireshark, this looks as follows:

Later on, I discovered the same behaviour when a security policy’s action is set to “Drop” along with “Send ICMP Unreachable”, e.g. here:

This looks as follows in Wireshark (different ICMPv6 type, but the same wrong address):

RFC 4443 “Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification” defines how the source address of an originating ICMPv6 packet MUST look like:

The very only case, in which an IPv6 packet is sent from the unspecified address, is the Duplicate Address Detection process and its corresponding Neighbour Solicitation. (RFC 4861 and RFC 4291.)

Two more notes:

1. This only happens with IPv6. For legacy IP, the correct data interface IPv4 address is chosen as the source address.
2. This only appears on VLAN interfaces, but not on “normal” Layer 3 interfaces. <- This is interesting because some years ago I found another IPv6 bug which was related to a VLAN interface as well. To me, this seems like the IPv6 routines for different types of interfaces on PAN-OS are not exactly the same.

On a mere Layer 3 interfaces those ICMPv6 packets are sourced correctly:

PAN Ticket -> Bug

In October 2022, I created a ticket at the PAN support, telling them about this bug. Luckily, I was able to convince the ticket owner that this has to be fixed since it is clearly a violation of the Internet standards. (To be fair, I also told them that I don’t believe that this is a big thing nor security-related.) The bug got the bug ID PAN-214336. After some months, I got an update from the engineering team, saying that this should be fixed with PAN-OS 10.2.5. Unfortunately, this wasn’t the case. Neither is this bug listed in the release notes nor is it actually fixed with PAN-OS 10.2.5, as I tested it again. Furthermore, it is present in the PAN-OS 11.x major versions as well.

Hopefully, it will get fixed someday. At least, you guys can google it now. :)

Soli Deo Gloria.

Photo by Valeriy Borzov on Unsplash.

When implementing NTP servers, it’s always an interesting part to check whether the server is “up and running” and reachable from the clients. While I’ve done many basic NTP checks out of Linux, I lacked a small docu to do this with Windows. It turned out that there’s no need for third-party software because Windows already includes a tool to test NTP connections: w32tm.

So there is this tool called W32Time with many options to not just test NTP servers, but to configure Windows’ time service and so on. In our simple test case, we just want to query a server with these two options:

w32tm /stripchart /computer:HOSTNAME-OR-IP

This displays a chart showing the offset of your local computer’s time (displayed as a | ) to the time delivered through NTP (shown as a * ). Missing replies are marked with a failure:

C:\Users\jweber2>w32tm /stripchart /computer:ntp3.weberlab.de
ntp3.weberlab.de wird verfolgt [[2001:470:1f0b:16b0::dcfb:123]:123].
Es ist 29.09.2023 09:31:07.
09:31:07, d:+00.1196315s o:+01.2824897s  [                           |   *                       ]
09:31:09, d:+00.0666032s o:+01.2738181s  [                           |  *                        ]
09:31:11, d:+00.1376563s o:+01.1849168s  [                           |  *                        ]
09:31:13, d:+00.1089684s o:+01.2882442s  [                           |   *                       ]
09:31:16, d:+00.0238122s o:+01.2431296s  [                           |  *                        ]
09:31:18, d:+00.2107968s o:+01.2974940s  [                           |   *                       ]
09:31:20, d:+00.0223062s o:+01.2419022s  [                           |  *                        ]
09:31:22, d:+00.1731343s o:+01.2643331s  [                           |  *                        ]
09:31:24, d:+00.0819059s o:+01.2757827s  [                           |   *                       ]
09:31:26, d:+00.0359837s o:+01.2362617s  [                           |  *                        ]
09:31:28, d:+00.1817614s o:+01.2444196s  [                           |  *                        ]
09:31:30, Fehler: 0x800705B4
09:31:34, d:+00.0135745s o:+01.2475575s  [                           |  *                        ]
09:31:36, d:+00.0184675s o:+01.2438881s  [                           |  *                        ]
^C

When the offset is really low, only the asterisk is seen:

C:\Users\jweber2>w32tm /stripchart /computer:ntp3.weberlab.de
ntp3.weberlab.de wird verfolgt [[2001:470:1f0b:16b0::dcfb:123]:123].
Es ist 29.09.2023 11:26:47.
11:26:47, d:+00.1652050s o:+00.0843570s  [                           *                           ]
11:26:49, d:+00.1185437s o:+00.0199126s  [                           *                           ]
11:26:52, d:+00.1057113s o:+00.0478628s  [                           *                           ]
11:26:54, d:+00.0935680s o:+00.0391876s  [                           *                           ]
11:26:56, d:+00.1525955s o:+00.0712276s  [                           *                           ]

Interestingly, when looking at the packets with Wireshark, you can see that the NTP version number is set to 1, while normally clients/servers are using version 4: (Packet 523 did not get an answer for whatever reason and was marked by w32tm accordingly.)

One more tool tip though: With w32tm /query /status you can verify which time source your computer is synced to. In my case, this was a local CMOS clock rather than an NTP server. That’s why the offset (in the first chart above) was about 1 second rather than smaller ones in the range of ms.

C:\Users\jweber2>w32tm /query /status
Sprungindikator: 3(nicht synchronisiert)
Stratum: 0 (nicht angegeben)
Präzision: -23 (119.209ns pro Tick)
Stammverzögerung: 0.0000000s
Stammabweichung: 0.0000000s
Referenz-ID: 0x00000000 (nicht angegeben)
Letzte erfolgr. Synchronisierungszeit: nicht angegeben
Quelle: Local CMOS Clock
Abrufintervall: 10 (1024s)

Please note that I’m neither testing the classical NTP authentication nor NTS at this point. It’s only about the reachability of the NTP server.

Soli Deo Gloria.

Photo by Brad Neathery on Unsplash.

Another update of the Ultimate PCAP is available. Again, there are some special new packets in there which I want to point out here. Feel free to download the newest version to examine those new protocols and packets by yourself. Featuring: SNMPv3, WoL, IPMI, HSRP, Zabbix, Pile of Poo, and Packet Comments. ✅

SNMPv3 with AuthPriv

We all know that SNMPv2c (the “security not my problem” protocol) is heavily insecure, hence we should use SNMPv3 with AuthPriv to get authentication and privacy. Here are some packets to look at. A checkmk instance queries a Meinberg LANTIME M200 NTP server. IPv6 and legacy IP:

Now, here is the neat thing about Wireshark: You can add the auth and priv keys to get those packets decrypted. Here they are for those packets:

• Username: checkmk
• SHA256: rWezwNZIbgLN6Fka4iZ8
• AES128: jwmY294SQXt4AD9kPGKD

I also added that information in the packet comments of the very first SNMPv3 packet for IPv6 and legacy IP. This is how you can add them: Edit -> Preferences -> Protocols -> SNMP -> Users Table -> Edit -> Create new entry:

After that, Wireshark shows “Decrypted ScopedPDU” data in readable text within the Packet Details pane as well as at the Packet Bytes pane:

Wake-on-LAN, WoL

Waking up a Raspi via Wake-on-LAN packets. There are two variants to send the magic packet, which is sixteen repetitions of the target computer’s 48-bit MAC address:

1. directly following the Ethernet frame, Ethertype 0x0842 (though not officially registered), using this tool: sudo etherwake -i eno1 B8:27:EB:BC:CD:B4
2. encapsulated in an IP and UDP packet/datagram (UDP destination port 9, the “Discard” protocol), sent to the Ethernet and IP broadcast address, using: wakeonlan B8:27:EB:BC:CD:B4

Both variants are display filtered with simply wol:

Some more details about WoL are on the Wireshark Wiki.

IPMI/RMCP+

“The Intelligent Platform Management Interface (IPMI) is a set of computer interface specifications for an autonomous computer subsystem that provides management and monitoring capabilities independently of the host system’s CPU, firmware (BIOS or UEFI) and operating system”, Wikipedia. Don’t ask me any details about it, please. :D You can monitor and power on/off servers and so on. In my case, I’m using the ipmitool (version 1.8.18) to query an HP ProLiant DL380p Gen8 server (iLO firmware 2.78) like this:

weberjoh@nuc:~$ ipmitool -I lanplus -H esxi3-ilo.weberlab.de -U Testuser -P ThisIsThePassword sensor
UID Light        | 0x0        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
Sys. Health LED  | na         | discrete   | na    | na        | na        | na        | na        | na        | na
01-Inlet Ambient | 24.000     | degrees C  | ok    | na        | na        | na        | na        | 42.000    | 46.000
02-CPU 1         | 40.000     | degrees C  | ok    | na        | na        | na        | na        | 70.000    | na
03-CPU 2         | 59.000     | degrees C  | ok    | na        | na        | na        | na        | 70.000    | na
04-P1 DIMM 1-3   | 36.000     | degrees C  | ok    | na        | na        | na        | na        | 87.000    | na
05-P1 DIMM 4-6   | na         |            | na    | na        | na        | na        | na        | 87.000    | na
06-P1 DIMM 7-9   | 36.000     | degrees C  | ok    | na        | na        | na        | na        | 87.000    | na
07-P1 DIMM 10-12 | 37.000     | degrees C  | ok    | na        | na        | na        | na        | 87.000    | na
08-P2 DIMM 1-3   | 42.000     | degrees C  | ok    | na        | na        | na        | na        | 87.000    | na
09-P2 DIMM 4-6   | na         |            | na    | na        | na        | na        | na        | 87.000    | na
10-P2 DIMM 7-9   | 36.000     | degrees C  | ok    | na        | na        | na        | na        | 87.000    | na
11-P2 DIMM 10-12 | 37.000     | degrees C  | ok    | na        | na        | na        | na        | 87.000    | na
12-HD Max        | 35.000     | degrees C  | ok    | na        | na        | na        | na        | 60.000    | na
13-Chipset       | 55.000     | degrees C  | ok    | na        | na        | na        | na        | 105.000   | na
14-P/S 1         | 39.000     | degrees C  | ok    | na        | na        | na        | na        | na        | na
15-P/S 2         | 37.000     | degrees C  | ok    | na        | na        | na        | na        | na        | na
16-P/S 2 Zone    | 42.000     | degrees C  | ok    | na        | na        | na        | na        | 75.000    | 80.000
17-VR P1         | 43.000     | degrees C  | ok    | na        | na        | na        | na        | 115.000   | 120.000
18-VR P2         | 46.000     | degrees C  | ok    | na        | na        | na        | na        | 115.000   | 120.000
19-VR P1 Mem     | 39.000     | degrees C  | ok    | na        | na        | na        | na        | 115.000   | 120.000
20-VR P1 Mem     | 39.000     | degrees C  | ok    | na        | na        | na        | na        | 115.000   | 120.000
21-VR P2 Mem     | 41.000     | degrees C  | ok    | na        | na        | na        | na        | 115.000   | 120.000
22-VR P2 Mem     | 44.000     | degrees C  | ok    | na        | na        | na        | na        | 115.000   | 120.000
23-VR P1Vtt Zone | 38.000     | degrees C  | ok    | na        | na        | na        | na        | 90.000    | 95.000
24-VR P2Vtt Zone | 43.000     | degrees C  | ok    | na        | na        | na        | na        | 90.000    | 95.000
25-HD Controller | 76.000     | degrees C  | ok    | na        | na        | na        | na        | 100.000   | na
26-iLO Zone      | 40.000     | degrees C  | ok    | na        | na        | na        | na        | 90.000    | 95.000
27-LOM Card      | na         |            | na    | na        | na        | na        | na        | 100.000   | na
28-PCI 1         | na         |            | na    | na        | na        | na        | na        | 100.000   | na
29-PCI 2         | na         |            | na    | na        | na        | na        | na        | 100.000   | na
30-PCI 3         | na         |            | na    | na        | na        | na        | na        | 100.000   | na
31-PCI 4         | na         |            | na    | na        | na        | na        | na        | 100.000   | na
32-PCI 5         | na         |            | na    | na        | na        | na        | na        | 100.000   | na
33-PCI 6         | na         |            | na    | na        | na        | na        | na        | 100.000   | na
34-PCI 1 Zone    | 35.000     | degrees C  | ok    | na        | na        | na        | na        | 65.000    | 70.000
35-PCI 2 Zone    | 36.000     | degrees C  | ok    | na        | na        | na        | na        | 66.000    | 71.000
36-PCI 3 Zone    | 36.000     | degrees C  | ok    | na        | na        | na        | na        | 66.000    | 71.000
37-PCI 4 Zone    | na         |            | na    | na        | na        | na        | na        | 65.000    | 70.000
38-PCI 5 Zone    | na         |            | na    | na        | na        | na        | na        | 65.000    | 70.000
39-PCI 6 Zone    | na         |            | na    | na        | na        | na        | na        | 65.000    | 70.000
40-I/O Board 1   | 42.000     | degrees C  | ok    | na        | na        | na        | na        | 66.000    | 71.000
41-I/O Board 2   | na         |            | na    | na        | na        | na        | na        | 66.000    | 71.000
42-VR P1 Zone    | 33.000     | degrees C  | ok    | na        | na        | na        | na        | 95.000    | 100.000
43-BIOS Zone     | 48.000     | degrees C  | ok    | na        | na        | na        | na        | 90.000    | 95.000
44-System Board  | 39.000     | degrees C  | ok    | na        | na        | na        | na        | 80.000    | 85.000
45-SuperCap Max  | 29.000     | degrees C  | ok    | na        | na        | na        | na        | 65.000    | na
46-Chipset Zone  | 43.000     | degrees C  | ok    | na        | na        | na        | na        | 75.000    | 80.000
47-Battery Zone  | 41.000     | degrees C  | ok    | na        | na        | na        | na        | 75.000    | 80.000
48-I/O Zone      | 42.000     | degrees C  | ok    | na        | na        | na        | na        | 75.000    | 80.000
49-Sys Exhaust   | 39.000     | degrees C  | ok    | na        | na        | na        | na        | 75.000    | 80.000
50-Sys Exhaust   | 39.000     | degrees C  | ok    | na        | na        | na        | na        | 75.000    | 80.000
Fan 1            | 7.840      | percent    | ok    | na        | na        | na        | na        | na        | na
Fan 2            | 7.840      | percent    | ok    | na        | na        | na        | na        | na        | na
Fan 3            | 8.624      | percent    | ok    | na        | na        | na        | na        | na        | na
Fan 4            | 16.464     | percent    | ok    | na        | na        | na        | na        | na        | na
Fan 5            | 27.440     | percent    | ok    | na        | na        | na        | na        | na        | na
Fan 6            | 27.440     | percent    | ok    | na        | na        | na        | na        | na        | na
Power Supply 1   | 185        | Watts      | ok    | na        | na        | na        | na        | na        | na
Power Supply 2   | 0          | Watts      | ok    | na        | na        | na        | na        | na        | na
Power Meter      | 196        | Watts      | ok    | na        | na        | na        | na        | na        | na
Power Supplies   | 0x0        | discrete   | 0x0280| na        | na        | na        | na        | na        | na
Fans             | 0x0        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
Memory           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
C1 P1I Bay 1     | 0x1        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
C1 P1I Bay 2     | 0x1        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
C1 P1I Bay 3     | 0x1        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
C1 P1I Bay 4     | 0x1        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
C1 P2I Bay 5     | 0x1        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
C1 P2I Bay 6     | 0x1        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
C1 P2I Bay 7     | 0x1        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
C1 P2I Bay 8     | 0x1        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
weberjoh@nuc:~$
weberjoh@nuc:~$
weberjoh@nuc:~$ ipmitool -I lanplus -H 192.168.3.83 -U Testuser -P ThisIsThePassword power status
Chassis Power is on
weberjoh@nuc:~$

This is how it looks like in Wireshark. No session setup since it relies on UDP (port 623). The protocol is detected as IPMB and RMCP+ (display filters: ipmb and rmcp), while you can see some more flags in the Info column. It seems like encryption is used – but don’t ask me how.

On a Palo, it is detected as “rmcp”:

For more information about IPMI, have a look at these very detailed Wiki articles at Thomas-Krenn.AG.

HSRP Version 1 aka 0

It’s a common best practice to set standby version 2 whenever you’re using HSRP on a Cisco router. That’s why I only had those version 2 HSRP packets in the Ultimate PCAP. Nevertheless, I let out this version statement just to capture some HSRP version 1 packets, which are specified for IPv4 only. Obvious differences: the destination IPv4 multicast address is 224.0.0.2 (rather than 224.0.0.102) and Wireshark lists its protocol as HSRP rather than HRSPv2. Little confusion though the “version” field within the HSRP packets is set to 0. But this seems to be correct according to RFC 2281. (Captured out of GNS3, by the way.)

Zabbix Traffic

Thanks to Markku Leiniö I added some different Zabbix connections. Please refer to his GitHub repo or his blog posts concerning any details. As of now (Wireshark version 4.0) there is no official Zabbix dissector integrated.

Filtering for Pile of Poo

Just something funny at the end. At the keynote of the Wireshark conference “SharkFest” 2022 in Europe, Gerald Combs, the creator of Wireshark, talked about various new display filter possibilities, e.g., filtering for emojis such as the pile of poo. Challenge accepted. ;) Here is my proof of concept, or, as @noIPv6 has pointed out, it is a poo of concept. Search for  udp contains "?"  within the Ultimate PCAP and you’ll find it. Try it by yourself by doing a DNS lookup of “pile-of-poo.weberdns.de”.

Packet Comments

No protocol, but a Wireshark feature: you can add/read/edit/delete packet comments. Find some in my DHCP captures, for example. Either by using the display filter pkt_comment, by finding one of those packets (green bar within the packet details), or by a custom column:

Soli Deo Gloria.

Photo by Amber Flowers on Unsplash.

Klemmt es im Netzwerk, so helfen Ping und Traceroute, Fehler und Engpässe einzukreisen. Wir erklären die Funktionsweise und helfen Angriffe aufzudecken.

Unter den Befehlen für Netzwerkanalysen gehören Ping und Traceroute zu den am häufigsten verwendeten. Die Ping-Funktion ist schnell beschrieben: Mittels ping <hostname|ip> schicken Sie ICMP-Pakete vom Typ Echo-Request vom Client zu einem Ziel. Das Ziel antwortet mit Echo-Reply-Paketen. Im Konsolenfenster erscheint dann unter anderem die Antwortzeit in Millisekunden (ms). Sie gibt an, wie viel Zeit vom Versenden bis zum Empfang der Quittung verstrichen ist. Und sie gibt die genaue Dauer eines Ende-zu-Ende-Dialogs an, die Round Trip Time, RTT.

Wenn Sie Ping eine Weile laufen lassen (unter Windows mit dem Parameter -t,unter Unix standardmäßig bis zum Abbruch per Strg+C), werden etwaige Stausituationen auf der betreffenden Internet-Strecke beziehungsweise Lastprobleme des angepingten Servers sichtbar. Aber Achtung: Ob eine entfernte Applikation überlastet ist oder nicht antwortet, darüber verrät Ping nichts.

Denn Ping-Pakete beantwortet der entfernte Netzwerk-Stack und die Ping-Laufzeit hängt – sofern nachgelagerte Router nicht verstopft sind – nur davon ab, ob der Stack zur Ping-Beantwortung noch Kapazitäten übrig hat. Client-Anfragen an einen Service beantwortet hingegen die zugehörige Applikation und die Auslastungen der beiden Elemente hängen nicht miteinander zusammen. Deshalb sollte man Fragen nach der Auslastung von Diensten mit Applikations-Pings wie httping, dnsping oder smtpping nachgehen.

Der Traceroute-Befehl nutzt ein beim IPv6-Protokoll “Hop Limit” beziehungsweise bei IPv4 “Time To Live” genanntes Feld in den Headern der IP-Pakete. Beide hat man ursprünglich spezifiziert, um zu vermeiden, dass IP-Pakete endlos in versehentlich konfigurierten Routing-Loops kreisen und solche Loops verstopfen. Die Methode ist einfach: Der Absender gibt seinen IP-Paketen ein Hop Limit (maximale Anzahl von Zwischenstationen) von zum Beispiel 128 mit auf den Weg. Unterwegs dekrementiert jeder Router auf der Strecke den Wert um 1 und gibt dann das IP-Paket weiter. Kommt bei einem Router ein solches Paket mit Hop-Limit 0 an, muss er es verwerfen und dem Absender “Hop Limit Exceeded” melden. So ist gesichert, dass die IP-Pakete zwar weit reisen können, aber nicht endlos im Internet wabern und der Absender erfährt, dass die betreffende Strecke eine Endlosschleife enthält.

Der Befehl Traceroute nutzt diese Funktion, indem er IP-Pakete mit inkrementierendem Hop-Limit ab dem Wert 1 verschickt. So erzwingt Traceroute, dass sich alle Router auf dem Weg zum Ziel der Reihe nach melden; jeder einzelne schickt “Hop Limit Exceeded” zurück. Die Meldungen nutzt Traceroute, um die Laufzeit zu den Routern auf der Strecke zu ermitteln und überhaupt, um die Router zu identifizieren. Windows verschickt mit dem Traceroute-Befehl so wie Ping ICMP-Echo-Request-Pakete, Unix-Implementierungen verwenden UDP-Pakete mit Source-Ports beginnend ab 33434. Um Firewall-Blockaden zu umgehen oder um Policy Based Routings zu erkennen, bieten sich Layer-4-Traceroutes (LFT) an.

Antwortzeiten und Traceroute

Lange Antwortzeiten

Kommen wir zur gängigsten Fehlinterpretation von Traceroute: Schallt mal wieder der Ruf durchs Haus: “Schatz, das Internet is’ mal wieder lahm” und speziell die Verbindung zum Server X, schnappt man sich Traceroute und schaut sich die Antwortzeiten des Servers und der Router auf der Strecke dorthin an. Fallen die Antwortzeiten eines einzelnen Routers höher aus als bei den übrigen, glaubt man, das Problem identifiziert zu haben: Dieser Router beziehungsweise dessen Verbindungen sind überlastet. Oder noch schlimmer: Manche Router antworten gar nicht und erscheinen nur als Sternchen * in der Ausgabe. Dazu Auszüge eines Beispiels:

tracert -4 -d netsec.blog

1 <1 ms <1 ms <1 ms 192.168.110.33
2  1 ms  1 ms <1 ms 192.168.7.1
3  5 ms  7 ms  3 ms 100.124.1.10
4  4 ms  3 ms  3 ms 100.127.1.147
5  5 ms  7 ms  3 ms 185.22.46.177
6  5 ms  3 ms  4 ms 80.81.192.239
7  8 ms  7 ms  7 ms 87.230.115.1
8  7 ms  7 ms  7 ms 87.230.114.4
9 35 ms 37 ms 35 ms 87.230.114.222
10  * * *  Zeitüberschreitung der Anforderung.
11  7 ms  7 ms  7 ms 5.35.226.136

Dabei antwortet Hop Nummer 9 deutlich langsamer als vorherige Router und Hop Nummer 10 schweigt sogar. Hier ist doch eindeutig was kaputt im Internet, nicht wahr?

Leider weit gefehlt. Das erkennt man leicht daran, dass die Antworten des letzten Glieds der Kette konstant nach 7 ms eingehen. Dieser Wert steht für die Gesamtlatenz der untersuchten Strecke.

Wie kommt es dann aber, dass einzelne Router dazwischen weit langsamer antworten? Zunächst mal ist die Annahme falsch, dass die Antwortzeiten der einzelnen Router direkt mit der Verzögerung dieser Pfadabschnitte zusammenhängen. Denn tatsächlich bestehen moderne Hochgeschwindigkeitsrouter aus mindestens einer Data Plane und einer Control Plane. Das Weiterleiten von IP-Paketen, die Primäraufgabe von Routern, übernimmt die Data Plane, eine durch spezielle Hardware wie ASICs optimierte Einheit, die ihren Job sehr effizient und schnell erledigt. Die Data Planes befördern im Internet jegliche IP-Datenpakete, auch ICMP-Anfragen.

Kommt nun ein IP-Paket mit einem Hop-Limit von 1 an, wird es nicht weitergeleitet. Stattdessen setzt die Data-Plane den Wert auf 0 und verwirft das Paket gemäß der Spezifikation. Es ist dann die Control Plane, die “Hop Limit exceeded” an die Quelladresse meldet. Control Planes sind jedoch weit langsamer als der Rest des Routers. Eine hohe Latenz in der Traceroute-Ausgabe lässt also lediglich auf eine Überlastung der ohnehin schwächeren Control-Plane schließen – ist aber kein harter Beleg für einen Engpass im Routing. Und hat der Netzwerk-Admin eines Routers ICMP-Fehlermeldungen abgedreht, taucht dieser Router nur als Stern in der Traceroute-Ausgabe auf. Auch können ICMP-Antworten ausbleiben, wenn der Admin ein Rate-Limit gesetzt hat und der Schwellenwert bei Ihrer Messung gerade überschritten ist. Derart konfigurierte Router geben also keinen Rückschluss auf die Netzauslastung.

Wichtiges Detail: Anders als die übrigen Traceroute-Messwerte liefert die letzte Ausgabezeile dieselbe Aussage wie ein konventioneller Ping an das Zielsystem. Diesen Messwert steuert nämlich der Netzwerk-Stack des Empfängers per Echo-Reply bei und nicht ein Router beziehungsweise dessen Control-Plane.

Wenn aber ab einem Hop sämtliche nachfolgenden Router beispielsweise um 100 Millisekunden langsamer sind als vorherige, dann ist die Wahrscheinlichkeit hoch, dass ab diesem Router tatsächlich eine längere Laufzeit für alle Verbindungen besteht. Das kann man gelegentlich bei IP-Verbindungen beobachten, die per Unterseekabel um die halbe Welt gehen.

Traceroute-Wege

Wenn Sie klären wollen, ob es in Ihrem Firmennetz oder auf einer Internet-Strecke klemmt, so ist auch hier Traceroute die erste Anlaufstelle. Aber Achtung: Einerseits ist in großen Netzwerken und erst recht im globalen Internet asymmetrisches Routing üblich, andererseits gibt Traceroute nur die Messwerte für den Hinweg aus! Über den Rückweg vom Server zum Client, welcher ja rund die Hälfte der per Ping gemessenen Gesamtlaufzeit ausmacht, kann daher mit einer einfachen Traceroute-Messung keine Aussage getroffen werden.

Man kann nämlich nicht ausschließen, dass eine hohe Latenz ausschließlich auf ein lahmes Rückweg-Routing zurückgeht. Um also sowohl den Hin- als auch den Rückweg einer IP-Verbindung zu analysieren, braucht man zwei Traceroutes: Einen vom Client zum Server und einen vom Server zum Client (Applikations-Pings können nicht aufdecken, ob der Hin- oder der Rückweg lahmt). Dafür muss der Admin die Kontrolle über beide Enden haben, was im Heim- oder Firmennetz oft der Fall ist. Bei fremden Servern, die irgendwo im Internet stehen, ist das in der Regel nicht möglich. Zudem verhindert bei IPv4 die verbreitete Network Address Translation, die in Routern zwischen der öffentlichen IP-Adresse und dem privaten Netzwerk vermittelt, die Messung des Rückwegs zurück zum Client.

Dass sich zudem bei ICMP-Analysen Hinweg und Rückweg (für die Meldungen) ebenfalls unterscheiden können, sei nur ergänzend gesagt; sie können somit ebenfalls irreführende Messwerte liefern. Am Fazit ändert das wenig: Ein Traceroute liefert nur Hinweise, aber keine Beweise.

Und noch ein Eintrag für das kleine Know-how-Heftchen: Router senden die Hop-Limit-Exceeded-Meldungen immer von dem Interface aus, an dem das ursprüngliche Paket eingetroffen ist. Deshalb erscheinen in den Traceroutes je nach Verkehrsrichtung – stromauf- oder stromabwärts – verschiedene Absender-IP-Adressen. Das erschwert die Identifikation einzelner Router immens. Deshalb gebührt jenen Providern Dank, die den IP-Adressen der Router-Interfaces konsistente Hostnamen geben (Reverse DNS mittels PTR Records im weltweiten Domain Name System). Ohne die bleibt man auf Mutmaßungen zurückgeworfen.

Ping und Routing

Ping-Verwirrung

Ping hat den Vorteil, dass die ausgegebene Zeit den vollständigen Hin- und Rückweg abbildet. Dabei wird routingtechnisch genau derselbe Weg vermessen, den die Pakete von Applikationen entlang laufen, egal ob der aktuelle Pfad symmetrisch oder asymmetrisch ist. Vorsicht ist aber bei der Interpretation der TTL-Angabe geboten: Diese bezieht sich auf das vom Zielrouter neu verschickte Echo-Reply. Deshalb sagt dieser Wert nichts über den Hinweg aus. Aussagen wie “durch die TTL der Ping-Antwort wissen wir, dass der Server so und so viele Hops entfernt ist” treffen nicht zu.

Routing-Entscheidungen

Wieso gibt es überhaupt asymmetrisches Routing? Hierzu muss man zunächst die Funktionsweise von Routern und deren Forwarding-Entscheidungen verstehen: Jeder Router pflegt eine eigene Routing-Tabelle. Darin stehen entweder statische Routen (selten, beziehungsweise nur in kleinen Netzen) oder Einträge, die Protokolle wie OSPF (Open Shortest Path First in internen Netzen) oder BGP liefern (Border Gateway Protocol im weltweiten Internet).

Die Entscheidung, über welchen Weg ein Zielnetzwerk erreicht wird, fällt jeder Router eigenständig anhand seiner Routing-Tabelle – unabhängig von allen anderen Routern um ihn herum. Deshalb gilt: Nur weil Router A ein bestimmtes Zielnetz kennt, heißt das noch nicht, dass Router B dieses ebenfalls kennt. Und nur weil ein Router den Hinweg zu einem Zielnetz kennt, heißt das noch nicht, dass der Rückweg über die gleichen Router läuft.

Ein Netzwerk-Admin kann die Entscheidungen der einzelnen Router anhand diverser Parameter und deren Routing-Protokolle anpassen. Solange alle relevanten Netze zum gleichen Autonomen System (AS) gehören, hat der Admin volle Kontrolle. Wenn es allerdings um das Routing im Internet geht, wird es schwierig.

Beispielsweise kann eine Firma, die Anschlüsse von zwei verschiedenen ISPs verwendet und daher ihre Präfixe per BGP bekannt gibt, bestimmen, dass ausgehender Verkehr bevorzugt über Router 1 von ISP 1 läuft – den Router kontrolliert schließlich der Admin. Was die Rückrouten betrifft, anhand derer alle möglichen Router dieser Welt ihre Routing-Entscheidungen treffen, wird es schwierig bis unmöglich – schließlich stehen diese Router unter der Kontrolle anderer Admins. Im Internet sind daher asymmetrische Pfade eher der Standard als die Ausnahme. Zwar kann man das Path Pretend von BGP nutzen, um draußen zum Beispiel mitzuteilen, dass externe Router für die Rückroute den Pfad über ISP 2 nicht bevorzugen sollen. – viele ISPs richten sich sogar danach –, aber man kann es nicht erzwingen. De facto kommt Traffic trotzdem immer über beide ISPs zurück.

Möchten Sie wissen, über welche Autonomen Systeme Ihre IP-Verbindungen laufen? Lassen Sie Traceroute auf Unix mit der Option -A laufen, damit es die Nummern der AS (ASN) meldet. Technisch wird dabei die IP-Adresse des jeweiligen Routers per Whois-Befehl nachgeschlagen; die Adressen stecken in den Whois-Datenbanken von IP-Registraren wie dem RIPE. Häufig sind nur wenige Autonome Systeme beteiligt: Das AS Ihres ISP, ein bis zwei Transitnetzwerke sowie das AS des Ziels. Innerhalb eines AS-Netzwerks laufen die Pakete meist über mehrere, dem jeweiligen AS unterstellte Router.

Ping mit Traceroute

Ein hervorragendes Tool für das Netzwerk-Monitoring ist My traceroute, kurz mtr. Auf Linux installieren Sie es mit den Standard-Paketmanagern, auf macOS wahlweise via HomeBrew oder MacPorts. Windows Nutzer nehmen WinMTR. Das einfachste Befehlsmuster sieht wie bei ping oder traceroute aus: mtr heise.de.

In der Voreinstellung sendet das Tool sekündlich je 30 Ping-Pakete mit Hop-Limits von 1 bis 30. Es misst also den gesamten Pfad kontinuierlich und gibt die Antwortzeiten der einzelnen Hops in mehreren Spalten aus: letzter Ping, bester Wert, schlechtester Wert, Durchschnitt (Avg) und Standardabweichung (StDev).

Damit lassen sich etwaige Ausfälle auf dem Routingpfad live feststellen. Wenn ein paar Hops plötzlich schweigen und stattdessen Zeilen neuer Router auftauchen, dann ist das ein Beleg dafür, dass gerade ersatzweise eine neue Route eingeschlagen wurde.

Die zusätzliche Angabe des prozentualen Verlusts der Antwortpakete (Loss%) gibt einen Überblick über die empfangenen ICMP-Meldungen. Drücken Sie die Taste D zwei Mal, um die Anzeige des Display Mode so umzuschalten, dass er die letzten Ping-Laufzeiten farblich hervorhebt.

Angriffspunkt

In der Ausgabe von Traceroute sollte für jedes Netzwerksegment genau ein Router stehen. Bei IPv6 ist das durch die festen /64er-Masken einfach zu erkennen, bei IPv4 haben Sie zumindest in Ihren eigenen Netzen die Möglichkeit herauszufinden, wie die Subnetzmasken lauten. Sollten mal in einem Segment zwei Router stehen, stimmt etwas nicht. Entweder haben Sie dort wirklich mehrere Router – dann verfügen nicht alle über korrekte (statische) Routen – oder jemand hat einen Router als Man-in-the-Middle in Ihr Netz eingeschleust.

Dabei gibt sich ein Angreifer wahlweise via ARP-Spoof (IPv4) oder via NDP-Spoof (IPv6) als Default Router aus, wodurch er jedweden Traffic empfängt (Hop 1) und diesen an den echten Router weiterleitet (Hop 2), um die IP-Verbindungen nicht zu unterbrechen. So liest er alles mit, ohne gleich aufzufallen.

Aber was genau vorliegt, das bringen nur weitere Analysen zutage. Ein guter Startpunkt ist die Suche nach dem Switch-Port, an dem der verdächtige Router angeschlossen ist. Lesen Sie dazu im Client die MAC-Adresse des Default-Routers aus dem ARP-Cache aus und schauen Sie am Switch nach, über welchen Port diese MAC angesprochen wird. Dann lesen Sie die MAC Ihres tatsächlichen Default-Routers aus und vergleichen diese mit der vom Client verwendeten. Wenn sich die beiden unterscheiden, verfolgen Sie das Kabel vom verdächtigen Switch-Port, um den überzähligen und vermutlich eingeschleusten Router zu finden.

Eine Beispielausgabe für ein Traceroute ohne zusätzlichem Router sieht so aus:

tracert 2001:db8:72ed:a9d7:ba3f:57be:af5b:de2f
1 <1 ms  1 ms  1 ms 2001:db8:72ed:46::1
2  1 ms  1 ms  1 ms 2001:db8:72ed:a9d7:ba3f:57be:af5b:de2f

Und so sieht die Ausgabe aus, wenn ein zusätzlicher Router im Segment steckt:

tracert 2001:db8:72ed:a9d7:ba3f:57be:af5b:de2f
1  1 ms <1 ms <1 ms 2001:db8:72ed:46:5d31:5dc1:315:13ef
2  1 ms  1 ms  1 ms 2001:db8:72ed:46::1
3  1 ms  1 ms  1 ms 2001:db8:72ed:a9d7:ba3f:57be:af5b:de2f

Wireshark und tcpdump

Ein letzter Tipp zum Abschluss: Wenn Sie den Verkehr mit Wireshark oder tcpdump mitschneiden, dann lohnt sich meistens die Angabe eines Capture-Filters, um nicht gleich alle, sondern nur die relevanten Pakete mitzuschneiden. Gängig sind Filter unter Angabe der Quell- und/oder Ziel-IP-Adressen sowie der TCP-/UDP-Ports, beispielsweise host 192.168.1.10 and host 8.8.8.8 and port 53.

Doch mit solchen Filtern fängt man leider gar keine ICMP-Fehlermeldungen ein. Dabei ist leicht vorstellbar, dass ein Router auf dem Pfad nützliche ICMP-Pakete sendet, die sogar beim mitschneidenden Gerät ankommen – ohne geeigneten Filter werden sie aber ignoriert. Lösung: Immer den Zusatz or icmp or icmp6 an das Ende des Filters anhängen. So greift man alle ICMP- und ICMPv6-Pakete ab (im Filter ohne v geschrieben, also icmp6), egal von welcher Quell-IP-Adresse sie stammen. So kommen Sie bei Ihrer späteren Analyse in Wireshark versteckten Problemen außerhalb der eigentlichen IP-Sessions auf die Schliche.

Soli Deo Gloria!

Photo by Chris Liverani on Unsplash.