Some months ago, my co-worker and I ran into an interesting issue: a notebook with a newly installed Ubuntu 20.04 does only work with IPv4, but this office network is dual-stacked (IPv4 and IPv6). Other Linux clients as well as Windows and Mac systems still work fine. They all get an IPv4 configuration by DHCPv4 and an IPv6 configuration by stateful DHCPv6 from the same DHCP server, relayed by a Cisco ASA 5500-X. What’s wrong with Ubuntu 20.04?
Our first idea was, that we forgot to activate global IPv6 during installation. But it was enabled. Our next idea was an active local firewall that blocks DHCPv6 packets or IPv6 in general. This is a frequent problem because DHCPv6 is using UDP ports 546 and 547. But the same notebook at a home network works fine with IPv6 and IPv4. We had to look at our network…
In this network, a Cisco ASA is acting as the default router to the internet. It’s also acting as a DHCP relay system for IPv4 and IPv6 to forward DHCP packets of the clients to the central DHCP server and back. But the DHCPv6 server didn’t get a SOLICIT message from the client; DHCPv4 works.
Some log lines from the ASA got out attention:
DHCPv6: Bogus option CLIENTID(1), len 18 IPv6 DHCP_RELAY: Discard SOLICIT with invalid or no options
At this time we didn’t understand the log lines. Because we had no better idea, we decided to activate some debug commands at our ASA:
ASA-X# debug ipv6 dhcp ASA-X# debug ipv6 dhcp client ASA-X# debug ipv6 dhcp server ASA-X# debug ipv6 dhcprelay ASA-X# debug timestamps ASA-X# term mon ASA-X#
Maybe that’s a few commands too much, but we wanted to get a hint of what we are doing wrong. Less than 100 seconds later we got the following lines:
8410825.8800: DHCPv6: Received SOLICIT from fe80::3afd:e025:f6b:75d9 on vlan6 8410825.8800: IPv6 DHCP: detailed packet contents 8410825.8800: src fe80::36fc:a6d5:f6b:75d9 (vlan6) 8410825.8800: dst ff02::1:2 8410825.8800: type SOLICIT(1), xid 4903931 8410825.8800: option RAPID-COMMIT(14), len 0 8410825.8800: option IA-NA(3), len 12 8410825.8800: IAID 0x2d1aa133, T1 0, T2 0 8410825.8800: option UNKNOWN(39), len 10 8410825.8800: option ORO(6), len 8 8410825.8800: DNS-SERVERS,DOMAIN-LIST,UNKNOWN,SNTP-ADDRESS 8410825.8800: option CLIENTID(1), len 18 8410825.8800: 0004xxxxxxxxxxxxxxxxxxxxxxxxxxxxFDA2 8410825.8800: option ELAPSED-TIME(8), len 2 8410825.8800: 8410825.8800: IPv6 DHCP_RELAY: DHCPD/RA: Message received from cluster module 8410825.8800: DHCPv6: Bogus option CLIENTID(1), len 18 8410825.8800: IPv6 DHCP_RELAY: Discard SOLICIT with invalid or no options
This tells us: there is a client that wants to get an IPv6 configuration (solicit) but the ASA decides the DHCPv6 client ID with a length of 18 octets is wrong. Because of this, the ASA throws away the solicit message.
OK. Now we know the device which causes the problem. But what’s wrong?
We searched around and found, Cisco knows this issue: ASA dropping DHCPv6 SOLICIT from Unix-based systems (unfortunately only accessible with a Cisco Account). They recommend a workaround: “Modify the client UUID to use UUID of 1”. We had no idea why changing the system UUID should be a way to get DHCPv6 working…
Now we had a more detailed look at the messages we got in the ASA console. There we saw from Windows system lines like this:
8411325.7600: option CLIENTID(1), len 14 8411325.7600: 0001000126C41D660050568FADC3
The Ubuntu system causes these lines:
8411955.8800: option CLIENTID(1), len 18 8411955.8800: 000416BD597F6092FBB0866BC7B6E23DFDA2
Some more hours later we understood the issue like this: Cisco implemented the relaying of DUID types 1-3, [not UUID!] defined in RFC3315 (July 2003), but not the additional DUID type 4, defined in RFC6355 (August 2011). And we get the hidden hint from Cisco for their workaround: change the DUID (DHCP unique identifier) to a type 1 DUID.
For Ubuntu Linux we didn’t find a (for us working) way the change DUID type to 1; but in a new Debian 10 installation of another co-worker, we run in a similar issue. After installation the system used a DUID type 4, too:
# grep duid /var/lib/NetworkManager/dhclient6*.lease default-duid "\000\004\254z\203\375rCa*xxxxxxxxxxxxxx\275\233";
For the Debian Linux system, we found the workaround: we delete the lease file and create a new one with a DUID type 1 (DUID-LLT = link-layer address plus time ).
rm /var/lib/NetworkManager/dhclient6-*.lease dhclient -6 -D LLT -r ens32 systemctl restart NetworkManager.service
Of course, you can use a DUID type 2 (vendor-assigned unique ID based on Enterprise Number) or DUID type 3 (link-layer address), too.
This is just a workaround for a network with less affected clients. If you have to publish this solution for a large bring-your-own-device (BYOD) network with many affected clients: Have fun!
At the time of writing these lines, there is an update at the Cisco case which indicates that there should be a new version of the ASA operating system which should fix the DUID type 4 issue of the DHCPv6 relay agent. We are looking forward to get the new version, maybe in a few years. :-)