An analysis of some falsified leap second warnings that appeared in November 2021 on public NTP servers out of the NTP Pool Project.
Introduction
When using time scales such as UTC that do not use daylight saving time, each day has a strict 60 x 60 x 24 = 86400 seconds pattern. But due to variation in the earth’s rotation, the last day of a month may have 86399 or 83401 seconds; these are caused by negative/positive leap seconds. Leap seconds are announced by IERS six months prior to the event. All previous leap seconds have been positive and occurred on the last day of December or June. The most recent leap second as of now occurred on December 31, 2016.
Unexpected Leap Second Warning
On November 27, 2021, several public Network Time Protocol (NTP) servers began advertising “LI=1” (leap indicator), meaning that a positive leap second is coming at the end of the day. This was odd since no leap second was scheduled during 2021. Leap seconds normally occur only at the end of June 30 or December 31, UTC. Of the thousands of NTP servers I am monitoring, none had been advertising LI=1 in recent months.
But this wasn’t the end of the warnings. At 2021-11-30 00:00 a largely different set of NTP servers began sending LI=1. Some NTP software keeps an internal leap-second-pending flag and delays leap second processing until the last day of the current month. Following is the count of NTP servers as recognized by my monitoring system:
2021-11-27 LI=1 warning:
- 72 servers total
- 53 stratum 1 (all indicated reference ID=GPS); LI=1 was steady when it occurred
- 17 stratum 2
- 2 stratum 3
2021-11-30 LI=1 warning:
- 86 servers total (stratum varied for some servers)
- 4 stratum 1 (two indicated reference ID=GPS; two indicated reference ID=PPS)
- 58 stratum 2
- 28 stratum 3
- 6 stratum 4
A number of servers had leap indicators that changed from 0 to 1 and back again. Only four stratum 2 NTP servers were common in the November 27 and November 30 sets.
Leap Second Execution
2021-11-28 00:00:00: For most of these NTP servers, the error was corrected after 1 second; during the one-second interval, the time-of-day was frozen at 00:00:00. Eight servers were in error by 1-second for between two and eight minutes. The 1-second error persisted on one server for 00:39:26.
2021-12-01 00:00:00: The duration of the 1-second time of day difference was generally longer, persisting for up to 01:22:31.
Affected Clients
As part of a study on unwanted high-rate NTP request bursts, client NTP requests were regularly recorded on several NTP servers including the client’s leap indicator.
| NTP Server Location | Fraction of NTP clients sending LI=1 just before 2021-12-01 00:00 UTC (based on one 20 minute sample for each server) |
| Wellington, NZ | 0.025% overall, 130 clients affected |
| San Francisco, US | 0.294% overall, 7011 clients affected |
| London, UK | 0.635% overall, 4791 clients affected |
All three of these servers had LI set to 0, which is correct. It seems that clients with LI=1 were using one or more of the affected NTP servers.
The New Zealand server was continuously collecting logs which allowed the number of LI=1 clients to be tracked throughout these two days.
Miroslav Lichvar (who maintains chrony) commented: “I suspect the actual percentages were likely even larger as most clients don’t put their leap state in their requests.“
Common Thread – Impacted NTP Servers
Among the NTP servers advertising LI=1 on 2021-11-27 was one of my home servers, an inexpensive, high-performance stratum 1 LeoNTP. This was seen on Twitter:
⏲️ LeoNTP users – please update FW to v1.24 to avoid erroneous Nov 2021 leap second warning: https://t.co/gvbF90VLgF
Ublox GPS chipset we use [incorrectly] reports leap second event date as GPS week 2185 (Nov 2021) instead of 1929 (Dec 2016.) pic.twitter.com/ZWEkuAB5U2— Leo_Bodnar (@LeoBodnar) July 25, 2021
Upon applying the new firmware my LeoNTP server correctly set LI=0.
At least 42 of the Stratum 1/GPS servers advertising LI=1 on 2021-11-27 were LeoNTP units. Little is known about the other 16. None of these servers showed incorrect behaviour on 2021-11-30.
I estimate that these 42 LeoNTP servers were receiving in excess of 30,000 requests/second on 2021-11-27. These servers were likely providing time to hundreds of thousands and possibly millions of clients, all were being told that a leap second was coming. It isn’t known how many of those clients executed a leap second on 2021-11-28 00:00:00. The detailed server logs indicate that a number did execute the leap second and were in error by one second for some minutes.
Event Summary and Fallout
November 27, 2021:
- 53 NTP stratum 1 servers, including 42 LeoNTP units, begin advertising LI=1. A fix was available from the manufacturer but was not installed on many public servers.
- At midnight (2021-11-28 00:00:00), most of these servers incorrectly executed a leap second procedure resulting in a short (typically one second) time of day error.
- The number of clients executing an unplanned leap second procedure is unknown.
November 30, 2021:
- 86 NTP servers, largely different from the November 27 set, began advertising LI=1 at various times during the day. The leap indicator was erratic on some servers.
- At midnight (2021-12-01 00:00:00), many of the servers executed an unwanted leap second procedure. A one-second time offset was present for up to 1.3 hours.
- The number of clients executing an unplanned leap second procedure is unknown.
A few messages were exchanged in Time-nuts and NTP mailing lists. Otherwise, this event apparently passed unnoticed.
Déjà Vu
The erroneous leap second warning was anticipated.
Tom Van Baak described [PDF] a similar event happening almost two decades earlier.
The November 2021 incident had only a 1-second error and was simply accepted by many NTP clients.
Background: Leap Second Dissemination
Leap seconds typically occur on the last day of June or December. Leap seconds can occur in other months, but that has never happened. Bulletin C, published by the International Earth and Rotation Reference Systems Service (IERS) twice per year, gives the upcoming leap second status. The last leap second was December 31, 2016. No leap second was announced for 2021.
Upcoming leap second announcements are redistributed via GPS, NIST’s WWV/WWVB and ACTS service, PTB’s DCF77 and other mechanisms, including NTP. Rather than depending on possibly noisy upstream sources, many use a leap second table as a definitive source of past and upcoming leap seconds.
Leap seconds can only occur on the last day of a month, some NTP software does not enforce that rule.
Leap seconds have triggered many problems. Apparently the impact of the errant November 2021 leap second was insignificant. There is strong sentiment for eliminating leap seconds but the required decisive action has not happened.
Photo by Walker Fenton on Unsplash.