… since we all can use
pool.ntp.org? Easy answer: Many modern (security) techniques rely on accurate time. Certificate validation, two-factor authentication, backup auto-deletion, logs generation, and many more. Meanwhile we use an unauthenticated protocol (via stateless UDP) from unauthenticated sources (NTP pool) to rely on! Really?
If you are using couple of different NTP sources it might be not that easy for an attacker to spoof your time – though not unfeasible at all. And think about small routers with VPN endpoints and DNSSEC resolving enabled, or IoT devices such as cameras or door openers – they don’t even have a real-time clock with battery inside. They fully rely on NTP.
This is what this blogpost series is all about. Let’s dig into it. ;)
Why am I starting a long series about NTP? Because I have many customers and colleagues that are way too lazy when it comes to NTP. “It’s working with some servers on the Internet, so why should I care?”
Why NTP matters
I basically want to cite this mail from the NANOG (North American Network Operators Group) mailing list:
“How about you set the time on your server ahead by 5 years. Got any idea what would happen?
- Most of your passwords would expire.
- All your SSL certs would expire.
- All your TOTPs, like Google Authenticator would fail.
- All your IPSEC tunnels would drop, and refuse to restart.
- Many of your cron jobs would got nuts, possibly deleting all your logs.
- Much of your DNSSEC would expire.
- Many of your backups would be deleted since they ‘expired’.
Until recently, setting your iPhone to 1 Jan 1970 would brick it. I’m sure there are many more examples, but likely you can no longer log in, via SSH or HTTPS, and your iPhone is dead. I think any of those would qualify as more than an annoyance.”
Think about it. 
Want some more resources? Have a look at this paper: Bypassing HTTP Strict Transport Security [PDF] which almost only leverages a man-in-the-middle tool for NTP. Uh.
Or this one: “The most common cause for secure connection errors turns out to be user systems having the wrong time“, Panos Astithas at Feeling safer online with Firefox.
And something from : “What can we say about time and the Internet? The safest assumption is that most systems will be in sync with a UTC reference clock source as long as the definition of “in sync” is a window of 24 hours! If the ‘correct’ behaviour of an Internet application relies on a tighter level of time convergence with UTC time than this rather large window, then it’s likely that a set of clients will fall outside of the application’s view of what’s acceptable“, at APNIC Labs – What’s the Time?.
And what about the Internet of Things? IoT devices merely rely on NTP since they don’t have a built-in battery-powered real-time clock. That is: After booting they don’t have a time at all until they get one over NTP. Spoofing the NTP servers or packets would immediately mess up their time.
And one more example out of a networker’s daily business: Taking multipoint packet captures without synchronized times is a nightmare! You have to sync the time on your capture devices to have a chance to find your relevant packets all over your traces.
What to do? Usage of own Servers & Authentication!
You should use three on-site NTP servers as well as NTP authentication. That is:
- Use your own NTP servers inside your own network with different external clock sources. Advantages:
- You have your own stratum 1 NTP servers that are independent of other NTP instances on the Internet. Unknown NTP servers might be out of sync or not reachable over time.
- Using own antennas for external sources omits the dependency on single third parties. You should use at least two different external source such as DCF77 and GPS. That is: You should not only rely on the time codes of a single foreign country.
- You can rely on your own hardware and their validity because it is not easy to alter them at all (as long as they reside in your own physically secure data center). You can uniformly control your servers, such as inserting leap seconds.
- Your whole company gets almost exactly the same time since they’re using the same NTP servers throughout your enterprise which is not the case if every single device uses the NTP pool on its own which results in using hundreds of different NTP servers.
- All NTP connections are routed within your own network, hence no one on the Internet can spoof your packets. Furthermore, you have low network latency and jitter compared to routed packets through the Internet.
- Your own devices can alert you in case of failures, e.g. by mail, SNMP, syslog.
- Use three independent NTP servers. Why? Because of redundancy. One server might fail, hence you need at least two. However, if one out of two fails (that is: delivers a falsified timestamp) your clients don’t know which of those two sources to trust. Hence you should use at least three NTP servers. If one out of three fails, your clients are still able to get the correct time, since two correct sources win against one faulty. For further redundancy you should position them in different physical locations as well as logical IP subnets.
- Use NTP authentication on your NTP servers and clients. With an authentication mechanism leveraging hash functions such as SHA-1, your NTP clients can validate that they are talking to the correct source and are not spoofed by an attacker using a man-in-the-middle (MITM) attack. NTP authentication works with symmetric pre-shared keys. You need to generate different keys (at least per security zone, better per device) on the server in order to configure the clients appropriately. If one key gets lost, you can replace it while not loosing the security of all other keys/devices.
Blogpost Series w/ Raspberry Pis and Meinberg LANTIME
For this blogpost series I primarily used a couple of Raspberry Pis. Simply because they are cheap, have small power consumption, and you can easily use the IO ports to connect other devices such as a GPS or DCF77 receiver modules. Works fine. However: You should NOT rely on those Pis in your production environment. Why? Because they don’t have a real-time clock which keeps the own time accurate. Hence they merely rely on their stratum 0 input. If this signal is lost, you will drift in time. Furthermore, they are not stable at all, hard to update (as you will see during my series) and everything needs to be configured by hand. Hence: Only use dedicated NTP appliances that are made to serve precise timestamps all over your network. For my posts I am additionally using a Meinberg LANTIME M-200 device which I am comparing to the Pis at some points.
This blogpost series is split into three parts:
- Building stratum 1 NTP servers: How to use Raspberry Pis for NTP servers, advantages of purpose-built NTP appliances, load-balancing NTP, and so on.
- Using NTP authentication: Securing NTP packets with symmetric keys on several devices such as firewalls.
- Monitoring NTP servers: Getting telemetry data and stats from NTP servers.
All posts are related to my three IPv6-only stratum 1 NTP servers:
ntp1.weberlab.de
, a Raspberry Pi with a DCF-77 receiver,ntp2.weberlab.de
, a Raspberry Pi with a GPS receiver andntp3.weberlab.de
, a Meinberg LANTIME M200 NTP Server with DCF77 and phase modulated PRNG.
Feel free to use them via
ntp.weberlab.deonly. This FQDN will list only the IP addresses of those servers that are available. Of course you should only use them for test purposes, NOT for your enterprise, as this article is all about running your own servers. ;)
Throughout this series I am only covering NTP but not PTP (Precision Time Protocol). Furthermore, I am not attacking NTP via MITM attacks (at least not now) but only show how to prevent them with NTP authentication, nor do I attack the external time sources DCF77 or GPS. And I am not diving into DDoS attacks that use NTP servers (reflection & amplification attack), since I assume that you’ll primarily use your servers only within your own network and not on the public Internet.
Further Reading
As always you can read even more. ;) If you’re interested in the deep details about the network time protocol itself you should have a look at the Internet Protocol Journal volume 15, number 4 [PDF] and its article “Protocol Basics: The Network Time Protocol” from Geoff Huston. Anyway, here are some more links:
- RFC 5905: Network Time Protocol Version 4: Protocol and Algorithms Specification
- RFC 7822: Network Time Protocol Version 4 (NTPv4) Extension Fields
- Time Synchronization Blog – NTP vs PTP: Network Timing Smackdown!
- Time Synchronization Blog – PTP: Why is IEEE 1588 so accurate?
- Aaron Toponce – Real Life NTP
- Meinberg – Network Time Protocol
- Ivan Pepelnjak – It’s Good to be on Time [PDF]
- (de) PTB – Zeitsynchronisation von Rechnern mit Hilfe des “Network Time Protocol”
- RIPE Labs – NTP Measurements with RIPE Atlas
- Cloudflare System Status – Some DNS lookups causing 5xx errors due to leap second bug
- APNIC Labs – What’s the Time?
As well as security related stuff:
- Boston University – The Security of the Network Time Protocol
- ACM Queue – Securing the Network Time Protocol
- Hanno’s blog – Don’t update NTP – stop using it
- Stephen Röttger – Analysis of the NTP Autokey Procedures [PDF]
Disclaimer
I am not set against the NTP pool project in general. It is a great community driven project that I am using client-side (for many small devices such as routers, Raspberry Pis, IoT, …) as well as server-side since I am contributing with at least one online NTP server. However, my focus with this blogpost series is the enterprise network. And you should not use the NTP pool within your enterprise as at all, but own independent stratum 1 NTP appliances along with NTP authentication.
Featured image “Breitling Super Avenger” by W.A.Smith is licensed under CC BY-NC-ND 2.0.