I found the most interesting part of the NIST outage post [1] is NIST's special Time Over Fiber (TOF) program [2] that "provides high-precision time transfer by other service arrangements; some direct fiber-optic links were affected and users will be contacted separately."
I've never heard of this! Very cool service, presumably for … quant / HFT / finance firms (maybe for compliance with FINRA Rule 4590 [3])? Telecom providers synchronizing 5G clocks for time-division duplexing [4]? Google/hyperscalers as input to Spanner or other global databases?
Seriously fascinating to me -- who would be a commercial consumer of NIST TOF?
I never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems. Sub-microsecond latency, yes, but when that mattered it was in order to do something as soon as possible rather than as close as possible to Wall Clock Time X.
Still useful for post-trade analysis; perhaps you can determine that a competitor now has a faster connection than you.
The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.
> I never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems.
> The required accuracy (Tables 1 and 2 in that document)
no, Tables 1 and 2 say divergence, not accuracy
accuracy is a mix of both granularity and divergence
regardless, your statement before:
> The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.
> accuracy is a mix of both granularity and divergence
I respectfully disagree.
In context, "granularity" is nothing more than a resolution constraint on reported timestamps. Its inclusion adjacent to the specified "divergence from UTC" is a function of market manipulation surveillance objectives as discussed in preamble item (2), and really doesn't have anything to do with accuracy proper.
My guess would be scientific experiments where they need to correlate or sequence data over large regions. Things like correlating gravitational waves with radio signals and gamma ray bursts.
Would the police actually try to investigate from where came the jammer? Might the competing firm possibly even finance an investigation themselves privately? And if so, would the police then accept the evidence?
But they do not need absolute time, and internal rubidium clocks can keep the required accuracy for a few days. After that, sync can be transferred with a portable plug, which is completely viable in tactical/operational level EW systems.
Google doesn't use chrony specifically, just an algorithm that is somewhat chrony-like (but very different in other ways). It's called Google TrueTime.
This page: https://tf.nist.gov/tf-cgi/servers.cgi shows that NIST has > 16 NTP servers on IPv4, of those, 5 are in Boulder and were affected by the power failure. The rest were fine.
However, most entities should not be using these top-level servers anyway, so this should have been a problem for exactly nobody.
Who does use those top-level servers? Aren’t some of them propagating the error or are all secondary level servers configured to use dispersed top-level servers? And how do they decide who is right when they don’t match?
Is pool.ntp.org dispersed across possible interference and error correlation?
You can look at who the "Stratum 2" servers are, in the NTP.org pool and otherwise. Those are servers who sync from Stratum 1, like NIST.
Anyone can join the NTP.org pool so it's hard to make blanket statements about it. I believe there's some monitoring of servers in the pool but I don't know the details.
For example, Ubuntu systems point to their Stratum 2 timeservers by default, and I'd have to imagine that NIST is probably one of their upstreams.
An NTP server usually has multiple upstream sources and can steer its clock to minimize the error across multiple servers, as well as detecting misbehaving servers and reject them ("Falseticker"). Different NTP server implementations might do this a bit differently.
From my own experience managing large numbers of routers, and troubleshooting issues, I will never use pool.ntp.org again. I’ve seen unresponsive servers as well as incorrect time by hours or days. It’s pure luck to get a good result.
Instead I’ll stick to a major operator like Google/Microsoft/Apple, which have NTP systems designed to handle the scale of all the devices they sell, and are well maintained.
I believe if you use time.nist.gov it round robins dns requests, so there’s a chance you’d have connected to the Boulder server. So for some people they would have experienced NIST 5 μs off.
Nitpick: UTC stands for Coordinated Universal Time. The ordering of the letters was chosen to not match the English or the French names so neither language got preference.
That doesn't quite match what the wikipedia page says:
> The official abbreviation for Coordinated Universal Time is UTC. This abbreviation comes as a result of the International Telecommunication Union and the International Astronomical Union wanting to use the same abbreviation in all languages. The compromise that emerged was UTC, which conforms to the pattern for the abbreviations of the variants of Universal Time (UT0, UT1, UT2, UT1R, etc.).
> ... in English the abbreviation for coordinated universal time would be CUT, while in French the abbreviation for "temps universel coordonné" would be TUC. To avoid appearing to favor any particular language, the abbreviation UTC was selected.
Not exactly the topic of discussion but also not not on topic: just wanted to sing praise for chrony which has performed better than the traditional os-native NTP clients in our testing on a myriad of real and virtualized hardware.
Maybe I missed something, but I don't quite understand the video title "NIST's NTP clock was microseconds from disaster". Is there some limit of drift before it's unrecoverable? Can't they just pull the correct time from the other campus if it gets too far off?
I'm missing the nuance or perhaps the difference between the first scenario where sending inaccurate time was worse than sending no time, versus the present where they are sending inaccurate time. Sorry if it's obvious.
The 5us inaccuracy is basically irrelevant to NTP users, from the second update to the Internet Time Service mailing list[1]:
To put a deviation of a few microseconds in context, the NIST time scale usually performs about five thousand times better than this at the nanosecond scale by composing a special statistical average of many clocks. Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet; uncertainties on the order of 1 millisecond (one thousandth of one second) are more typical due to asymmetry and fluctuations in packet delay.
> Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet
How do those other applications obtain the precise value they need without encountering the Internet issue?
> If those other applications use their own local GPS clocks, what is the significance of NIST (and the 5μs inaccuracy) in their scenario?
Verification and traceability is one reason: it's all very well to claim you're with-in ±x seconds, but your logs may have to say how close you are to the 'legal reality' that is the official time of NIST.
NIST may also send out time via 'private fibre' for certain purposes:
That is not correct at all. How did you arrive at that conclusion?
GPS has its own independent timescale called GPS Time. GPS Time is generated and maintained by Atomic clocks onboard the GPS satellites (cesium and rubidium).
It has its own timescale, but that still traces back to NIST.
In particular, the atomic clocks on board the GPS satellites are not sufficient to maintain a time standard because of relativistic variations and Doppler effects, both of which can be corrected, but only if the exact orbit is known to within exceeding tight tolerances. Those orbital elements are created by reference to NIST. Essentially, the satellite motions are computed using inverse GPS and then we use normal GPS based on those values.
I think GP might’ve been referring to the part of Jeff’s post that references GPS, which I think may be a slight misunderstanding of the NIST email (saying “people using NIST + GPS for time transfer failed over to other sites” rather than “GPS failed over to another site”).
The GPS satellite clocks are steered to the US Naval Observatory’s UTC as opposed to NIST’s, and GPS fails over to the USNO’s Alternate Master Clock [0] in Colorado.
Yes, either Rb, Cs, or H standards depending on which GNSS system you're using.
For the most critical applications, you can license a system like Fugro AtomiChron that provides enhanced GNSS timing down to the level of a few nanoseconds. There are a couple of products that do similar things, all based on providing better ephemerides than your receiver can obtain from the satellites themselves.
That's one hell of a healthy profit margin there O.o
The SiTime MEMS oscillator is about 100€ for one single chip, the mosaic-T GPS receiver is about 400€. Add 50€ for the rest (particularly the power input section looks complicated) and 50€ for handling, probably 600€ in hardware cost... sold for 2.500€.
The real money I think went into certification and R&D for a low-volume product - even though most of the hard work is done by the two ICs, getting everything orchestrated (including the PCB itself) to perform to that level of accuracy is one hell of a workload.
A lot of organizations also colocate timing equipment near the actual clocks, and then have 'dark fiber' between their equipment and the main clock signals.
Then they disperse and use the time as needed.
According to jrronimo, they even had one place splice fiber direct between machines because couplers were causing problems! [1]
If I put my machine near the main clock signal, I have one clock signal to read from. The comment above was asking about how to average across many different clocks, presumably all in different places in the globe? Unless there's one physical location with all of the ones you're averaging, you're close to one and far from all the others so how is it done without the internet?
Can you do PTP over the internet? I have only seen it in internal environments. GPS is probably the best solution for external users to get time signals with sub-µs uncertainties.
It's a good question, and I wondered the same. I don't know, but I'd postulate:
As it stands at the minute, the clocks are a mere 5 microseconds out and will slowly get better over time. This isn't even in the error measurement range and so they know it's not going to have a major effect on anything.
When the event started and they lost power and access to the site, they also lost their management access to the clocks as well. At this point they don't know how wrong the clocks are, or how more wrong they're going to get.
If someone restores power to the campus, the clocks are going to be online (all the switches and routers connecting them to the internet suddenly boot up), before they've had a chance to get admin control back. If something happened when they were offline and the clocks drifted significantly, then when they came online half the world might decide to believe them and suddenly step change to follow them. This could cause absolute havoc.
Potentially safer to scram something than have it come back online in an unknown state, especially if (lots of) other things are are going to react to it.
In the last NIST post, someone linked to The Time Rift of 2100: How We lost the Future --- and Gained the Past. It's a short story that highlights some of the dangers of fractured time in a world that uses high precision timing to let things talk to each other: https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...
> […] where sending inaccurate time was worse than sending no time […]
When you ask a question, it is sometimes better to not get an answer—and know you have not-gotten an answer—then to get the wrong answer. If you know that a 'bad' situation has arisen, you can start contingency measures to deal with it.
If you have a fire alarm: would you rather have it fail in such a way that it gives no answer, or fail in a way where it says "things are okay" even if it doesn't know?
I work at a particle accelerator. We use White Rabbit (https://white-rabbit.web.cern.ch/) to synchronize some very sensitive devices, mostly the RF power systems and related data acquisition systems, down to nanosecond accuracy.
As far as I'm aware they just timestamp the sample streams based on a local gps backed atomic reference. Then when they get the data/tapes in one computing center they can just run a more sophisticated correlation entirely in software to smooth things out.
I worked at Altera (FPGA supplier) as the Ethernet IP apps engineer for Europe for a few years. All the big telecoms (Nokia, Ericsson, Cisco, etc) use Precision Time Protocol (PTP) in some capacity and all required clocks to be ns levels for accuracy. Sometimes as low a 10ns at the boundary. Any imperfection in the local clock directly converts into timestamp error, and timestamp error is what limits PTP synchronization performance. Timestamps are the fundamental observable in PTP.
Quantization and jitter create irreducible timestamp noise.
That noise directly limits offset and delay estimation.
Errors accumulate across network elements and internal clock error must be much smaller than the system requirement.
I think most people would look at the error and think "what's the big deal" but at all the telecoms customers would be scrambling to find a clock that hasn't fallen out of sync.
We don't use NTP, but for robotics, stereo camera synchronization we often want the two frames to be within ~10us of eachother. For sensor fusion we then also need a lidar on PTP time to be translated to the same clock domain as cameras, for which we also need <~10us.
We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.
> We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.
This doesn't seem right to me. NTP with default settings should be monotonic. So no jumps. If you disable it Linux enters 11-minute mode, IIRC, and that may not be monotonic.
Pedantically, a monotonic function need not have a constant first derivative. To take it further, in mathematics it is accepted for a monatomic function to have a countable number of discontinuities, but of course in the context of a digital clock that only increments in discrete steps, that’s of little bearing.
But that’s all besides the point since most sane time sync clients (regardless of protocol) generally handle small deviations (i.e. normal cases) by speeding up or slowing down the system clock, not jumping it (forward or backward).
You are correct, NTP prefers to jump first (if needed) and then slew afterwards (which is exactly what we want!), although it can jump again if the offset is too large.
In our case the jumps where because we also have PTP disciplining the same system clock, when you have both PTP and NTP fighting over the same clock, you will see jumping with the default settings.
For us it was easier to just do a one time NTP sync at the beginning/boot, and then sync the robots local network with only PTP afterwards.
For a low precision environment to avoid sudden jumps I used SetSystemTimeAdjustment on Windows (now SetSystemTimeAdjustmentPrecise) to smoothly steer the system clock to match the GPS supplied time signal.
We use nanosecond precision driven by GPS clocks. That timestamp in conjunction with star tracker systems gives us reliable positioning information for orbital entities.
(Assuming "precision" really meant "accuracy") The network equipment I work on requires sub microsecond time sync on the network for 5G providers and financial trading customers. Ideally they'd just get it from GPS direct, but that can be difficult to do for a rack full of servers. Most of the other PTP use cases I work with seem to be fine with multiples of microseconds, e.g. Audio/Video over the network or factory floor things like PLCs tend to be find with a few us over the network.
Perhaps a bit more boring than one might assume :).
Lightning detection. You have a couple of ground stations with known positions that wait for certain electromagnetic puses, and which record the timestamps of such pulses. With enough stations you can triangulate the location of the source of each pulse. Also a great way to detect nuclear detonations.
There is a german club that builds and distrubutes such stations (using GPS for location and timing), with a quite impressive global coverage by now:
At a previous role, we needed nanosecond precision for a simulcast radio communications system. This was to allow for wider transmission for public safety radio systems without having to configure trunking. We could even adjust the delay in nanoseconds to move the deadzones away from inhabited areas.
We solved this by having GPS clocks at each tower as well as having the app servers NTP with each other. The latter burned me once due to some very dumb ARP stuff, but that's a story for another day.
Your speakers do so that people's voices match their mouth movements. The speaker clocks need to be in-sync with the cpu clocks and they operate at different frequencies.
How do you even get usable microsecond precision sync info from a server thousands of kilometers away? The latency is variable so the information you get can't be verified / will be stale the moment it arrives. I'm quite ignorant on the topic.
Yes, but always got it from GPS so presumably they'd be off about the same amount.
Distributed sonar, allows placing receivers willy-nilly and aligning the samples later.
Remote microphone switching - though for this you wouldn't notice 5us jitter, it's just that the system we designed happened to have granularity that good.
When we collected, correlated, and measured all controlling messages in a whole 4G network. Millisecond precision meant guaranteed out of order message flows.
Nuclear measurements, where the speed of a gamma ray flying across a room vs a neutron is relevant. But that requires at least nanosecond time resolution, and you’re a long way from thinking about NTP.
I believe LTE and 5G networks require it to coordinate timeslots between overlapping cells. Of course, they can use whatever reference they want, as long as all the cells are using the same one - it doesn't have to be UTC. Some (parts of) networks transmit it across the network, while others have independent GPS receivers at each cell site.
Synchronization is also required for SDH networks. Don't know if those are still used.
Someone else referenced low power ham radio modes like WSPR, which I also don't know much about, but I can imagine they have timeslots linked to UTC and require accuracy. Those modes have extremely low data rates and narrow bandwidths, requiring accurate synchronization. I don't know if they're designed to self-synchronize, or need an external reference.
When multiple transmitters are transmitting the same radio signal (e.g. TV) they might need to be synchronized to a certain phase relationship. Again, don't know much about it
WSPR doesn't require tight synchronization, but it does require pretty stable frequency sources over periods of 10s of seconds.
It is very common to integrate a GPS in a WSPR beacon to discipline the transmit frequency, but with modest thermal management, very ordinary crystal oscillators have very nice stability.
A database like Google Spanner has higher latency in proportion to the uncertainty about the time. Driving the time uncertainty down into the microsecond range, or lower, keeps latency low.
If you have a widely distributed system and you need tight synchronization, it can be much easier to regulate to an outside stable source than to try to make a stable consensus algorithm.
I mean, we routinely benchmark things that take microseconds or less. I've seen a 300 picosecond microbenchmark (single cycle at 3GHz). No requirement that absolute time is correct, though.
??? The power outage was voluntary and surrounding towns chose not to turn off power. They could absolutely make infrastructure changes, and I’m sure the backups for power could make changes too.
It sounds like GPS, and thus a GPS-based stratum 1 server, uses these time servers, but they were successfully failed over:
> Jeff finished off the email mentioning the US GPS system failed over successfully to the WWV-Ft. Collins campus. So again, for almost everyone, there was zero issue, and the redundancy designed into the system worked like it's supposed to.
So failures in these systems are potentially correlated.
The author mentions another solution. Apparently he runs his own atomic clock. I didn’t know this was a thing an individual could do.
> But even with multiple time sources, some places need more. I have two Rubidium atomic clocks in my studio, including the one inside a fancy GPS Disciplined Oscillator (GPSDO). That's good for holdover. Even if someone were jamming my signal, or my GPS antenna broke, I could keep my time accurate to nanoseconds for a while, and milliseconds for months. That'd be good enough for me.
The CSACs that I have in a couple devices are 'atomic', and use Rubidium, but they're a bit lower accuracy than Cesium clocks [1] or Hydrogen Masers [2].
There are a few folks on the time-nuts mailing list who own such exotic pieces of hardware, but those are pretty far out of reach for most!
Atomic clocks cover a pretty big range of performance nowadays. You can pick up a used but serviceable rubidium frequency reference for a few hundred dollars but the difference between it and the top of the line clocks is almost as big as the difference between a it and a good pendulum clock.
Be aware that there are members of the NTP pool with less-than-honorable intentions and you don't get to pick-and-choose. Yes, they all should provide the time, but they also get your IP address.
For example: unlike the IPv4 space, the IPv6 space is too big too scan, so a number of "researchers" (if you want to call them that) put v6-capable NTP servers in the NTP pool to gather information about active v6 blocks to scan/target.
Is this one of those extraordinary claims that requires evidence? Or is it generally true that there are homey-pots in many of these services (NTP, mirrors, etc)
Most places that need accurate time get it from GPS. That is 10-100 ns.
Also, you can use multiple NIST servers. They have ones in Fort Collins, CO and Gaithersburg, MD. Most places shouldn't use NIST directly but Stratum 1 name servers.
Finally, NTP isn't accurate enough, 10-100 ms, for microsecond error to matter.
Use NTP with ≥4 diverse time sources, just as RFC 5905 suggests doing. And use GPS.
(If you're reliant upon only one source of a thing, and that thing is important to you in some valuable way, then you're doing it wrong. In other words: Backups, backups, backups.)
I found the most interesting part of the NIST outage post [1] is NIST's special Time Over Fiber (TOF) program [2] that "provides high-precision time transfer by other service arrangements; some direct fiber-optic links were affected and users will be contacted separately."
I've never heard of this! Very cool service, presumably for … quant / HFT / finance firms (maybe for compliance with FINRA Rule 4590 [3])? Telecom providers synchronizing 5G clocks for time-division duplexing [4]? Google/hyperscalers as input to Spanner or other global databases?
Seriously fascinating to me -- who would be a commercial consumer of NIST TOF?
[1] https://groups.google.com/a/list.nist.gov/g/internet-time-se...
[2] https://www.nist.gov/pml/time-and-frequency-division/time-se...
[3] https://www.finra.org/rules-guidance/rulebooks/finra-rules/4...
[4] https://www.ericsson.com/en/blog/2019/8/what-you-need-to-kno...
I never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems. Sub-microsecond latency, yes, but when that mattered it was in order to do something as soon as possible rather than as close as possible to Wall Clock Time X.
Still useful for post-trade analysis; perhaps you can determine that a competitor now has a faster connection than you.
The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.
> I never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems.
mifid ii (uk/eu) minimum is 1us granularity
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:...
It's 1 us granularity, which means you should report your timestamps with six figures after the decimal point.
The required accuracy (Tables 1 and 2 in that document) is 100 us or 1000 us depending on the system.
> The required accuracy (Tables 1 and 2 in that document)
no, Tables 1 and 2 say divergence, not accuracy
accuracy is a mix of both granularity and divergence
regardless, your statement before:
> The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.
is not true
> accuracy is a mix of both granularity and divergence
I respectfully disagree.
In context, "granularity" is nothing more than a resolution constraint on reported timestamps. Its inclusion adjacent to the specified "divergence from UTC" is a function of market manipulation surveillance objectives as discussed in preamble item (2), and really doesn't have anything to do with accuracy proper.
> mifid ii (uk/eu) minimum is 1us granularity
1us is nothing special for GPS/NTP/PTP appliances (especially with OCXO/rubidium oscillators):
* https://www.microchip.com/en-us/products/clock-and-timing/sy...
* https://www.meinbergglobal.com/english/productinfo/gps-time-...
My guess would be scientific experiments where they need to correlate or sequence data over large regions. Things like correlating gravitational waves with radio signals and gamma ray bursts.
those are GPS based too. You typically would have a circuit you trained off off 1PPS and hopefully had a 10 or so satellites in view.
You can get 50ns with this. Of course, you would verify at NIST.
> ...and hopefully had a 10 or so satellites in view.
I believe you'll need 12 GPS sats in view to gain incremental accuracy improvement over 8.
GPS could be blocked easily, and AFAIK even given corrupted inputs. And HFT could possibly benefit from blocking or corrupting competitors GPS.
Deploying a GPS jammer in civilized territory is a great way to go to prison.
Would it actually go so far?
Would the police actually try to investigate from where came the jammer? Might the competing firm possibly even finance an investigation themselves privately? And if so, would the police then accept the evidence?
People have done far more evil things for money.
It's not like foreign adversaries care.
> I've never heard of this! Very cool service, presumably for … quant / HFT / finance firms (maybe for compliance with FINRA Rule 4590 [3])?
To start with, probably for scientific stuff, à la:
* https://en.wikipedia.org/wiki/White_Rabbit_Project
But fibre-based time is important in case of GNSS time signal loss:
* https://www.gpsworld.com/china-finishing-high-precision-grou...
> a commercial consumer
Where does it say these are commercial consumers?
https://en.wikipedia.org/wiki/Schriever_Space_Force_Base#Rol...
> Building 400 at Schriever SFB is the main control point for the Global Positioning System (GPS).
SIGINT as a source clock for others in a network doing super accurate TDOA for example.
But they do not need absolute time, and internal rubidium clocks can keep the required accuracy for a few days. After that, sync can be transferred with a portable plug, which is completely viable in tactical/operational level EW systems.
> Google/hyperscalers as input to Spanner or other global databases?
Think Google might have rolled their own clock sources and corrections.
Ex: Sundial, https://www.usenix.org/conference/osdi20/presentation/li-yul... / https://storage.googleapis.com/gweb-research2023-media/pubto... (pdf)
science equipment, distributed radio-telescopes where you need to precisely align data received at different locations
I think Google uses chrony instead of NTP
Google doesn't use chrony specifically, just an algorithm that is somewhat chrony-like (but very different in other ways). It's called Google TrueTime.
I'm sure all of that is true, but so is "Department of Defense".
They're also the largest holder of IPv4 space, still. https://bgp.he.net/report/peers#_ipv4addresses
Why does the DoD hold so many IPv4s?
Only Boulder servers lost sync.
To say NIST was off is clickbait hyperbole.
This page: https://tf.nist.gov/tf-cgi/servers.cgi shows that NIST has > 16 NTP servers on IPv4, of those, 5 are in Boulder and were affected by the power failure. The rest were fine.
However, most entities should not be using these top-level servers anyway, so this should have been a problem for exactly nobody.
IMHO, most applications should use pool.ntp.org
Who does use those top-level servers? Aren’t some of them propagating the error or are all secondary level servers configured to use dispersed top-level servers? And how do they decide who is right when they don’t match?
Is pool.ntp.org dispersed across possible interference and error correlation?
You can look at who the "Stratum 2" servers are, in the NTP.org pool and otherwise. Those are servers who sync from Stratum 1, like NIST.
Anyone can join the NTP.org pool so it's hard to make blanket statements about it. I believe there's some monitoring of servers in the pool but I don't know the details.
For example, Ubuntu systems point to their Stratum 2 timeservers by default, and I'd have to imagine that NIST is probably one of their upstreams.
An NTP server usually has multiple upstream sources and can steer its clock to minimize the error across multiple servers, as well as detecting misbehaving servers and reject them ("Falseticker"). Different NTP server implementations might do this a bit differently.
From my own experience managing large numbers of routers, and troubleshooting issues, I will never use pool.ntp.org again. I’ve seen unresponsive servers as well as incorrect time by hours or days. It’s pure luck to get a good result.
Instead I’ll stick to a major operator like Google/Microsoft/Apple, which have NTP systems designed to handle the scale of all the devices they sell, and are well maintained.
I believe if you use time.nist.gov it round robins dns requests, so there’s a chance you’d have connected to the Boulder server. So for some people they would have experienced NIST 5 μs off.
Nitpick: UTC stands for Coordinated Universal Time. The ordering of the letters was chosen to not match the English or the French names so neither language got preference.
Universal Time, Coordinated.
This is how I say it in my head.
It's also the time in Iceland, conveniently
That doesn't quite match what the wikipedia page says:
> The official abbreviation for Coordinated Universal Time is UTC. This abbreviation comes as a result of the International Telecommunication Union and the International Astronomical Union wanting to use the same abbreviation in all languages. The compromise that emerged was UTC, which conforms to the pattern for the abbreviations of the variants of Universal Time (UT0, UT1, UT2, UT1R, etc.).
Follow the citation: https://www.nist.gov/pml/time-and-frequency-division/how-utc...
> ... in English the abbreviation for coordinated universal time would be CUT, while in French the abbreviation for "temps universel coordonné" would be TUC. To avoid appearing to favor any particular language, the abbreviation UTC was selected.
That's an interesting rationale.
Reminds me of when a group is divided into two parts, dubbed group 1 and group A, such that neither feels secondary.
And if you have 3, group 1, group A, and group Alpha. Beyond that, just use colors.
It also stands for Universel Temps Coordonné.
It's le temps universel coordonné in French.
I know some HFT people who made a few hundred K off of this.
Not exactly the topic of discussion but also not not on topic: just wanted to sing praise for chrony which has performed better than the traditional os-native NTP clients in our testing on a myriad of real and virtualized hardware.
Chrony is the default already in some distros (RHEL and SLES that I know of), probably for this very reason.
Maybe I missed something, but I don't quite understand the video title "NIST's NTP clock was microseconds from disaster". Is there some limit of drift before it's unrecoverable? Can't they just pull the correct time from the other campus if it gets too far off?
I'll consider it clickbait...
... unless someone with real experience needing those tolerances chimes in and explains why it's true.
I'd have thought Jeff to be above clickbait, but here we are
The web article title doesn't mention any disaster. I don't see the embedded youtube video - probably have to many privacy options on.
I'm missing the nuance or perhaps the difference between the first scenario where sending inaccurate time was worse than sending no time, versus the present where they are sending inaccurate time. Sorry if it's obvious.
The 5us inaccuracy is basically irrelevant to NTP users, from the second update to the Internet Time Service mailing list[1]:
To put a deviation of a few microseconds in context, the NIST time scale usually performs about five thousand times better than this at the nanosecond scale by composing a special statistical average of many clocks. Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet; uncertainties on the order of 1 millisecond (one thousandth of one second) are more typical due to asymmetry and fluctuations in packet delay.
[1] https://groups.google.com/a/list.nist.gov/g/internet-time-se...
> Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet
How do those other applications obtain the precise value they need without encountering the Internet issue?
> How do those other applications obtain the precise value they need without encountering the Internet issue?
They do not use the Internet: they use local (GPS) clocks with internal high-precision clocks for carry-over in case GNSS signal is unavailable:
* https://www.ntp.org/support/vendorlinks/
* https://www.meinbergglobal.com/english/products/ntp-time-ser...
* https://syncworks.com/shop/syncserver-s650-rubidium-090-1520...
* https://telnetnetworks.ca/solutions/precision-time/
If those other applications use their own local GPS clocks, what is the significance of NIST (and the 5μs inaccuracy) in their scenario?
> If those other applications use their own local GPS clocks, what is the significance of NIST (and the 5μs inaccuracy) in their scenario?
Verification and traceability is one reason: it's all very well to claim you're with-in ±x seconds, but your logs may have to say how close you are to the 'legal reality' that is the official time of NIST.
NIST may also send out time via 'private fibre' for certain purposes:
* https://en.wikipedia.org/wiki/White_Rabbit_Project
'Fibre timing' is also important in case of GNSS signal disruption:
* https://www.gpsworld.com/china-finishing-high-precision-grou...
GPS gets its time from NIST (though during this incident they failed over to another NIST site, so it wasn't impacted).
That is not correct at all. How did you arrive at that conclusion?
GPS has its own independent timescale called GPS Time. GPS Time is generated and maintained by Atomic clocks onboard the GPS satellites (cesium and rubidium).
It has its own timescale, but that still traces back to NIST.
In particular, the atomic clocks on board the GPS satellites are not sufficient to maintain a time standard because of relativistic variations and Doppler effects, both of which can be corrected, but only if the exact orbit is known to within exceeding tight tolerances. Those orbital elements are created by reference to NIST. Essentially, the satellite motions are computed using inverse GPS and then we use normal GPS based on those values.
I think GP might’ve been referring to the part of Jeff’s post that references GPS, which I think may be a slight misunderstanding of the NIST email (saying “people using NIST + GPS for time transfer failed over to other sites” rather than “GPS failed over to another site”).
The GPS satellite clocks are steered to the US Naval Observatory’s UTC as opposed to NIST’s, and GPS fails over to the USNO’s Alternate Master Clock [0] in Colorado.
[0] https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-N...
TIL/remembered GNSS satellites have onboard atomic clocks. Makes a lot of sense, but still pretty cool. Something like this, I guess?
https://en.wikipedia.org/wiki/Rubidium_standard
Yes, either Rb, Cs, or H standards depending on which GNSS system you're using.
For the most critical applications, you can license a system like Fugro AtomiChron that provides enhanced GNSS timing down to the level of a few nanoseconds. There are a couple of products that do similar things, all based on providing better ephemerides than your receiver can obtain from the satellites themselves.
You can get AtomiChron as an optional subscription with the SparkPNT GPSDO, for instance (https://www.sparkfun.com/sparkpnt-gnss-disciplined-oscillato...).
> You can get AtomiChron as an optional subscription with the SparkPNT GPSDO, for instance (https://www.sparkfun.com/sparkpnt-gnss-disciplined-oscillato...).
That's one hell of a healthy profit margin there O.o
The SiTime MEMS oscillator is about 100€ for one single chip, the mosaic-T GPS receiver is about 400€. Add 50€ for the rest (particularly the power input section looks complicated) and 50€ for handling, probably 600€ in hardware cost... sold for 2.500€.
The real money I think went into certification and R&D for a low-volume product - even though most of the hard work is done by the two ICs, getting everything orchestrated (including the PCB itself) to perform to that level of accuracy is one hell of a workload.
A lot of organizations also colocate timing equipment near the actual clocks, and then have 'dark fiber' between their equipment and the main clock signals.
Then they disperse and use the time as needed.
According to jrronimo, they even had one place splice fiber direct between machines because couplers were causing problems! [1]
[1] https://news.ycombinator.com/item?id=46336755
If I put my machine near the main clock signal, I have one clock signal to read from. The comment above was asking about how to average across many different clocks, presumably all in different places in the globe? Unless there's one physical location with all of the ones you're averaging, you're close to one and far from all the others so how is it done without the internet?
If you must use the internet, PTP gets closer.
Alternate sources include the GPS signal, and the WWVB radio signal, which has a 60kHz carrier wave accurate to less than 1 part in 10^12.
Can you do PTP over the internet? I have only seen it in internal environments. GPS is probably the best solution for external users to get time signals with sub-µs uncertainties.
It's a good question, and I wondered the same. I don't know, but I'd postulate:
As it stands at the minute, the clocks are a mere 5 microseconds out and will slowly get better over time. This isn't even in the error measurement range and so they know it's not going to have a major effect on anything.
When the event started and they lost power and access to the site, they also lost their management access to the clocks as well. At this point they don't know how wrong the clocks are, or how more wrong they're going to get.
If someone restores power to the campus, the clocks are going to be online (all the switches and routers connecting them to the internet suddenly boot up), before they've had a chance to get admin control back. If something happened when they were offline and the clocks drifted significantly, then when they came online half the world might decide to believe them and suddenly step change to follow them. This could cause absolute havoc.
Potentially safer to scram something than have it come back online in an unknown state, especially if (lots of) other things are are going to react to it.
In the last NIST post, someone linked to The Time Rift of 2100: How We lost the Future --- and Gained the Past. It's a short story that highlights some of the dangers of fractured time in a world that uses high precision timing to let things talk to each other: https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...
> […] where sending inaccurate time was worse than sending no time […]
When you ask a question, it is sometimes better to not get an answer—and know you have not-gotten an answer—then to get the wrong answer. If you know that a 'bad' situation has arisen, you can start contingency measures to deal with it.
If you have a fire alarm: would you rather have it fail in such a way that it gives no answer, or fail in a way where it says "things are okay" even if it doesn't know?
Out of curiosity, can anyone say the most impactful things they've needed incredibly accurate time for?
I work at a particle accelerator. We use White Rabbit (https://white-rabbit.web.cern.ch/) to synchronize some very sensitive devices, mostly the RF power systems and related data acquisition systems, down to nanosecond accuracy.
Spanner
(See https://docs.cloud.google.com/spanner/docs/true-time-externa...)
Does it need to be this close to NIST, or just relative to each other? Because the latter one is solved by PTP.
As far as I remember, near each other, but the comment I was replying to was specifically about needing accuracy.
It's been over a decade now since I managed the truetime team at google, things may have changed since :)
Spacecraft state vectors.
As a very coarse number, 5µs is 1500 meters of radio travel.
If (and it isn't very conceivable) GPS satellites were to get 5µs out of whack, we would be back to Loran-C levels of accuracy for navigation.
Not sure but synthetic massive aperture radio telescope would need syncing their local clocks.
I defer to the experts.
As far as I'm aware they just timestamp the sample streams based on a local gps backed atomic reference. Then when they get the data/tapes in one computing center they can just run a more sophisticated correlation entirely in software to smooth things out.
GPS
Telling people at the bar that I have an atomic clock at home.
cesium or rubidium?
From NPR (22 points) https://news.ycombinator.com/item?id=46351105
Has anyone here ever needed microsecond precision? Would love to hear about it.
I worked at Altera (FPGA supplier) as the Ethernet IP apps engineer for Europe for a few years. All the big telecoms (Nokia, Ericsson, Cisco, etc) use Precision Time Protocol (PTP) in some capacity and all required clocks to be ns levels for accuracy. Sometimes as low a 10ns at the boundary. Any imperfection in the local clock directly converts into timestamp error, and timestamp error is what limits PTP synchronization performance. Timestamps are the fundamental observable in PTP. Quantization and jitter create irreducible timestamp noise. That noise directly limits offset and delay estimation. Errors accumulate across network elements and internal clock error must be much smaller than the system requirement.
I think most people would look at the error and think "what's the big deal" but at all the telecoms customers would be scrambling to find a clock that hasn't fallen out of sync.
We don't use NTP, but for robotics, stereo camera synchronization we often want the two frames to be within ~10us of eachother. For sensor fusion we then also need a lidar on PTP time to be translated to the same clock domain as cameras, for which we also need <~10us.
We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.
> We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.
This doesn't seem right to me. NTP with default settings should be monotonic. So no jumps. If you disable it Linux enters 11-minute mode, IIRC, and that may not be monotonic.
Pedantically, a monotonic function need not have a constant first derivative. To take it further, in mathematics it is accepted for a monatomic function to have a countable number of discontinuities, but of course in the context of a digital clock that only increments in discrete steps, that’s of little bearing.
But that’s all besides the point since most sane time sync clients (regardless of protocol) generally handle small deviations (i.e. normal cases) by speeding up or slowing down the system clock, not jumping it (forward or backward).
You are correct, NTP prefers to jump first (if needed) and then slew afterwards (which is exactly what we want!), although it can jump again if the offset is too large.
In our case the jumps where because we also have PTP disciplining the same system clock, when you have both PTP and NTP fighting over the same clock, you will see jumping with the default settings.
For us it was easier to just do a one time NTP sync at the beginning/boot, and then sync the robots local network with only PTP afterwards.
For a low precision environment to avoid sudden jumps I used SetSystemTimeAdjustment on Windows (now SetSystemTimeAdjustmentPrecise) to smoothly steer the system clock to match the GPS supplied time signal.
On Linux I think the adjtimex() system call does the equivalent https://manpages.ubuntu.com/manpages/trusty/man2/adjtimex.2....
It smears out time differences which is great for some situations and less ideal for others.
Also I implemented a simple PID controller that could automatically compensate against some types of internal clock drift.
Worked really well for the project.
Avoiding time jumps was really worthwhile.
In your stereo camera example, are these like USB webcams or something like MIPI CSI attached devices?
MIPI CSI, with USB cameras I usually am getting jitter in the millisecond range.
We use nanosecond precision driven by GPS clocks. That timestamp in conjunction with star tracker systems gives us reliable positioning information for orbital entities.
https://en.wikipedia.org/wiki/Star_tracker
(Assuming "precision" really meant "accuracy") The network equipment I work on requires sub microsecond time sync on the network for 5G providers and financial trading customers. Ideally they'd just get it from GPS direct, but that can be difficult to do for a rack full of servers. Most of the other PTP use cases I work with seem to be fine with multiples of microseconds, e.g. Audio/Video over the network or factory floor things like PLCs tend to be find with a few us over the network.
Perhaps a bit more boring than one might assume :).
Lightning detection. You have a couple of ground stations with known positions that wait for certain electromagnetic puses, and which record the timestamps of such pulses. With enough stations you can triangulate the location of the source of each pulse. Also a great way to detect nuclear detonations.
There is a german club that builds and distrubutes such stations (using GPS for location and timing), with a quite impressive global coverage by now:
https://www.blitzortung.org
At a previous role, we needed nanosecond precision for a simulcast radio communications system. This was to allow for wider transmission for public safety radio systems without having to configure trunking. We could even adjust the delay in nanoseconds to move the deadzones away from inhabited areas.
We solved this by having GPS clocks at each tower as well as having the app servers NTP with each other. The latter burned me once due to some very dumb ARP stuff, but that's a story for another day.
We need nanosecond precision for trading - basically timestamping exchange/own/other events and to measure latency.
You probably want to ask about accuracy. Any random microcontroller from the 90s needs microsecond precision.
Your speakers do so that people's voices match their mouth movements. The speaker clocks need to be in-sync with the cpu clocks and they operate at different frequencies.
High speed finance is msec and below. Fastest publically known tick to trade is just shy of 14 nanos.
Timekeeping starts to become really hard, often requiring specialized hardware and protocols.
How do you even get usable microsecond precision sync info from a server thousands of kilometers away? The latency is variable so the information you get can't be verified / will be stale the moment it arrives. I'm quite ignorant on the topic.
The high frequency trading guys
edit: also the linked slides in TFA
Yes, but always got it from GPS so presumably they'd be off about the same amount.
Distributed sonar, allows placing receivers willy-nilly and aligning the samples later.
Remote microphone switching - though for this you wouldn't notice 5us jitter, it's just that the system we designed happened to have granularity that good.
When we collected, correlated, and measured all controlling messages in a whole 4G network. Millisecond precision meant guaranteed out of order message flows.
Nuclear measurements, where the speed of a gamma ray flying across a room vs a neutron is relevant. But that requires at least nanosecond time resolution, and you’re a long way from thinking about NTP.
If you want sample accurate audio transmission you're going to want resolution on the order of 10s of microseconds.
I believe LTE and 5G networks require it to coordinate timeslots between overlapping cells. Of course, they can use whatever reference they want, as long as all the cells are using the same one - it doesn't have to be UTC. Some (parts of) networks transmit it across the network, while others have independent GPS receivers at each cell site.
Synchronization is also required for SDH networks. Don't know if those are still used.
Someone else referenced low power ham radio modes like WSPR, which I also don't know much about, but I can imagine they have timeslots linked to UTC and require accuracy. Those modes have extremely low data rates and narrow bandwidths, requiring accurate synchronization. I don't know if they're designed to self-synchronize, or need an external reference.
When multiple transmitters are transmitting the same radio signal (e.g. TV) they might need to be synchronized to a certain phase relationship. Again, don't know much about it
WSPR doesn't require tight synchronization, but it does require pretty stable frequency sources over periods of 10s of seconds.
It is very common to integrate a GPS in a WSPR beacon to discipline the transmit frequency, but with modest thermal management, very ordinary crystal oscillators have very nice stability.
A database like Google Spanner has higher latency in proportion to the uncertainty about the time. Driving the time uncertainty down into the microsecond range, or lower, keeps latency low.
Telecom.
Precision Time Protocol gets you sub-microsecond.
https://en.wikipedia.org/wiki/Precision_Time_Protocol
Lots of things do. Shoot, even plain old TDM needs timing precision on the order of picoseconds to nanoseconds.
But does that require accurate UTC?
If you have a widely distributed system and you need tight synchronization, it can be much easier to regulate to an outside stable source than to try to make a stable consensus algorithm.
I mean, we routinely benchmark things that take microseconds or less. I've seen a 300 picosecond microbenchmark (single cycle at 3GHz). No requirement that absolute time is correct, though.
Are there any plans being made to prevent this happening in the future?
Yes, the US government is banning all those democrat windmills that conspired to blow over the NTP server.
It would greatly decrease electricity prices too. God knows how expensive it is to power those gigantic spinning things
I laughed audibly at that!
??? The power outage was voluntary and surrounding towns chose not to turn off power. They could absolutely make infrastructure changes, and I’m sure the backups for power could make changes too.
Now I'm curious... How the hell do you synchronize clocks to such an extreme accuracy? Anybody have a good resource before I try to find one myself?
Look up PTP White Rabbit.
Thank you!
More discussion:
NTP at NIST Boulder Has Lost Power
https://news.ycombinator.com/item?id=46334299
Gah, just when you think you can trust time.nist.gov
Suggestions from the community for more reliable alternatives?
> Gah, just when you think you can trust time.nist.gov
You still can...
If you're that considered about 5 microseconds: Build your own Stratum 1 time server https://github.com/geerlingguy/time-pi
or just use ntppool https://www.ntppool.org/en/
It sounds like GPS, and thus a GPS-based stratum 1 server, uses these time servers, but they were successfully failed over:
> Jeff finished off the email mentioning the US GPS system failed over successfully to the WWV-Ft. Collins campus. So again, for almost everyone, there was zero issue, and the redundancy designed into the system worked like it's supposed to.
So failures in these systems are potentially correlated.
The author mentions another solution. Apparently he runs his own atomic clock. I didn’t know this was a thing an individual could do.
> But even with multiple time sources, some places need more. I have two Rubidium atomic clocks in my studio, including the one inside a fancy GPS Disciplined Oscillator (GPSDO). That's good for holdover. Even if someone were jamming my signal, or my GPS antenna broke, I could keep my time accurate to nanoseconds for a while, and milliseconds for months. That'd be good enough for me.
The CSACs that I have in a couple devices are 'atomic', and use Rubidium, but they're a bit lower accuracy than Cesium clocks [1] or Hydrogen Masers [2].
There are a few folks on the time-nuts mailing list who own such exotic pieces of hardware, but those are pretty far out of reach for most!
[1] https://www.microchip.com/en-us/products/clock-and-timing/co...
[2] https://www.microchip.com/en-us/products/clock-and-timing/co...
Atomic clocks cover a pretty big range of performance nowadays. You can pick up a used but serviceable rubidium frequency reference for a few hundred dollars but the difference between it and the top of the line clocks is almost as big as the difference between a it and a good pendulum clock.
Be aware that there are members of the NTP pool with less-than-honorable intentions and you don't get to pick-and-choose. Yes, they all should provide the time, but they also get your IP address.
For example: unlike the IPv4 space, the IPv6 space is too big too scan, so a number of "researchers" (if you want to call them that) put v6-capable NTP servers in the NTP pool to gather information about active v6 blocks to scan/target.
Do you have any acticles or references about this? That would be great research (pun intended) to find out
Is this one of those extraordinary claims that requires evidence? Or is it generally true that there are homey-pots in many of these services (NTP, mirrors, etc)
Most places that need accurate time get it from GPS. That is 10-100 ns.
Also, you can use multiple NIST servers. They have ones in Fort Collins, CO and Gaithersburg, MD. Most places shouldn't use NIST directly but Stratum 1 name servers.
Finally, NTP isn't accurate enough, 10-100 ms, for microsecond error to matter.
their handling it responsibly seems like more evidence for trusting them, not less?
Yes.
Use NTP with ≥4 diverse time sources, just as RFC 5905 suggests doing. And use GPS.
(If you're reliant upon only one source of a thing, and that thing is important to you in some valuable way, then you're doing it wrong. In other words: Backups, backups, backups.)
Use the other servers as well: https://tf.nist.gov/tf-cgi/servers.cgi
For instance, time-a-wwv.nist.gov.
One should configure a number of different NTP sources instead of just a single host.
I'm more concerned about what you think they did to earn your trust in the first place