> And what’s worse, malicious threat actors can manipulate time synchronization protocols in many cases to trigger this vulnerability at the time of their choosing.
If you switch to 64 bit timestamps, and the network protocol supports dates > 2038, can you then just trigger the rollover bugs by pretending it's 2*64 - 1 seconds after epoch start?
Also, if the actions are potentially so severe, and NTP (or whatever is used) so vulnerable, why haven't we seen many such attacks in the wild?
Update: to be clear I'm not arguing that there isn't a problem, I've already run into it myself. I'm trying to understand how severe it is, how exploitable, and how robust a solution could be.
I fixed a recent Y38 bug in some classic ASP code. The bug was nothing more than a simple `Date() + 5000` computation (adding 5000 days to the current date) as a sort of expiry date applied to something; I don't recall the exact details. VB6 did not take kindly to computing any date value beyond the Y38 max and threw an error. In practice this error ended up denying service to everyone even though the Y38 max date was 14 years in the future. You never know what little bugs like that are lurking in such legacy code.
I think this effort would benefit from trying to qualify what “unpredictable ways” actually means. If anyone is testing devices, a catalog of test results describing the actual failure modes that were revealed would help make this whole thing more concrete.
I think many software engineers know that if you want to make any organization care about this type of issue, you need to be ready to demonstrate the severity and impact.
Even if the system boots properly, there's various critical systems that depend upon having the correct time. Say goodbye to things like HTTPS and SSL/TLS certificates.
Will the root-certificates still be trusted in 12 years? Will we largely use the same TLS versions? And if systems can be updated to account for that, shouldn't they also be able to be updated to deal with the timestamps limitation?
For comparison I've revived a decade old Axis PTZ camera recently and it can't be used with HTTPS because it only supports TLS 1.0 which is deprecated across the board these days lmao. The UI is so bugged out it's not possible to change the default username and password anymore.
There's two kinds of internet connected devices these days, those that keep getting updated and those that drift into incompatibility and die as the rest of the ecosystem evolves around them. If these supposed critical devices will still be in use in 12 years without any maintenance then they're unlikely to have any actual importance.
It means 'anywhere between being bricked and no problem at all, and we can't give you any idea of how severe or how likely any of those possiblities are'. The only way you can really know is to thorougly audit your system and/or test it. Preferably both.
Y2K showed that you don't need details beyond vague threats of "medication administered at the wrong time" and "planes falling out of the air" to get organizations and the public to care. No idea how that's going to tie into the conspiracy-heavy media landscape we inhabit now.
(Note I do think this is a serious issue that needs to be addressed. And I'd love to see specific examples. I'm just pushing back against the idea that examples would make much difference to advocacy efforts)
“Planes falling pit of the sky” still gets used both as an example of overblown Y2k fear-mongering AND the reason why all those quiet preparations were necessary.
What? Y2K did have many demonstrable problems... Having a 2 digit year did obviously cause problems. The reason nothing happened is because a shit ton of time and money was spent making sure it didn't.
It was pretty easy for orgs with affected systems to produce concrete examples for themselves. Maybe to everyone else it seemed vague, but for the people who had to deal with it, it was taken pretty seriously from top to bottom.
It was thankless work that is still glossed over and waved away today, but it was all a very big deal throughout the late 90s.
All I can say is that at my level in an org, if I want to say “instead of developing X this quarter, I want to test the effects of 2038 on Y,” that’s a far easier conversation if I can say something like “in similar embedded devices they crashed and wouldn’t even respond to OTA updates” vs “something bad could happen. Not sure.”
The latter is just a ripe plumb, left to rot in the backlog.
That's nonsense. Orgs spent time and resources on it because they grabbed a test server and demonstrated it caused problems. It's not some weird ethereal untestable bug. They set the dev server to a minute before midnight and went "oh shit".
That and quite often the problems started showing up years before 2000 itself. "Hey, the scheduler is giving me a meeting 80 years ago" type weirdness when it crossed the boundry.
Telling home users to check that their existing smart devices will still work in 13 years seems like overkill. It seems unlikely that more than a tiny fraction of them will still be in use then, if any.
Businesses installing new smart infrastructure and devices will need to pay attention to this, and in 10-15 years they'll need to work out what to replace, of course.
Agreed. A serious approach to this problem would be: Identify critical computers which are currently 13+ years old (most likely embedded systems). Assume that the same sorts of systems will be 13+ years old in 2038. Focus on raising awareness with that particular target audience, e.g. give talks about the 2038 problem at embedded systems conferences. Try to get it included in university curricula. Etc.
Goes back at least slightly before that, as I've had 2038epochalypse.com registered since March 2017, but I can't recall whether I thought I was being clever or whether I heard it somewhere else.
Are they doing anything to fix it or just raising awareness?
Here's an example of measuring packages that report warnings for software that has suspicious conversions. Compile with `-Wconversion` with both 32-bit and 64-bit time_t, and see what the difference is.
https://github.com/mkj/yocto-y2038
That is using yocto, but you could probably do something similar with other less-embedded distros too, if you can rebuild the world.
FWIW I didn't find much interesting with that apart from busybox dhcpd.
It looks mostly like a project for self promotion of the two authors. Maybe they offer some consulting services.
Funniest is that one of them wrote that they have "learned about it after Y2K bug".
I thought one learns about this overflow in a "introduction to programming" class...
It also says nothing about a formal education, just that he has worked in IT since his teens. I didn't hear of the 2038 problem myself until the whole Y2K debacle, but then, I was in my teens at the time.
MAX_SAFE_INTEGER = 2^53 - 1 = 9007199254740991. The next integer 2^53 is representable, while the following one 2^53 + 1 isn't.
2^53 + 1 milliseconds = 9007199254740993 milliseconds = 104249991 days 8 hours 59 minutes 0.993 seconds. Since the zero point is midnight 1970-01-01, we reach the first non-representable millisecond at 287396-10-12T8:59:00.993 (be careful to use the Gregorian leap rule if you want to calculate this manually).
Fwiw, it's much worse for 64-bit nanoseconds, running out in the year 2262 for signed, and the year 2554 for unsigned. As for who needs nanoseconds, trading systems, particle physics, and globally distributed systems do.
Which systems use a single variable for nanoseconds time? The APIs I know of all use a struct with two members, one for seconds and the other for the nanoseconds within the second.
At least ext4 (and its predecessors ext2/ext3) uses 32-bit seconds for the timestamps, and has separate fields (available only when the inode size of the filesystem is greater than 128 bytes) for nanoseconds and for extending the seconds fields to 34 bits.
I witnessed my first y38 bug a couple weeks ago - just as a matter of someone doing forecasting in a system with an incorrect casting.
At least, the good part is that people will get "hunches" about y38, just like you start getting "hunches" about bugs related to locales, time zones, character encodings, currency roundings, etc...
I don't know if there are courses, books, etc... about all those matters that are definitely non "computer sciency", but occupy so much of our engineering time ?
15 years ago i was working at a startup in SV and a kid we hired was saying how he was sad he missed out on y2k because he was too young. I filled him with joy when I mentioned the 2038 bug. lol
Patched in OpenBSD 5.5 [1] which was released in 2014. Patches were upstreamed into software in the ports tree, so hopefully that made life a little bit easier for other operating systems. Linux seems to have followed around 2020.
I don't know about "largely", but I can point to a least one instance where the most week know comedy show at the time in France ("Les guignols") had a bit about how the Y2K bug was a "scam" orchestrated by "Big Tech" to make people buy new computers - given that nothing terrible happened on Y2K.
The sketch was a riff about the kindergarten joke :
- why do you [insert weird action] ?
- to scare the girafes away
- but there are no girafes here
- of course, I've been [insert weird action here]
And I was a software engineering student at the time, and I tried explaining they were missing the point, but I think it will be how we're approached this time.
So, I think nothing will be done, or not enough, and bugs will happen in 12 years.
But, thankfully, all our computers will have been replaced to allow running js.
Well, by 2038 the chatbots will have long since fixed all the time format problems, all our problems, and will be working away at problems of which we cannot concieve.
So, ahem, your “early retirement” is assured?
A year is a lot longer than it used to be. And quickly getting longer. Measured in change.
Except "those people" are the ones who will run the USA for the foreseeable future.
Fixing Y38 will require some public spending. If "researching cancer" is not considered worthy of public spending any more, I'm curious about how the nerds will manage to justify replacing lots of embedded chips "because maths".
If it takes the usual 8 years to replace the current administration with one that accepts listening to experts, and unless big donors can make a profit by organizing the transition, we can expect serious efforts to only start around 2032 globally. No idea if that will be enough this time, we'll see...
(On the bright side, maybe they can charge customers ? That would work.)
Even though it might seem unlikely, you absolutely can't eliminate the possibility of planes falling out of the air unless you have thoroughly audited and tested all their systems in advance. Not having planes in the air at the exact moment of rollover - just in case - might be a good idea too. Similarly financial markets and other key infrastructure.
13 years is plenty of time to make a start on all of this, and will allow the costs to be spread over that time period without undue hurry. It's also long enough that it will be possible for much software to be allowed to reach end-of-life without being audited, while applying rigorous testing to new software. Procrastination, though, is not a good option.
Why should floppy disks be a problem? Even if they are fantastically out of date, if they can be maintained as a niche legacy technology, they're not a problem. They won't suddenly stop working, it will just slowly get more and more expensive to get the parts. Supply chains are magic - at the right price, someone will always be willing to make more floppies or drives. Of course that price might be $500 per floppy and $50,000 per drive...
> And what’s worse, malicious threat actors can manipulate time synchronization protocols in many cases to trigger this vulnerability at the time of their choosing.
If you switch to 64 bit timestamps, and the network protocol supports dates > 2038, can you then just trigger the rollover bugs by pretending it's 2*64 - 1 seconds after epoch start?
Also, if the actions are potentially so severe, and NTP (or whatever is used) so vulnerable, why haven't we seen many such attacks in the wild?
Update: to be clear I'm not arguing that there isn't a problem, I've already run into it myself. I'm trying to understand how severe it is, how exploitable, and how robust a solution could be.
I fixed a recent Y38 bug in some classic ASP code. The bug was nothing more than a simple `Date() + 5000` computation (adding 5000 days to the current date) as a sort of expiry date applied to something; I don't recall the exact details. VB6 did not take kindly to computing any date value beyond the Y38 max and threw an error. In practice this error ended up denying service to everyone even though the Y38 max date was 14 years in the future. You never know what little bugs like that are lurking in such legacy code.
I feel like the hard part is flashing the new firmware
I think this effort would benefit from trying to qualify what “unpredictable ways” actually means. If anyone is testing devices, a catalog of test results describing the actual failure modes that were revealed would help make this whole thing more concrete.
I think many software engineers know that if you want to make any organization care about this type of issue, you need to be ready to demonstrate the severity and impact.
Worst case scenario, it bricks your device: https://issuetracker.google.com/issues/36928638?pli=1
Even if the system boots properly, there's various critical systems that depend upon having the correct time. Say goodbye to things like HTTPS and SSL/TLS certificates.
Will the root-certificates still be trusted in 12 years? Will we largely use the same TLS versions? And if systems can be updated to account for that, shouldn't they also be able to be updated to deal with the timestamps limitation?
For comparison I've revived a decade old Axis PTZ camera recently and it can't be used with HTTPS because it only supports TLS 1.0 which is deprecated across the board these days lmao. The UI is so bugged out it's not possible to change the default username and password anymore.
There's two kinds of internet connected devices these days, those that keep getting updated and those that drift into incompatibility and die as the rest of the ecosystem evolves around them. If these supposed critical devices will still be in use in 12 years without any maintenance then they're unlikely to have any actual importance.
For those in a similar situation, <http://frogfind.com/> is useful.
It means 'anywhere between being bricked and no problem at all, and we can't give you any idea of how severe or how likely any of those possiblities are'. The only way you can really know is to thorougly audit your system and/or test it. Preferably both.
So that means the best tool is a 2038 test environment - which people then install their application(s) and test it e2e to see what the impacts are.
However, I’m not sure how you make a 2038 test environment
It assumes that the OS/Kernel etc… are defacto frozen to 2025 or whatever increment until 2038
What was the y2k solution for the people that implemented those fixes in the 90s?
libfaketime is useful for testing these kinds of things.
It intercepts system calls to get the time and reports a fake time to the application.
https://github.com/wolfcw/libfaketime
Demonstrate? Or just scaremonger?
Y2K showed that you don't need details beyond vague threats of "medication administered at the wrong time" and "planes falling out of the air" to get organizations and the public to care. No idea how that's going to tie into the conspiracy-heavy media landscape we inhabit now.
(Note I do think this is a serious issue that needs to be addressed. And I'd love to see specific examples. I'm just pushing back against the idea that examples would make much difference to advocacy efforts)
“Planes falling pit of the sky” still gets used both as an example of overblown Y2k fear-mongering AND the reason why all those quiet preparations were necessary.
Is there any plausible mechanism for Y2K bugs to cause planes to fall from the sky?
Why not? From five years ago, on the most modern type:
https://www.theregister.com/2020/04/02/boeing_787_power_cycl...
Seems to be unrelated at first look, but considering the complexities of systems, and cascading failure modes?
What? Y2K did have many demonstrable problems... Having a 2 digit year did obviously cause problems. The reason nothing happened is because a shit ton of time and money was spent making sure it didn't.
Agreed. My point is that the orgs paying for all these updates were mostly motivated by the vague claims of experts rather than concrete examples
It was pretty easy for orgs with affected systems to produce concrete examples for themselves. Maybe to everyone else it seemed vague, but for the people who had to deal with it, it was taken pretty seriously from top to bottom.
It was thankless work that is still glossed over and waved away today, but it was all a very big deal throughout the late 90s.
> It was thankless work that is still glossed over and waved away today, but it was all a very big deal throughout the late 90s.
Mike Judge even made a movie about it! Office Space might be the most recognition turn of the millennium programmers will receive.
All I can say is that at my level in an org, if I want to say “instead of developing X this quarter, I want to test the effects of 2038 on Y,” that’s a far easier conversation if I can say something like “in similar embedded devices they crashed and wouldn’t even respond to OTA updates” vs “something bad could happen. Not sure.”
The latter is just a ripe plumb, left to rot in the backlog.
That's nonsense. Orgs spent time and resources on it because they grabbed a test server and demonstrated it caused problems. It's not some weird ethereal untestable bug. They set the dev server to a minute before midnight and went "oh shit".
That and quite often the problems started showing up years before 2000 itself. "Hey, the scheduler is giving me a meeting 80 years ago" type weirdness when it crossed the boundry.
Telling home users to check that their existing smart devices will still work in 13 years seems like overkill. It seems unlikely that more than a tiny fraction of them will still be in use then, if any.
Businesses installing new smart infrastructure and devices will need to pay attention to this, and in 10-15 years they'll need to work out what to replace, of course.
Agreed. A serious approach to this problem would be: Identify critical computers which are currently 13+ years old (most likely embedded systems). Assume that the same sorts of systems will be 13+ years old in 2038. Focus on raising awareness with that particular target audience, e.g. give talks about the 2038 problem at embedded systems conferences. Try to get it included in university curricula. Etc.
"Epochalypse" is a nice coinage. Not a new one either, this 2017 article attributes it to Mikko Hypponen (of F-Secure). https://www.tomsguide.com/us/2038-bug-bh2017,news-25551.html
Goes back at least slightly before that, as I've had 2038epochalypse.com registered since March 2017, but I can't recall whether I thought I was being clever or whether I heard it somewhere else.
Heh, I've got five domains of a similar name too. Going to reach out to the owners of the website this post is referring to and see if they want them.
I've had my "2038 consulting" sites since Feb 2011, but someone got epochalypse dot com registered August 2007.
Who will register a domain for the upcoming year 292271025015 problem?
Y292B271M25K15 is less catchy than Y2K
Are they doing anything to fix it or just raising awareness?
Here's an example of measuring packages that report warnings for software that has suspicious conversions. Compile with `-Wconversion` with both 32-bit and 64-bit time_t, and see what the difference is. https://github.com/mkj/yocto-y2038
That is using yocto, but you could probably do something similar with other less-embedded distros too, if you can rebuild the world.
FWIW I didn't find much interesting with that apart from busybox dhcpd.
btw yocto has been using 64 bit time_t since last year: https://docs.yoctoproject.org/dev/migration-guides/migration...
It looks mostly like a project for self promotion of the two authors. Maybe they offer some consulting services.
Funniest is that one of them wrote that they have "learned about it after Y2K bug". I thought one learns about this overflow in a "introduction to programming" class...
It also says nothing about a formal education, just that he has worked in IT since his teens. I didn't hear of the 2038 problem myself until the whole Y2K debacle, but then, I was in my teens at the time.
When will milliseconds since the Unix epoch not be precisely representable in IEEE double anymore? (As it's used in JavaScript.)
MAX_SAFE_INTEGER = 2^53 - 1 = 9007199254740991. The next integer 2^53 is representable, while the following one 2^53 + 1 isn't.
2^53 + 1 milliseconds = 9007199254740993 milliseconds = 104249991 days 8 hours 59 minutes 0.993 seconds. Since the zero point is midnight 1970-01-01, we reach the first non-representable millisecond at 287396-10-12T8:59:00.993 (be careful to use the Gregorian leap rule if you want to calculate this manually).
Fwiw, it's much worse for 64-bit nanoseconds, running out in the year 2262 for signed, and the year 2554 for unsigned. As for who needs nanoseconds, trading systems, particle physics, and globally distributed systems do.
Which systems use a single variable for nanoseconds time? The APIs I know of all use a struct with two members, one for seconds and the other for the nanoseconds within the second.
It's a pretty common format for timestamps on modern file systems used by Unixy operating systems (e.g. ext4, XFS, APFS, ZFS, btrfs, etc.).
At least ext4 (and its predecessors ext2/ext3) uses 32-bit seconds for the timestamps, and has separate fields (available only when the inode size of the filesystem is greater than 128 bytes) for nanoseconds and for extending the seconds fields to 34 bits.
In the year 287390 (285420.9 years after 1970) according to my calculations using DuckDuckGo’s calculator.
If you ask ChatGPT it’s:
Why would you ask ChatGPT?
AI is the next big thing, I hear.
That seems a bit small seeing how doubles can represent 53 bit integers accurately.
Well, 2^53 milliseconds is indeed 285 414 years.
I witnessed my first y38 bug a couple weeks ago - just as a matter of someone doing forecasting in a system with an incorrect casting.
At least, the good part is that people will get "hunches" about y38, just like you start getting "hunches" about bugs related to locales, time zones, character encodings, currency roundings, etc...
I don't know if there are courses, books, etc... about all those matters that are definitely non "computer sciency", but occupy so much of our engineering time ?
I've been planning for decades to have a party on that day.
Only 13 years left!
Wait, in my invitation it says the party already happened in 1901.
Is it too early to raise awareness for the Y10k bug?
15 years ago i was working at a startup in SV and a kid we hired was saying how he was sad he missed out on y2k because he was too young. I filled him with joy when I mentioned the 2038 bug. lol
I worry that for a lot of devices (IoT, etc.), the fix will be "throw it away and buy a new one"
Well, thats already the "fix" for the current problems those devices face.
Patched in OpenBSD 5.5 [1] which was released in 2014. Patches were upstreamed into software in the ports tree, so hopefully that made life a little bit easier for other operating systems. Linux seems to have followed around 2020.
[1]: https://www.openbsd.org/lyrics.html#55
Such a shame that nobody uses it.
Epochalypse is clever. I'd been calling it Y2K38 so it made sense to people outside the industry.
The fact that it's not common knowledge and not taught in schools and universities is concerning.
Is it? I was starting my career during Y2K. It was widely regarded as a bust, and I’d anticipate people expecting the same damned thing?
It was largely regarded as a success. Why do you say a bust?
Sorry, I meant from the perspective of people expecting chaos to happen. I agree it was wildly successful in that basically nothing happened!
I remember some people looking forward to the y2k chaos. There was a smug sense of “look at how badly those brainiac computer guys screwed up”
I remember a rumor that “if enough people take their phone off the hook before midnight it’ll take down the phone network”
I don't know about "largely", but I can point to a least one instance where the most week know comedy show at the time in France ("Les guignols") had a bit about how the Y2K bug was a "scam" orchestrated by "Big Tech" to make people buy new computers - given that nothing terrible happened on Y2K.
The sketch was a riff about the kindergarten joke :
- why do you [insert weird action] ?
- to scare the girafes away
- but there are no girafes here
- of course, I've been [insert weird action here]
And I was a software engineering student at the time, and I tried explaining they were missing the point, but I think it will be how we're approached this time.
So, I think nothing will be done, or not enough, and bugs will happen in 12 years.
But, thankfully, all our computers will have been replaced to allow running js.
"What am I paying my IT department for? My computers never have problems! It's a scam, I tells ya!"
my capitalist side remembers that many people made retirement money because of Y2K.
Maybe I should start thinking about an early retirement :)
Well, by 2038 the chatbots will have long since fixed all the time format problems, all our problems, and will be working away at problems of which we cannot concieve.
So, ahem, your “early retirement” is assured?
A year is a lot longer than it used to be. And quickly getting longer. Measured in change.
How exactly? asking for a friend
"You still know php4? We need someone to fix this 50000 loc file that was written in 2005. Willing to pay big money."
Wouldn't it just be a matter of recompiling as 64-bit? No need to mess with the 50000 LOC.
I would much prefer we put this off till the last minute so that I have one more job available to me before retirement. Thanks.
Eh, I've got 13 years to deal with this. No big deal.
The best thing about waiting until the last minute to fix something is it will only take a minute.
Born too early for Y2K, but just in time for the Epochalypse lol
Well we don’t know if it will actually be an e/apocalypse. Which sounds very scary and doomsy to me.
But it will be the end of time, as we know it!
yeah sure.
My my
(Not) Looking forward to 2038 onwards as people ignore the fact we coordinated to fix this and start saying it was all a big hoax.
Same thing happened with Y2K. You can’t let such people bother you, or you’ll be miserable.
Except "those people" are the ones who will run the USA for the foreseeable future.
Fixing Y38 will require some public spending. If "researching cancer" is not considered worthy of public spending any more, I'm curious about how the nerds will manage to justify replacing lots of embedded chips "because maths".
If it takes the usual 8 years to replace the current administration with one that accepts listening to experts, and unless big donors can make a profit by organizing the transition, we can expect serious efforts to only start around 2032 globally. No idea if that will be enough this time, we'll see...
(On the bright side, maybe they can charge customers ? That would work.)
Will people use planes falling out of the sky as the reason the fix must be priority 1 this time?
Even though it might seem unlikely, you absolutely can't eliminate the possibility of planes falling out of the air unless you have thoroughly audited and tested all their systems in advance. Not having planes in the air at the exact moment of rollover - just in case - might be a good idea too. Similarly financial markets and other key infrastructure.
13 years is plenty of time to make a start on all of this, and will allow the costs to be spread over that time period without undue hurry. It's also long enough that it will be possible for much software to be allowed to reach end-of-life without being audited, while applying rigorous testing to new software. Procrastination, though, is not a good option.
Planes will still be using floppy disks. Or they will have upgraded to the worst brand of Android, as usual.
Why should floppy disks be a problem? Even if they are fantastically out of date, if they can be maintained as a niche legacy technology, they're not a problem. They won't suddenly stop working, it will just slowly get more and more expensive to get the parts. Supply chains are magic - at the right price, someone will always be willing to make more floppies or drives. Of course that price might be $500 per floppy and $50,000 per drive...
It’s Y2K all over again. Signed? RUFKM?
In 1980, it was probably more important to be able to represent 1960 than 2040, so to me it makes sense they were signed.