Home » Security Bloggers Network » Preparing for Y2038 (Already?!)

Preparing for Y2038 (Already?!)

by Erik Nygren on January 10, 2019

It somehow doesn’t seem that long ago, but nineteen years ago during Y2K I spent my New Year’s Eve in the Akamai Network Operations center, waiting to respond to anything that might go awry as the clock struck midnight in key time zones such as Greenwich and Boston. As of January 9, 2019, we are roughly half-way from Y2K to “Y2038”, the next large time epoch roll-over event. In 2038 on January 19^th the “Unix Epoch Time” will exceed the size of a signed 32-bit integer “time_t” value (2³¹-1), or roughly 2.1 billion seconds since the epoch of 00:00:00 on January 1, 1970. We have somewhat more time to deal with the systems that will break nineteen years from now. However, as we get closer there will be increasing impacts on software working with future dates.

Shortly after Y2K we made jokes about “next up, Y2038!” but back then it felt an eternity in the future and likely to be someone else’s problem. Now that we’re halfway there, and we have already reached the point where Y2038 issues can cause software failures, it is a good opportunity to start planning for Y2038. For example, software may already be having issues working with 20-year certificate lifetimes or 20-year mortgages, and the frequency of these issues will only increase as we get closer to Y2038. At Akamai we are already running strategically-targeted internal planning and testing for Y2038, and it seems likely that the scope of this work will continue to grow over the next 19 years as this important effort increases in urgency.

Very little went wrong on January 1, 2000 for us (short of some Javascript on an Akamai marketing site displaying “19100” as the current date!), but one thing that gets missed is that the limited global impact that evening was due to two factors: 1) the amount of advanced preparation that was done; 2) many “Y2K problems” actually occurred years in advance rather than during the roll-over itself. Leap seconds are in some ways scarier than date-format issues in that they are harder to test for and much less of the impact happens in advance. There is a risk that the lack of impacts of Y2K may cause organizations and technologists to under-prepare for Y2038. It is also harder to explain “Y2038” to lay people than Y2K, potentially making it harder to prioritize and focus advanced work. The large number of embedded Internet of Things (IoT) devices becoming ubiquitous in our environment also makes the likely risk and potential impact considerably higher for Y2038 than it was for Y2K.

Many years ago I heard a perhaps apocryphal anecdote about an early Y2K production impact. A warehouse had two automated jobs: one which looked for pallets of expired goods and sent them for disposal, and a second one that looked for low inventory and ordered more of a product. Canned tomatoes were the first product to have expiration dates start crossing into the year 2000, and a Y2K bug directed a forklift operator to dispose of those tomatoes (expiring in 1900!) as they arrived. The system then identified that there was low inventory on canned tomatoes and would order more. A few weeks later they would arrive, and a few days later the forklift operator would be dispatched to dispose of them. This cycle continued until the forklift operator finally noticed something awry and escalated the problem. It is likely that some of the first Y2038 issues will be quite similar in nature.

Our initial experience with Y2038 planning (besides seeing shock from people upon hearing concerns being raised that at first sound 19 years away) is that an incremental and focused approach is needed at this stage. Much more involved and comprehensive programs will certainly be needed some number of years down the road. In this initial phase, some areas to focus on include: 1) software dealing with future times and dates; 2) on-the-wire message and file formats; 3) devices with long deployed lifetimes, and their dependencies. Of course as we get closer it will become critical to start focusing on broader sets of systems, including ensuring that they can handle crossing the Y2038 transition safely.

Software dealing with future dates

Perhaps the most important area to focus on initially is software that deals with dates in the future, such as for handling X.509 certificates or for financial planning. There are many date and time format representations, not all of which will have a Y2038 problem. For example, software needing to deal with times in past centuries (well before 1970) often picked different date and time representations. However, in testing the cases of X.509 certificates (such as used for HTTPS) and certificate authorities (CAs) we have found numerous software issues in the lab that start to arise with certificates and CAs expiring past Y2038.

Sponsorships Available

OpenSSL in particular has multiple time formats, with ASN1_UTCTIME having a limit in Y2049 (a distinct issue to plan for from Y2038), so use the ASN1_TIME functions provide compatibility with all ranges of time. Converting times out from a library such as OpenSSL out into 32-bit time_t is an additional area likely to have problems.

In many of these cases it has been possible to resolve the issues simply by porting and compiling legacy software for 64-bit architectures (eg, to move from a 32-bit integer time_t to a 64-bit time_t). In other cases more extensive changes have been needed, especially when times get cast into integers for math or when message wire formats get involved or for when values are stored in databases. In testing and fixing support for 20-year CAs, one thing that showed up is that downstream dependencies also come into play. For example, if a date 20 years in the future gets fed into a logging system or monitoring system, and if those in-turn feed into alerting systems or reporting databases or provisioning systems, then those may also all need fixes.

On-the-wire message and file formats

As mentioned above, when 32-bit timestamps are put into messages, databases, or file formats the impact can extend well beyond a specific system. These are also systems with external dependencies where more advanced planning is often needed as interactions cross system boundaries. For these collections of interoperating systems, changes may need to be released in a specific order and backwards compatibility often comes into play. Furthermore, if there are either formally or informally standardized protocols that use 32-bit epoch timestamp values in messages, any migration or fix could be predicated on fixing the standard. As such, these become important to worry about as with a dependency chain such as:

Update protocol/standards to support post-Y2038 timestamps.
Implement support for updated standard in software libraries.
Get new version of libraries incorporated into software packages.
Get software packages included in new shipping product.

Then if each of these takes a few years and the shipping product has a long lifetime then the long lead-times here may already be a problem.

Devices with long deployed lifetimes, and their dependencies

Another area to start focusing on early is devices with long deployment lifetimes. As just mentioned, following through the external dependencies these devices have is also important. Embedded devices shipping with 32-bit hardware may also not have an easy fix of “just compile for a 64-bit time_t” via a software update, or doing so could have unacceptable performance impact.

Connected automobiles and other IoT devices are likely an area of specific concern, but I’m sure there are many other similar types of devices and applications. For example, given current trends it is likely that over 10% of cars sold today will still be operating in Y2038, and with increases in vehicle age and number of vehicles on the road this may be even higher. If it takes a few years for shipping automobiles to be generally Y2038 compliant, with the current (and increasing) distribution of motor vehicle ages we may end up with a significant fraction of automobiles with the potential to have serious issues in nineteen years. This same pattern likely exists in some other industries as well, such as in Consumer Electronics (such as home gaming consoles and smart televisions) where devices may be shipping with 20+ year CA certificates pre-installed.

Devices with long deployed lifetimes may require more comprehensive testing of both the device and its dependendencies, such as testing that the operating system and software continue to work properly before, during, and after the Y2038 transition point.

Happy New Year!

The Y2038 issue is in a similar category as IPv6: it is a multi-decade roll-out that in the general case is Important but not yet Urgent (per the Eisenhower Matrix). From this perspective, now is as good a time as any to start planning, triaging, and testing before it becomes urgent (or too late). Focus first on software dealing with future dates, on-the-wire message and file formats, and devices with long deployed lifetimes. Next, use the experience from the initial focus to build a more comprehensive program over the coming years. Regardless, set a minimum bar and start making sure that new software, systems, protocols, and products you are building and deploying don’t introduce Y2038 issues, and flag concerns when you see 32-bit timestamps-since-the-unix-epoch included in new designs.

Erik Nygren is a Fellow and Chief Architect in Akamai’s Platform Engineering organization. He’ll be celebrating his 20th anniversary at Akamai this June.

Thank you to multiple folks at Akamai for their contributions to this article.

While precautions have been taken in the preparation of this document, Akamai Technologies, Inc. assumes no responsibility for errors, omissions, or for damages resulting from the use of the information herein. The information herein is subject to change without notice. Akamai and the Akamai wave logo are registered trademarks or service marks in the United States (Reg. U.S. Pat. & Tm. Off). Published January 10, 2019.