Time. The common enemy of all humans. It is unstoppable, uncontrollable, unforgiving, and unbelievably necessary to keep your IT systems functioning. No police box or DeLorean required.

What time is it?

Computer systems see time differently than we humans. Yes, they can present the exact second on the screen to painfully remind us we are late, but how do they know what time it is? Different systems measure it differently but each one uses something called Epoch Time. No, not the newspaper. This is a beginning date and time in history, down to the nanosecond, the computer system uses to calculate current time.

Epoch Time for Windows computers is January 1, 1601 at exactly 00:00:00 Universal Time (UTC). When you look at the clock in the corner of your screen, you are seeing a representation of all the nanoseconds (in 100-nanosecond blocks, because someone will correct me) that have passed since that point in history. No, really. That’s how it works.

Different vendors use different Epoch Times. Unix/Linux systems use the Epoch Time of January 1, 1970 00:00:00 UTC. Nearly all devices, including iPhones, Androids, network equipment, video cameras, and anything else you can think of, use the Unix Epoch Time. So, here is an example. Let’s say an event occurs exactly on February 29, 2024 at 00:00:00 UTC. The Unix Epoch Time of that event is 1709164800000 milliseconds since Epoch Time.

Why is Epoch Time important?

Time is different all around the world. Some places, you know who you are, think it’s cute to divide their time zone into half-hour segments. If an event occurs in India in the middle of the night, I see it real time in the US during the day. If I am searching through aggregated logs from hundreds of devices, like a Security Information and Event Management (SIEM) system, I may have millions of log entries for that second in time. By narrowing to the millisecond, I know EXACTLY when the event occurred.

The India event can be identified by its Epoch Time, down to the millisecond. Epoch Time is the same in the US as in India, or Mars. What time zone is Mars in? Regardless of where I am in the solar system, I will know exactly when the event occurred relevant to me. I can corroborate with other logs to see how the event might have affected other systems before or after the event. If all systems are properly synchronized, I can see the complete history of the event.

Synchronize Your Swatches!

How do we synchronize all of our devices to ensure they are within nanoseconds of one another? Sounds daunting. Enter the Network Time Protocol (NTP). The most trusted sources of NTP sync with the atomic clock then allow outside devices to query for the correct time. These are stratum 0, or first-level clocks, maintained in the US by the US Navy and certain universities. If you want to get in the weeds and learn the fascinating history of this protocol, click here. (Link)

Organizations typically synchronize only a few primary devices in their networks to stratum 0 sources and keep time sync in-house for all other devices. Core switches and routers typically sync external and all other devices and servers sync there. Cloud-based servers can introduce inconsistencies. The best recommendation is ensure internal and Cloud systems are ultimately syncing to the same stratum 0 servers.

Why? Well, NTP has a margin of error in the milliseconds. If your organization has thousands of devices filling the log files, milliseconds could create more work for your log viewers and IT staff. Nit-picky much? When you need to find an incident fast and you have 10 million logs entries to sort through, you’ll thank me. Read about how the Internet works here. (Link)

Just how fast is a nanosecond? One-billionth of a second! Can’t think that fast? You’re right! MIT researchers determined the fastest response times are measured in milliseconds. (Link)

Time Zones and Daylight Savings Time

How many time zones are there around the world? No, it’s not 24. Politics, borders, and the International Date Line created 38! Really? Yes! (Link) Leap Years and Daylight Savings Time mean time could be different anywhere, anytime. Underneath the hood, the system is taking UTC and adding or subtracting based on time zone and year. Most systems are smart enough to know when the date changes.

Daylight Savings Time is a major wrinkle that wreaks havoc for software and code developers. When the last Federal change to Daylight Savings Time occurred in 2007, we were forced to download new firmware and software updates to accommodate the change. Every network device required an update. No matter what side of the daylight savings time debate you’re on, you MUST maintain accurate time across all devices, and that starts with proper configuration and verification.

Sync Issues

Nearly all servers hosting data or granting access to systems synchronize with a group of other servers for redundancy. If time is off, even by milliseconds, the servers will refuse to communicate. Why? This is usually only an issue with systems sharing a database. Database transactions are time sensitive. The controlling master server of the database must ensure any transactions written include the correct time or overwrites may occur.

The Next Y2K

If you’ve been in the workforce for a few decades, like myself, you may remember Y2K. If not, Y2K, for Year 2000, refers to a coding issue. Developers coded years with only two (2) digits. So, after December 31, 1999, the system would think the next day was January 1, 1900 because applications were not coded to use Epoch Time. Tons of speculation abounded about what could go wrong. Fortunately, due to a massive effort to replace and repair older systems and code, very few systems were affected.

Epoch Time is stored as a 32-bit, binary number. What happens when it needs 33 bits? Right! The 32 bits will all read ZERO, and the date will roll back to December 13, 1901. Why not 1970? Math! The new number is 32-bits BEFORE Epoch Time. They actually call it the “Epochalypse”.

Repent! The End is Near! Well, no, it’s not. That will occur in 2038 and most developers and systems manufacturers have already started saving time as a 64-bit integer. Read more here. (Link)

Got A Minute?

If you’ve read this far, we GREATLY appreciate your time! If you found this post useful, entertaining, or educational, please share it with others. I always enjoy feedback, even the ones who correct me with specifics that I inevitably miss. Sign up for our newsletter to receive even more content and early access to posts.

I also love REQUESTS! Is there a technology, policy, feature, or threat you always wanted to know about but were afraid to ask? Send me a message. I won’t tell. Snitches get stitches.