SEC536: Adversarial AI - Penetration Testing AI Systems


Experience SANS training through course previews.
Learn MoreLet us help.
Contact usBecome a member for instant access to our free resources.
Sign UpWe're here to help.
Contact Us
When time lies, your investigation lies too.
That is not a metaphor. It is a technical reality that plays out in incident response, forensic analysis, and detection engineering, and most security teams do not discover it until a timeline stops making sense.
I have been doing this work for thirty-six years. In that time, I have watched investigations go wrong because the timestamps those analysts depended on were silently, invisibly wrong. The compromises and the logs were real, but the timelines were not.
There is a reason practitioners say, “If you truly understand time, you’ve done a lot of incident response.” This is my attempt to give you that understanding before you need it.
Before we examine what goes wrong with timestamps in security environments, we need to establish a fundamental issue. The systems most practitioners rely on for timekeeping were never engineered for forensic precision; they were engineered for coordinating human schedules.
Time zones, for example, were not invented by computer scientists. They were introduced in the nineteenth century to solve a railroad scheduling problem. Before standardized time, every city kept its own local solar time, and train timetables were a chaos of conflicting clocks. The railroads needed coordination across geography, and time zones provided it, blurring precision to prioritize consistency across a manageable region.
Daylight Saving Time compounds this. DST was introduced for economic and energy-conserving reasons, and it has been extended, contracted, and modified by governments ever since, for reasons that are largely political rather than technical. The decision to observe DST, and on what schedule, belongs entirely to individual countries and, in some cases, individual regions within those countries.
This matters because the concept of "local time" carries no universal meaning. It is a negotiated construct that is subject to change by political decision, sometimes unexpectedly.
Modern precision timekeeping does exist. Atomic clocks, GPS-based timing, and UTC itself are engineered to accurately measure in nanoseconds. That precision lives at the foundation of the system, but the civil constructs layered on top of it—like time zones and DST—were not built for the demands of log analysis or forensic investigation.
Most practitioners operate with a mental model of timekeeping that is far simpler than the underlying reality. UTC is the fixed anchor; that part is correct. But it’s easy to make assumptions that do not hold universally—for example, that offsets from it are whole-number multiples of one hour, or that the rules governing offsets are consistent and predictable.
There are more than forty distinct UTC offsets in active use around the world. India operates at UTC+5:30. Nepal is one of only three jurisdictions in the world using a 45-minute offset, at UTC+5:45. Lord Howe Island, off the coast of Australia, uses a base offset of UTC+10:30 and observes a DST shift of only 30 minutes, moving to UTC+11 in summer—a non-standard offset that changes in a non-standard way.
There are currently around 195 countries in the world, each making their own timekeeping decisions. And those decisions can change.
Offsets are not always whole hours, DST does not always shift by sixty minutes, and the rules in place today will not necessarily be the rules in place tomorrow. Any system making assumptions about offset granularity is operating on a premise that the real world does not consistently honor.
Approximately 70 of the world's countries observe Daylight Saving Time. The rules for when it starts and ends are not universal, not synchronized, and not permanent. These rules are political decisions, which means a legislature can change DST with no advance notice to your security team.
The United States and the European Union both observe DST, but on different schedules. The US moves clocks forward on the second Sunday in March and back on the first Sunday in November. The EU advances on the last Sunday in March and reverts on the last Sunday in October. During the gap between those transition dates, the offset between US and European systems changes temporarily, which can quietly break your calendar appointments, let alone your log correlation between transatlantic infrastructure every single year.
And just when you thought it couldn’t get even more exciting, how about this one: Australia observes DST from October through April, which is the calendar opposite of the Northern Hemisphere schedule. You still want more? Okay! Arizona does not observe DST, with one significant exception. The Navajo Nation within Arizona does observe DST. And within the Navajo Nation, the Hopi Reservation does not. You can drive across a single US state and cross multiple effective time zone boundaries without leaving its borders!
Governments have demonstrated repeatedly that time zone decisions can be made and reversed with minimal notice. North Korea created Pyongyang Time in 2015 (UTC+8:30) as a symbolic break from Japanese colonial-era timekeeping. Three years later, they reverted to UTC+9. Samoa crossed the International Date Line in 2011, jumping from UTC−11 to UTC+13 in a single step. Let’s break that one down and consider what it really means: Friday, December 30th, 2011, did not take place in Samoa! A full calendar day was eliminated by a trade policy decision when they skipped from December 29, 2011 to December 31, 2011. Even more fascinating than that? American Samoa, roughly 100 kilometers away, did not make the switch, and to this day, a short trip between the two crosses an entire calendar day.
Speaking of how changes can happen at any time, one is happening right now in British Columbia, Canada. In March, British Columbia fixed its UTC offset at −7, year-round, without waiting for alignment with neighboring US states. Starting November 1st, British Columbia (BC) will be one hour ahead of Washington, Oregon, and California for roughly five months each winter. Other Canadian provinces have legislation ready but are watching BC, along with US states. The Premier of BC has publicly expressed hope that American jurisdictions will follow. Whether they do or not, the offset discrepancy already exists, and any organization with log sources on both sides of that border needs to account for it today, not after the next incident.
What is the key takeaway? Time zone data for any region can change without notice. Your SIEM normalization rules and logging configurations are not a one-time setup task. They are living entities that require the same ongoing attention as any other security control.
Security teams depend on log data to reconstruct what happened during an incident. But can the timestamps in that log data actually be trusted? We have to ask this question, not because the systems are compromised, but because the logging formats themselves were (unintentionally) not designed with ambiguity in mind.
The original BSD syslog protocol, defined in RFC 3164, is still the dominant logging format in most environments. Its timestamp format records the month, day, and time, but no year, no time zone, and no UTC offset. A log entry reading `01:59:50` tells you nothing about what time that actually was. It assumes local time without defining what local time means. Even within (or without) a distributed environment spanning multiple geographies, an assumption of “local time” is definitely not safe, and we can never rely on assumptions to begin with.
Windows Event Logs take a different approach: timestamps are stored internally in UTC, which is the right design. The problem arises in how those logs are exported and displayed. Many tools convert UTC to local time before presenting or exporting the data, and the offset is not always preserved in the output. An analyst looking at a Windows event log entry may be reading local time without any indication that a conversion has occurred.
Web server logs present their own variability. Apache's combined log format does include a time zone offset in each entry, but that offset reflects the server's local time zone, which may not be UTC. Legacy and custom configurations may omit the offset entirely. IIS logs can be configured various ways and commonly default to UTC, but that is configuration-dependent and cannot be assumed.
Network devices are perhaps the most variable of all. Depending on the vendor, firmware version, and configuration, a given firewall may log in UTC, local time, or elapsed time since last reboot. Considering the last example, a device that has been running for several hundred days without a restart does not log wall-clock time at all—it logs uptime, which requires arithmetic to translate, and we can’t rely on that either!
The cumulative effect is an environment where log sources record time in at least three or four different implicit conventions, none of which are documented in the log entries themselves.
The problems described here are not theoretical. Here is an example of what this looks like in practice:
One event occurs, and three devices are all logging in local time with no time zone indicator.
A web server in the US/Pacific region logs an alert at `01:59:50`. A firewall in the United Kingdom logs a related event at `09:59:50`. A SIEM hosted in India logs a correlation match at `15:29:50`.
Did those three events happen at the same moment? You cannot answer that question with confidence from the log data alone.
The web server timestamp could reflect PST (UTC−8) or PDT (UTC−7), a one-hour variation depending on the time of year and whether the device's DST rules are current. The UK firewall presents the same ambiguity: GMT (UTC+0) in winter or BST (UTC+1) in summer. RFC 3164 syslog carries neither an offset nor a DST status flag. The SIEM in India is the only unambiguous entry—India Standard Time is UTC+5:30 year-round, with no DST—but resolving it requires knowing where that system is located, and that information is not in the log entry itself.
This is not a failure of tooling, but rather a failure of assumption. The analyst examining these logs is working with incomplete data and filling the gaps with conjecture.
Every problem described so far converges on the same solution: UTC.
Coordinated Universal Time (UTC) is the international standard maintained by the Bureau International des Poids et Mesures on the basis of atomic clocks operated worldwide. It does not observe Daylight Saving Time, ever. It carries no geographic offset. It does not shift when a government changes its clocks. A UTC timestamp means precisely the same thing everywhere on the planet, at every time of year.
If all log sources record timestamps in UTC, every entry from every source can be compared, sorted, and correlated directly, without conversion, without prior knowledge of where the source system is located, and without assumptions about what time zone rules were in effect at the time of the event. This consistency is the prerequisite for reliable log correlation.
RFC 5424, the current IETF syslog standard that supersedes RFC 3164, operationalizes this principle by requiring an explicit time zone offset in every message. Configuring that offset to UTC simultaneously solves the timestamp ambiguity problem and brings logging into alignment with the current standard. For any new deployment, there is no reason not to do this.
There is one important caveat, however: UTC solves the format problem, but it does not solve the accuracy problem.
A device configured to log in UTC but whose clock is drifting will produce correctly formatted but factually wrong timestamps. The solution: the Network Time Protocol, specified in RFC 5905. NTP maintains synchronization through a hierarchy: atomic clocks and GPS receivers at the top, Stratum 1 servers synchronized directly to those reference sources, and your infrastructure synchronized downstream from there.
UTC and NTP are complementary; neither alone is sufficient. UTC tells your systems what to say about time, and NTP ensures what they say is true.
Lastly, consider that clock skew monitoring belongs on your security dashboard. Unexplained drifts, especially on high-value systems, should be investigated as a potential indicator of tampering, not handed off as a maintenance ticket. NTP health is a security control and we must treat it like one.
Earth's rotation is not perfectly uniform. Earthquakes redistribute mass and alter the spin rate, as does tidal friction from the Moon's gravity. Over time, atomic time—the precise timescale that underpins UTC—gradually diverges from astronomical time based on Earth's actual rotation. The International Earth Rotation and Reference Systems Service (IERS) monitors this divergence and announces corrections roughly six months in advance.
The correction is called a leap second. A leap second can be added or subtracted to make a correction. If a single second is added to the end of a UTC minute, it creates a moment designated 23:59:60 UTC; this is a valid UTC moment, but most logging systems are not built to handle it. Some implementations respond by repeating 23:59:59, creating duplicate timestamps. Others step the clock backward, producing events that appear to predate events that actually preceded them. Reddit suffered a significant outage in 2012 from a leap second handling bug. Cloudflare suffered a similar incident in 2017.
In recent years, Earth has spun slightly faster than average, raising the possibility of a negative leap second—a 59-second minute. It has never been triggered. Where a positive leap second creates duplicate timestamps, a negative one deletes a second entirely. Any event that occurs in that missing second has no timestamp, making it harder to detect and harder to recover from.
In 2022, the General Conference on Weights and Measures (CGPM) voted to eliminate leap seconds by 2035, deferring the problem rather than solving it. UTC will be allowed to drift from astronomical time by up to one minute before any correction is made, though the exact correction mechanism remains under discussion.
Check whether your logging infrastructure handles leap seconds. If you have never looked, look now.
Get your logging in order before the breach, audit your sources, and document your configurations. Monitor your NTP, and build normalization that accounts for what time actually is: a political, inconsistent, and constantly evolving construct that your detection infrastructure depends on being accurate.
When time lies, your investigation lies too.
Watch the full talk from Secure Your Fortress 2026.
Download the Timestamp Audit Checklist to assess your log sources, ingestion pipeline, NTP health, and time zone change readiness.
To go deeper on the fundamentals every security practitioner needs, explore my course: SEC401: Security Essentials


Bryan is a SANS Senior Instructor and author of SEC401. With 30+ years of cybersecurity experience and 22 GIAC certifications—including the prestigious GSE—he's trained professionals from the FBI, NATO, and the UN. He is the CEO of Xploit Security Inc.
Read more about Bryan Simon