UPDATE: Ars Technica has written an article on this problematic feature: https://arstechnica.com/security/2023/08/windows-feature-that-resets-system-clocks-based-on-random-data-is-wreaking-havoc/
TL;DR: We have found the most likely root cause: W32time Secure Time Seeding which uses the legacy "time" value in SSL handshake headers, which is not the time, but instead random data in newer
SSL implementations, interprets it as the correct time, and sets the clock accordingly.
It can be turned off by setting the UtilizeSSLTimeData registry value:
reg add HKLM\SYSTEM\CurrentControlSet\Services\w32time\Config /v UtilizeSslTimeData /t REG_DWORD /d 0 /f
And w32tm can be instructed to reread its configuration: W32tm.exe /config /update
The longer answer...
Microsoft shipped a feature called Secure Time Seeding in November 2015 in Windows Server 2016+ and turned on by default.
It's an attempt to mitigate the problem where a system has no power to drive the
system clock (e.g. a power failure and bad CMOS battery) and thus have a
completely wrong time on boot, unable to securely communicate with other sources
to reliably get the correct time.
During outgoing SSL handshakes, it looks at "ServerUnixTime" (probably
gmt_unix_time in the specification).
The TLS 1.2 specification says the following (emphasis mine):
The current time and date in standard UNIX 32-bit format (seconds since the
midnight starting Jan 1, 1970, UTC, ignoring leap seconds) according to the
sender's internal clock. Clocks are not required to be set correctly by the
basic TLS protocol; higher-level or application protocols may define additional
requirements. Note that, for historical reasons, the data element is named
using GMT, the predecessor of the current worldwide time base, UTC.
Microsoft's initial blog says the following in their post (emphasis mine):
The ServerUnixTime is supposed to be the current system time on the server, but
it can also be set to a random value by some SSL implementations. We have
observed that most servers provide a fairly accurate value in this field and the
rest provide random values. We use this data field assuming it is somewhat
accurate but can also be incorrect.
It shows they have misinterpreted the specification and their implementation is
designed and operates under wrong assumptions. Their implementation might have
worked fine when the world was still using older and more insecure
implementations, but as more and more servers are updating, the premise is
completely wrong.
At least they don't trust a single source, but as less sources provides the time
and more sources use random values, it's probably only a matter of time before
the 4 bytes are similar enough to confuse their algorithm.
We took the approach to not trust the data from a lone server, irrespective of
who the server identifies itself as. We rely on corroborating information from
multiple servers to arrive at a common truth about the current time.
They further describe that they are using statistical methods to see when they
can interpret the random bytes as a correct time (emphasis mine):
The information from ServerUnixTime and OCSP validity periods are merged to
produce the smallest possible reliable time range value along with a confidence
score. When the confidence score is sufficiently high, this data becomes
information.
The gmt_unix_time field was discussed on the TLS mailinglist in September 2013, two
years before Microsoft shipped this feature, implemented in OpenSSL October 2013, and
shipped January 2014.
commit 2016265dfbab162ec30718b5e7480add42598158
Author: Nick Mathewson <nickm@torproject.org>
Date: Sun Oct 20 15:03:24 2013 -0700
Do not include a timestamp in the Client/ServerHello Random field.
Instead, send random bytes, unless SSL_SEND_{CLIENT,SERVER}RANDOM_MODE
is set.
This is a forward-port of commits:
4af793036f6ef4f0a1078e5d7155426a98d50e37
f4c93b46edb51da71f09eda99e83eaf193a33c08
3da721dac9382c48812c8eba455528fd59af2eef
2583270191a8b27eed303c03ece1da97b9b69fd3
While the gmt_unix_time record was added in an ostensible attempt to
mitigate the dangers of a bad RNG, its presence leaks the host's view
of the current time in the clear. This minor leak can help
fingerprint TLS instances across networks and protocols... and what's
worse, it's doubtful that the gmt_unix_time record does any good at
all for its intended purpose, since:
* It's quite possible to open two TLS connections in one second.
* If the PRNG output is prone to repeat itself, ephemeral
handshakes (and who knows what else besides) are broken.
commit 2927791d77ddaef687e92b1779e0bff89bdc279f
Author: Nick Mathewson <nickm@torproject.org>
Date: Sun Oct 20 15:08:58 2013 -0700
Fix another gmt_unix_time case in server_random
Major changes between OpenSSL 1.0.1e and OpenSSL 1.0.1f [6 Jan 2014]
- Don't include gmt_unix_time in TLS server and client random values
- (.. unrelated changes ..)
u/zanatwo observed this issue in March 2017 on r/sysadmin, but Microsoft was not aware of the issue and it is still enabled.
It was also observed by u/Thranx January 2022 on r/sysadmin.
And again starting this year by @chris1out on ServerFault.
As we and others are experiencing, the issue is increasing in frequency,
probably due to less servers reporting the actual time and just random values. This may continue to increase in frequency and hit more users.
The system clock is an important shared mutable state on the system, and
bugs which change the time to a wildly different value wreaks havoc on all
systems and have repercussions far beyond the single server it happens on. This
is without a doubt the most serious bug/misfeature I've ever encountered, and
Microsoft needs to disable this ASAP.
Please report this issue to
Microsoft if you're experiencing the same issue.
Thanks a lot to @test-is-prod for sharing their findings and pointing me to the
Reddit post by /u/zanatwo!
References:
