5

I found that after the latest leap second insertion (2016-12-31 23:59:60), our CentOS7 application that has worker threads sleeping for 1 second between jobs, started to wake the sleeping threads immediately instead of in a second. In general, all sleeps are awake 1 second ahead of expected wake time.

The simplest and working solution is to reboot the box. But that is not desirable in our case. Is there a way to fix this without rebooting?

PS. For reference, here's a simple program in C++ that reproduces the issue.

#include <boost/date_time.hpp>
#include <boost/thread.hpp>
#include <iostream>

using namespace std;


// this has to be run in a thread to be able to detect the issue
void check_thread()
{
    size_t expected_delay = 1000;
    cout << "Expected delay: " << expected_delay << " ms" << endl;
    boost::posix_time::ptime t1 = boost::posix_time::microsec_clock::universal_time();
    boost::this_thread::sleep(boost::posix_time::milliseconds(1000));
    boost::posix_time::ptime t2 = boost::posix_time::microsec_clock::universal_time();
    size_t actual_delay = (t2 - t1).total_milliseconds();
    cout << "Actual delay: " << actual_delay << " ms" << endl;
    if (abs(expected_delay - actual_delay) > 900) {
        cout << "Too big delay difference: " << (expected_delay - actual_delay) << endl;
        cout << "Possible leap second issue" << endl;
    }
    else {
        cout << "No issues found" << endl;
    }
}

int main()
{
    boost::thread_group g;
    g.create_thread(check_thread);
    g.join_all();
    return 0;
}

Building:

g++ sleep_test.cpp -Wl,-Bstatic -lboost_thread -lboost_system -lboost_date_time -Wl,-Bdynamic -rdynamic -pthread
amosk
  • 171

2 Answers2

6

Is your system time synched with ntpd or ptp? If not, update your tzdata package.

For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 7 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el7, or a later version, to receive this fix.

Resolve Leap Second Issues in Red Hat Enterprise Linux

Troy Osborne
  • 106
0

In addition to what Troy said, on RHEL7 systems that had not updated tzdata by the moment the leap second is applied, and not running ntpd, an extra step is required - manually set time 1 second forward, then revert it:

date -s "+1 sec"
date -s "-1 sec"
amosk
  • 171