Ubuntu 20.04 time sync problems and possibly incorrect status information

Question

I have been having some problems with crashes on my KVM host (Lubuntu 20.04), and when troubleshooting, I noticed some time-related errors.

Upon further investigation, to my horror, I saw that time was not being synced. I am sure it was set up before, I have no clue how it became un-setup.

admin@virtland:~$ sudo timedatectl
[sudo] password for admin: 
               Local time: Fri 2020-07-10 09:14:14 EDT  
           Universal time: Fri 2020-07-10 13:14:14 UTC  
                 RTC time: Fri 2020-07-10 13:14:14      
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: no                           
              NTP service: n/a                          
          RTC in local TZ: no                           
admin@virtland:~$ 

I found this thread and tried the top answer, but no no avail. https://askubuntu.com/questions/929805/timedatectl-ntp-sync-cannot-set-to-yes

admin@virtland:~$ sudo systemctl stop ntp
admin@virtland:~$ sudo ntpd -gq
10 Jul 09:17:57 ntpd[34358]: ntpd 4.2.8p12@1.3728-o (1): Starting
10 Jul 09:17:57 ntpd[34358]: Command line: ntpd -gq
10 Jul 09:17:57 ntpd[34358]: proto: precision = 0.070 usec (-24)
10 Jul 09:17:57 ntpd[34358]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): good hash signature
10 Jul 09:17:57 ntpd[34358]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): loaded, expire=2020-12-28T00:00:00Z last=2017-01-01T00:00:00Z ofs=37
10 Jul 09:17:57 ntpd[34358]: Listen and drop on 0 v6wildcard [::]:123
10 Jul 09:17:57 ntpd[34358]: Listen and drop on 1 v4wildcard 0.0.0.0:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 2 lo 127.0.0.1:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 3 enp6s0 10.0.0.18:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 4 lo [::1]:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 5 enp6s0 [fe80::7285:c2ff:fe65:9f19%3]:123
10 Jul 09:17:57 ntpd[34358]: Listening on routing socket on fd #22 for interface updates
10 Jul 09:17:58 ntpd[34358]: Soliciting pool server 209.50.63.74
10 Jul 09:17:59 ntpd[34358]: Soliciting pool server 4.53.160.75
10 Jul 09:18:00 ntpd[34358]: Soliciting pool server 69.89.207.199
10 Jul 09:18:00 ntpd[34358]: Soliciting pool server 72.30.35.88
10 Jul 09:18:01 ntpd[34358]: Soliciting pool server 173.0.48.220
10 Jul 09:18:01 ntpd[34358]: Soliciting pool server 162.159.200.1
10 Jul 09:18:01 ntpd[34358]: Soliciting pool server 108.61.73.243
10 Jul 09:18:02 ntpd[34358]: Soliciting pool server 208.79.89.249
10 Jul 09:18:02 ntpd[34358]: Soliciting pool server 208.67.75.242
10 Jul 09:18:02 ntpd[34358]: Soliciting pool server 91.189.94.4
10 Jul 09:18:03 ntpd[34358]: Soliciting pool server 91.189.89.198
10 Jul 09:18:03 ntpd[34358]: Soliciting pool server 67.217.112.181
10 Jul 09:18:04 ntpd[34358]: Soliciting pool server 91.189.89.199
10 Jul 09:18:04 ntpd[34358]: Soliciting pool server 64.225.34.103             
10 Jul 09:18:05 ntpd[34358]: Soliciting pool server 91.189.91.157             
10 Jul 09:18:06 ntpd[34358]: Soliciting pool server 2001:67c:1560:8003::c8    
10 Jul 09:18:06 ntpd[34358]: ntpd: time slew +0.001834 s                      
ntpd: time slew +0.001834s                                                    
admin@virtland:~$ sudo service ntp start                                       
admin@virtland:~$ sudo timedatectl                             
               Local time: Fri 2020-07-10 09:18:21 EDT                        
           Universal time: Fri 2020-07-10 13:18:21 UTC                        
                 RTC time: Fri 2020-07-10 13:18:21                            
                Time zone: America/New_York (EDT, -0400)                      
System clock synchronized: no                                                 
              NTP service: n/a                                                
          RTC in local TZ: no                                                 
admin@virtland:~$                  

I thought maybe I needed to use some more up-to-date instructions, so I tried this: https://linuxconfig.org/how-to-sync-time-on-ubuntu-20-04-focal-fossa-linux

admin@virtland:~$ timedatectl set-ntp off
Failed to set ntp: NTP not supported                                          
admin@virtland:~$ timedatectl set-ntp on
Failed to set ntp: NTP not supported                                          

Then I tried this, from a different thread:

admin@virtland:~$ sudo systemctl status systemd-timesyncd.service
[sudo] password for admin: 
● systemd-timesyncd.service
     Loaded: masked (Reason: Unit systemd-timesyncd.service is masked.)
     Active: inactive (dead)

I have never touched timesyncd.conf, but it is entirely commented out anyway:

cat /etc/systemd/timesyncd.conf                         
#  This file is part of systemd.                                              
#                                                                             
#  systemd is free software; you can redistribute it and/or modify it         
#  under the terms of the GNU Lesser General Public License as published by   
#  the Free Software Foundation; either version 2.1 of the License, or        
#  (at your option) any later version.                                        
#                                                                             
# Entries in this file show the compile time defaults.                        
# You can change settings by editing this file.                               
# Defaults can be restored by simply deleting this file.                      
#                                                                             
# See timesyncd.conf(5) for details.
[Time]

#NTP=

#FallbackNTP=ntp.ubuntu.com

#RootDistanceMaxSec=5

#PollIntervalMinSec=32

#PollIntervalMaxSec=2048 

I checked timedatectl again, and now it is on, but still not using NTP. I understand that NTP is more precise, and that can be important in some situations. Not sure if virtualization with pci passthrough needs extremely precise time or not.

From other stuff I was reading, I thought maybe NTP was conflicting with timesyncd. So remove ntp for the time being:

sudo systemctl stop ntp
sudo apt-get purge ntp

But after purging ntp, NTP showed as active!

admin@virtland:~$ timedatectl
               Local time: Fri 2020-07-10 09:34:52 EDT  
           Universal time: Fri 2020-07-10 13:34:52 UTC  
                 RTC time: Fri 2020-07-10 13:34:52      
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes                          
              NTP service: active                       
          RTC in local TZ: no     

Am I going crazy? Is NTP still here somehow? Nope.

admin@virtland:~$ sudo systemctl start ntp
Failed to start ntp.service: Unit ntp.service not found.

Apologies for not asking a more focused question, but what the heck is going on here?

I am well and truly lost. Also, I will edit this post later and make a not as to whether removing NTP (and thus activating it?!) fixed the stability problems that led me down this rabbit hole.

Edit: The next thing I did was disable ntp on timesyncd and (re)install NTP as described here. https://www.digitalocean.com/community/tutorials/how-to-set-up-time-synchronization-on-ubuntu-18-04

That resulted in:

admin@virtland:~$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.us.pool.ntp.o .POOL.          16 p    -   64    0    0.000    0.000   0.000
 1.us.pool.ntp.o .POOL.          16 p    -   64    0    0.000    0.000   0.000
 2.us.pool.ntp.o .POOL.          16 p    -   64    0    0.000    0.000   0.000
 3.us.pool.ntp.o .POOL.          16 p    -   64    0    0.000    0.000   0.000
 0.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 1.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 2.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 3.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 ntp.ubuntu.com  .POOL.          16 p    -   64    0    0.000    0.000   0.000
admin@virtland:~$ timedatectl
               Local time: Fri 2020-07-10 10:35:39 EDT  
           Universal time: Fri 2020-07-10 14:35:39 UTC  
                 RTC time: Fri 2020-07-10 14:35:40      
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: no                                                 
              NTP service: n/a                                                
          RTC in local TZ: no                                                 
admin@virtland:~$ systemctl status systemd-timesyncd 
● systemd-timesyncd.service                                                   
     Loaded: masked (Reason: Unit systemd-timesyncd.service is masked.)       
     Active: inactive (dead)                                                  
admin@virtland:~$ nano /etc/ntp.conf
admin@virtland:~$ systemctl status systemd-timesyncd                           
● systemd-timesyncd.service                                                   
     Loaded: masked (Reason: Unit systemd-timesyncd.service is masked.)       
     Active: inactive (dead)                                                  
admin@virtland:~$ ntpstat
unsynchronised                                                                
   polling server every 8 s 

I reversed those changes as recommended my Michael Hampton: Does this mean it's working?

boss@virtland:~$ sudo systemctl stop ntp
Failed to stop ntp.service: Unit ntp.service not loaded.
boss@virtland:~$ sudo timedatectl set-ntp yes
boss@virtland:~$ sudo timedatectl set-ntp on
boss@virtland:~$ ntpq -p
bash: /usr/bin/ntpq: No such file or directory
boss@virtland:~$ systemctl status systemd-timesyncd 
● systemd-timesyncd.service - Network Time Synchronization
     Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; >
     Active: active (running) since Fri 2020-07-10 10:49:18 EDT; 50s ago
       Docs: man:systemd-timesyncd.service(8)
   Main PID: 108365 (systemd-timesyn)
     Status: "Initial synchronization to time server 91.189.94.4:123 (ntp.ubu>
      Tasks: 2 (limit: 154317)
     Memory: 1.8M
     CGroup: /system.slice/systemd-timesyncd.service
             └─108365 /lib/systemd/systemd-timesyncd
Jul 10 10:49:17 virtland systemd[1]: Starting Network Time Synchronization...
Jul 10 10:49:18 virtland systemd[1]: Started Network Time Synchronization.
Jul 10 10:49:18 virtland systemd-timesyncd[108365]: Initial synchronization t>
lines 1-14/14 (END)

 timedatectl
               Local time: Fri 2020-07-10 10:52:56 EDT  
           Universal time: Fri 2020-07-10 14:52:56 UTC  
                 RTC time: Fri 2020-07-10 14:52:56      
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes                          
              NTP service: active                       
          RTC in local TZ: no            

So I guess it is working. Since the crashes that took me down this path are still happening, I guess the time wasn't the issue.

Answer 1

The timedatectl command reports that NTP is in use when either chronyd or systemd-timesyncd is enabled and running. Both of these are lightweight NTP clients suitable for most servers and workstations. It does not recognize the ntpd service, which is designed to be both an NTP client and server, and now that we have dedicated NTP client software, is no longer favored for systems that don't need to serve NTP onward to other systems or use specialized hardware such as GPS to obtain the time.

Answer 2

Just to clarify differences between the three most commnly used deamons:

timesyncd and ntpd are both time synchronization services available in Linux, but they differ significantly in terms of functionality, complexity, and use cases. Here's a breakdown of the key differences and insights on which might be better depending on your needs.

1. Purpose and Use Cases

- systemd-timesyncd:

• Purpose: A simple NTP client built into systemd for basic time synchronization with remote NTP servers.
• Use Case: Suitable for desktop users, lightweight servers, or environments where you need reliable, accurate time synchronization without the need for advanced features like custom time configurations.

- ntpd (Network Time Protocol Daemon):

• Purpose: A full-featured, standalone NTP implementation that provides both time synchronization and advanced configuration options.
• Use Case: Ideal for environments where high precision, custom configurations, or time server capabilities are required, such as in data centers, enterprise systems, or networks with complex timekeeping needs.

2. Functionality and Features

- systemd-timesyncd:

• Simple NTP client.
• Uses fewer system resources and has low overhead.
• Primarily synchronizes time with external NTP servers, without providing time to other systems.
• Does not support advanced NTP features like hardware timestamping, precise timekeeping, or local clock stratum levels.

- ntpd:

• A full NTP implementation, supporting both client and server modes.
• Can serve as an NTP server, distributing time to other systems.
• Supports complex configurations (e.g., peering between NTP servers, multiple time sources, clock discipline, leap second handling).
• Provides higher precision, including microsecond-level accuracy, depending on the hardware and network conditions.
• Can be configured to use hardware reference clocks (e.g., GPS receivers) for high-precision timekeeping.

3. Ease of Use and Configuration

- systemd-timesyncd:

• Easy to use and configure, with minimal setup required.
• Configuration is done through a basic /etc/systemd/timesyncd.conf file or automatically via DHCP.
• Suitable for users who just need accurate time synchronization without delving into detailed configurations.

- ntpd:

• More complex to set up and configure due to its extensive feature set.
• Requires manual configuration of NTP servers in /etc/ntp.conf.
• Offers more detailed control over time synchronization, but may require deeper knowledge of NTP protocols and time synchronization principles.

4. Resource Usage

- systemd-timesyncd:

• Lightweight, with very low CPU and memory overhead.
• Designed for systems where minimal resource consumption is critical (e.g., embedded devices, lightweight servers).

- ntpd:

• Consumes more resources, though still fairly low for most modern systems.
• May be overkill for systems that only need basic time synchronization, given its complexity.

5. Precision

- systemd-timesyncd:

• Provides good enough precision for most non-critical use cases (within milliseconds or tens of milliseconds).

- ntpd:

• Offers higher precision (down to microseconds, depending on the hardware and network).
• Better suited for applications where precise timekeeping is essential, such as financial systems, scientific experiments, or real-time systems.

6. Which Is Better?

- systemd-timesyncd is better if:

• You need simple, reliable time synchronization without any advanced requirements.
• You're running a lightweight server, desktop, or embedded device where resource efficiency is important.
• You prefer simplicity and minimal configuration.
• Precision requirements are not extremely high (milliseconds are acceptable).

- ntpd is better if:

• You require high-precision time synchronization (microsecond accuracy or better).
• You need to configure a time server to serve time to other devices on the network.
• You need advanced NTP features, such as hardware clock synchronization, multiple time sources, or peer mode.
• You are working in an environment with strict timekeeping requirements, such as in scientific research, finance, or real time systems.

7. Alternatives:

chrony: Another time synchronization tool that is often considered a middle ground between timesyncd and ntpd. It is lightweight like timesyncd but offers many advanced features and higher precision, making it a popular choice for server environments.

Summary:

• timesyncd: Ideal for most typical users and servers where simplicity and lightweight functionality are key.

• ntpd: Ideal for complex environments where precise timekeeping, server configurations, and advanced NTP features are necessary.

• chrony: A strong alternative that combines the simplicity of timesyncd with some of the advanced features of ntpd.