RTC synchronisation issues

Ubuntu Core docs
1

Many IoT devices rely on a real-time clock (RTC) to provide accurate system time when operating with intermittent connectivity, or completely offline.

To help devices like these, the Linux kernel provides a CONFIG_HCTOSYS configuration option that will update the system time from the RTC time early in the boot process, on driver initialisation.

  • Problem 1: A limitation of the RTC driver for Linux 5.7 and earlier (equivalent to Ubuntu kernels up-to and including v5.4) prevents loadable RTC modules from syncing system time to RTC time, regardless of the CONFIG_HCTOSYS configuration option. This means Ubuntu Core Systems using both Ubuntu kernel 5.4 and loadable modules for RTC drivers, cannot initialise the system time to the RTC time without an RTC kernel patch .

  • Problem 2: The Systemd unit systemd-timesyncd.service should be able to provide rough monotonic time across reboots, but a bug causes the dependent timesync timestamp file /var/lib/systemd/timesync/clock to be removed on every reboot. This effectively means systemd-timesyncd.service will fall-back to the Systemd build time on every boot.

  • Problem 3: Ubuntu Core Systems using Ubuntu kernel 5.15 can correctly sync the system time to the RTC time but there is no hard dependency to ensure the update or time correction occurs before snap services are started (although this is typically the case).

A solution was put in place to address problems 1, 2, and 3.

Proposed solution

The proposed solution introduces the following behavioural changes:

  • System time is synchronised from a consistency-checked RTC time during bootup if the RTC driver fails to do this (e.g. Ubuntu kernel 5.15 or earlier)
  • Deterministic time initialisation order is enforced to ensure system time is synced with RTC time before the snapd snap services starts

With the exception of a proposed solution to problem 2, current behaviour will not be affected unless enabled explicitly with a new command line parameter:

ubuntu_core.rtc_sys_time_init=/dev/rtc<x>

With the solution enabled, the rtc-sys-time-init@dev-rtc<x>.service unit is instantiated early in the boot.

This service waits for dev-rtc<x>.device to be created on RTC module initialisation, and then sets the system time to the highest of either 1) base snap build, 2) timesync or 3) RTC timestamps (if this was not done during module initialisation) (solving problem 1). The bugfix ensures a dependent timesync timestamp file /var/lib/systemd/timesync/clock is available (solving problem 2).

The rtc-sys-time-init@dev-rtc<x>.service service requires the systemd-timesyncd.service service to wait for completion of the system time initialisation. This enforces a deterministic order:

  1. RTC driver initialisation, creation of dev-rtc<x>.device
  2. rtc-sys-time-init@dev-rtc<x>.service sets system time to RTC time, if required
  3. systemd-timesyncd.service starts up and forwards system time to the timesync timestamp, if required (and not required if RTC works)
  4. Targets time-set.target and sys-init.target are reached in that order
  5. Services core.start-snapd.service and snapd.service start in that order
  6. Other snap services are started

This deterministic order ensures that the system time is set to the best known time by the time services are started to ensure that snap services do not experience unnecessary crude time adjustments that may upset time sensitive logic and make logs hard to interpret (solving problem 3✓).

The two diagrams below demonstrate the key behavioural differences introduced by this solution for Ubuntu Core using Ubuntu Kernel v5.4 and before. For Ubuntu Kernel v5.15 and later, the only difference is that system time will be set on RTC driver initialisation.

Boot behaviour diagrams

Diagram 1: boot behaviour before the solution

Boot behaviour before the solution
Boot behaviour before the solution911×1130

Diagram 2: boot behaviour with the solution

Boot behaviour with the solution
Boot behaviour with the solution472×1185

Using the solution

To set the RTC kernel configuration to sync system time is synced to RTC time:

CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc<x>"

If desired, set kernel configuration to allow systemd-timesyncd.service to update RTC on NTP synchronisation:

CONFIG_RTC_SYSTOHC=y
CONFIG_RTC_SYSTOHC_DEVICE="rtc<x>"

Add kernel command line parameter to use the solution that address 1 and 3:

ubuntu_core.rtc_sys_time_init=/dev/rtc<x>
# Important: The rtc specified must match that specified for
# CONFIG_RTC_HCTOSYS_DEVICE

For offline systems, it is important to consider that RTC initialisation will not happen, and that a solution needs to be provided by the developer e.g. factory setting, connectivity during commissioning, etc.

Required workaround

Ubuntu Kernel v5.4 and before does not sync the system time to RTC time on RTC module initialisation. This means the systemd-udev attribute hctosys is not set as expected, and the udev rule that links /dev/rtc to the intended /dev/rtc<x> does not work as expected and always links to /dev/rtc0. In the unlikely event that it is required to use an RTC other than RTC0, this is a problem that requires a workaround.

The recommended workaround is to use a install hook within the gadget snap to create a custom udev rule that links /dev/rtc to the intended /dev/rtc<x>:

snap/hooks/install:

mkdir  -p  /etc/udev/rules.d  ||  true
if  [  -d  "/etc/udev/rules.d"  ];  then
   cat  <<  EOF  >  /etc/udev/rules.d/90-rtc<x>.rules  SUBSYSTEM=="rtc",  KERNEL=="rtc<x>", SYMLINK+="rtc"
   EOF
fi

snap/snapcraft.yaml (snippet):

name: <something>-gadget

hooks:
  install:
    plugs:
      -rtc<x>-udev-service

plugs:
  rtc<x>-udev-service:
    interface: system-files
    write:
      - /etc/udev/rules.d

Verify the solution is working

Plot boot-up sequence

Use the systemd-analyze tool to generate a html based view of the boot sequence:

systemd-analyze plot > boot-sequence.html

The output html file can then be viewed in a web a browser:

systemd-analyze output after RTC solution
systemd-analyze output after RTC solution770×1114

Inspect the journal

Search for keyword rtc:

journalctl -b | grep rtc

The following are example result on AMD64 system with Ubuntu Kernel >v5.4 (snippet):

ubuntu kernel: Command line: snapd_recovery_mode=run console=ttyS0,115200n8 console=tty1 panic=-1 ubuntu_core.rtc_sys_time_init=/dev/rtc0
ubuntu kernel: rtc_cmos 00:04: registered as rtc0
[...]
ubuntu systemd[1]: Found device /dev/rtc0.
ubuntu systemd[1]: Starting Correct system time and sync to RTC time...
[...]
# This is the output of rtc-sys-time-init@dev-rtc<x>.service
# The messages here will indicate if it was required to sync system
# time to RTC (Ubuntu Kernel 5.4-) etc...
[...]
ubuntu rtc-sys-time-init[608]: System time is ahead of most recent timestamp, skipping fixup
ubuntu systemd[1]: rtc-sys-time-init@dev-rtc0.service: Succeeded.
ubuntu systemd[1]: Finished Correct system time and sync to RTC time.
1 Like