Many IoT devices rely on a real-time clock (RTC) to provide accurate system time when operating with intermittent connectivity, or completely offline.
To help devices like these, the Linux kernel provides a
CONFIG_HCTOSYS configuration option that will update the system time from the RTC time early in the boot process, on driver initialisation.
Problem 1: A limitation of the RTC driver for Linux 5.7 and earlier (equivalent to Ubuntu kernels up-to and including v5.4) prevents loadable RTC modules from syncing system time to RTC time, regardless of the CONFIG_HCTOSYS configuration option. This means Ubuntu Core Systems using both Ubuntu kernel 5.4 and loadable modules for RTC drivers, cannot initialise the system time to the RTC time without an RTC kernel patch .
Problem 2: The Systemd unit systemd-timesyncd.service should be able to provide rough monotonic time across reboots, but a bug causes the dependent timesync timestamp file /var/lib/systemd/timesync/clock to be removed on every reboot. This effectively means systemd-timesyncd.service will fall-back to the Systemd build time on every boot.
Problem 3: Ubuntu Core Systems using Ubuntu kernel 5.15 can correctly sync the system time to the RTC time but there is no hard dependency to ensure the update or time correction occurs before snap services are started (although this is typically the case).
A solution was put in place to address problems 1, 2, and 3.
- Proposed solution
- Boot behaviour diagrams
- Using the solution
- Required workaround
- Verify the solution is working
The proposed solution introduces the following behavioural changes:
- System time is synchronised from a consistency-checked RTC time during bootup if the RTC driver fails to do this (e.g. Ubuntu kernel 5.15 or earlier)
- Deterministic time initialisation order is enforced to ensure system time is synced with RTC time before the snapd snap services starts
With the exception of a proposed solution to problem 2, current behaviour will not be affected unless enabled explicitly with a new command line parameter:
With the solution enabled, the
rtc-sys-time-init@dev-rtc<x>.service unit is instantiated early in the boot.
This service waits for
dev-rtc<x>.device to be created on RTC module initialisation, and then sets the system time to the highest of either 1) base snap build, 2) timesync or 3) RTC timestamps (if this was not done during module initialisation) (solving problem 1). The bugfix ensures a dependent timesync timestamp file
/var/lib/systemd/timesync/clock is available (solving problem 2).
rtc-sys-time-init@dev-rtc<x>.service service requires the
systemd-timesyncd.service service to wait for completion of the system time initialisation. This enforces a deterministic order:
- RTC driver initialisation, creation of
rtc-sys-time-init@dev-rtc<x>.servicesets system time to RTC time, if required
systemd-timesyncd.servicestarts up and forwards system time to the timesync timestamp, if required (and not required if RTC works)
sys-init.targetare reached in that order
snapd.servicestart in that order
- Other snap services are started
This deterministic order ensures that the system time is set to the best known time by the time services are started to ensure that snap services do not experience unnecessary crude time adjustments that may upset time sensitive logic and make logs hard to interpret (solving problem 3✓).
The two diagrams below demonstrate the key behavioural differences introduced by this solution for Ubuntu Core using Ubuntu Kernel v5.4 and before. For Ubuntu Kernel v5.15 and later, the only difference is that system time will be set on RTC driver initialisation.
Boot behaviour diagrams
Diagram 1: boot behaviour before the solution
Diagram 2: boot behaviour with the solution
Using the solution
To set the RTC kernel configuration to sync system time is synced to RTC time:
If desired, set kernel configuration to allow
systemd-timesyncd.service to update RTC on NTP synchronisation:
Add kernel command line parameter to use the solution that address 1 and 3:
ubuntu_core.rtc_sys_time_init=/dev/rtc<x> # Important: The rtc specified must match that specified for # CONFIG_RTC_HCTOSYS_DEVICE
For offline systems, it is important to consider that RTC initialisation will not happen, and that a solution needs to be provided by the developer e.g. factory setting, connectivity during commissioning, etc.
Ubuntu Kernel v5.4 and before does not sync the system time to RTC time on RTC module initialisation. This means the
hctosys is not set as expected, and the udev rule that links
/dev/rtc to the intended
/dev/rtc<x> does not work as expected and always links to
/dev/rtc0. In the unlikely event that it is required to use an RTC other than RTC0, this is a problem that requires a workaround.
The recommended workaround is to use a install hook within the gadget snap to create a custom udev rule that links
/dev/rtc to the intended
mkdir -p /etc/udev/rules.d || true if [ -d "/etc/udev/rules.d" ]; then cat << EOF > /etc/udev/rules.d/90-rtc<x>.rules SUBSYSTEM=="rtc", KERNEL=="rtc<x>", SYMLINK+="rtc" EOF fi
name: <something>-gadget hooks: install: plugs: -rtc<x>-udev-service plugs: rtc<x>-udev-service: interface: system-files write: - /etc/udev/rules.d
Verify the solution is working
Plot boot-up sequence
systemd-analyze tool to generate a html based view of the boot sequence:
systemd-analyze plot > boot-sequence.html
The output html file can then be viewed in a web a browser:
Inspect the journal
Search for keyword rtc:
journalctl -b | grep rtc
The following are example result on AMD64 system with Ubuntu Kernel >v5.4 (snippet):
ubuntu kernel: Command line: snapd_recovery_mode=run console=ttyS0,115200n8 console=tty1 panic=-1 ubuntu_core.rtc_sys_time_init=/dev/rtc0 ubuntu kernel: rtc_cmos 00:04: registered as rtc0 [...] ubuntu systemd: Found device /dev/rtc0. ubuntu systemd: Starting Correct system time and sync to RTC time... [...] # This is the output of rtc-sys-time-init@dev-rtc<x>.service # The messages here will indicate if it was required to sync system # time to RTC (Ubuntu Kernel 5.4-) etc... [...] ubuntu rtc-sys-time-init: System time is ahead of most recent timestamp, skipping fixup ubuntu systemd: firstname.lastname@example.org: Succeeded. ubuntu systemd: Finished Correct system time and sync to RTC time.