General ‐ Time Synchronization

PTP

PTP is the main way the computers on the box are synced. The PTP configuration of each computer is managed via Linux Services that run on startup and decide which computer(s) will be the master time keeper. A introduction can be found here.

Overview

Description

• The CPT7 is the PTP master.
• The Jetson functions as a chrony server.
• If a jump in time is detected above 1s the ptp4l phy2sys and chronyd are configured to adjust the clock via a jump.
• The alphasense is directly connected to the NUC therefore a special configuration is set to sync the two ethernet interfaces.
• Everything else is quite standard.

Working without GPS

To set the time of the CPT7 which starts in 1980 if no GPS is available you can make use of boxi set_cpt7_time --ros. For this to work ensure l-nuc, l-opc is running. The script fetches the current time from the internet ant sets it via a rosservice call to the driver of the CPT7 - this is why one needs to run l-nuc. Alternatively one can also manual specify the time (read description of boxi).

All the instructions below are outdated

NTP

To reset the npt time (from the internet) on the opc, run service ntp restart. ntpq -p shows the status of the npt servers.

Startup Flow

• The Jetson first runs a custom service sync_time_once.service, which gets the RTC time from the jetson hardware and sets the Jetson system clock to match.

Note: The RTC time drifts. Ideally we would get the internet time instead and fall back to RTC time. EDIT: This might be what chrony is doing, need to check with Beni.

• After setting the system time, the ptp4l_mgbe[i].services (eg: here) start ptp using the configuration files in /etc/linuxptp/ptp4l.conf. These config files are just the linux default and are all the same across the box. This results in both network interfaces (mgbe0 and mgbe1) becoming ptp masters - note: they don't automatically talk to or sync with each other.

• Then, the phc2sys_mgbe[i].services set up clocks within the system to synchronise to the mgbe0 time, so that the different interfaces match.

ExecStart=/usr/sbin/phc2sys -w -s mgbe0 -c CLOCK_REALTIME -> Makes the system clock track the time on the mgbe0 network interface.

ExecStart=/usr/sbin/phc2sys -w -s mgbe0 -c mgbe1 -> Makes the mgbe1 network interface time track the time on the mgbe0 network interface.

• At this point, all the network interfaces and internal system clocks on the jetson should be synchronised to each other, and ptp is advertising jetson as a ptp master (there can be multiple).

Logs

See ptp logs from the most recent boot. Grep for 'master' to validate where the ptp master is assigned.

journalctl -u ptp4l_mgbe0.service -b

journalctl -u ptp4l_mgbe1.service -b

Nuc PTP

The Nuc has a similar setup to the Jetson, except that the ptp command in ptp4l_enp45s0.service forces the enp45s0 into slave mode with the -s flag.

enp45s0 interfaces with the Jetson, meaning it will then take the Jetson as time sync master and track Jetson time.

ptp4l_enp46s0.service doesn't have the same -s flag because it doesn't interface to the Jetson, so it can be a master. Instead, the phc2sys_NIC.service syncs the enp46s0 network interface time to the system time, which is in turn synced via phc2sys_system.service to the enp45s0 network interface time, which is tracking the Jetson. This means enp46s0 is tracking the Jetson time, but can be a master on it's own network interface, allowing others to sync via it to the jetson.

Logs

See ptp logs from the most recent boot. Grep for 'master' to see validate the ptp master is assigned.

journalctl -u ptp4l_enp45s0.service -b

journalctl -u ptp4l_enp46s0.service -b

OPC PTP

Similar to the Nuc, the OPC has a PTP service that starts in slave mode, in order to track the Jetson.

Note: this service isn't checked into box_configuration on git. We need to do that, otherwise new OPCs can become the PTP master and override the Jetson.

Logs

See ptp logs from the most recent boot. Grep for 'master' to validate where the ptp master is assigned.

journalctl -u ptp4l.service -b

Troubleshooting

Diagnosing Linux Service Start-Up

You can get a nice waterfall graph of the services starting up and their blocking dependencies via:

systemd-analyze plot > systemd-plot.svg

Then scp the plot to the opc and examine it with xdg-open systemd-plot.svg

Previous boot-up time troubleshooting issue with debugging steps here.

Demystifying Clock IDs/Addresses

In the ptp4l logs, the clocks are identified via strings that are hard to parse in order to diagnose where a clock is located that ptp is syncing to.

A useful tip is that the clock id that ptp4l logs as the new master has a component matching the HWaddress of its location. Running arp lets you see the devices on the network and their HWaddresses. From here you can find the one matching the id in the ptp4l logs to find the location of the clock.