Time Synchronization (PTP)
This tutorial describes setting up Precision Time Protocol (PTP) for time synchronization on your robot. We assume the IEEE standard 1588 and that your robot is running some version of Linux. You can use the information in this guide to either synchronize two or more computers or a sensor (typically a lidar) with your computer. If you want to synchronize your computer with a global time, refer to the:
- NTP tutorial for internet sync
- PPS tutorial for GNSS sync
PTP is a time synchronization protocol that enables microsecond-level precision, which is a couple of magnitudes better than NTP. When synchronizing two clocks, we need to account for two error components:
- Clock offset
- Clock drift
PTP accounts for both by periodically exchanging data packets (typically over Ethernet) between the master and client devices. Clock synchronization is essentially a control problem, so don't be surprised when you encounter terms such as the PID controller when reading PTP specifications. As a general rule, you should not place any networking devices, such as routers and switches, between the devices you want to synchronize unless they are PTP-enabled. Switches and routers can introduce additional jitter and delays that are not constant, and the synchronization quality can degrade due to these factors.
Install necessary packages:
sudo apt update
sudo apt install ethtoolUse ethtool -T <interface_name> to check if an interface supports PTP. It should return something like:
Time stamping parameters for enp6s0:
Capabilities:
hardware-transmit
software-transmit
hardware-receive
software-receive
software-system-clock
hardware-raw-clock
PTP Hardware Clock: 3
Hardware Transmit Timestamp Modes:
off
on
Hardware Receive Filter Modes:
none
allSo, we see that both the hardware and software PTP clocks are supported.
If your interface only offers software-system-clock, you might still continue this guide.
However, note that you won't achieve the microsecond-level precision, so maybe consider using a different ethernet port.
If your interface supports neither software nor hardware timestamping, get a new computer.
Now that we know we can use PTP, download and install linuxptp. While it is possible to use apt package, the preferred way is to download the newest release from GitHub. apt repositories are unfortunately outdated.
git clone -b v4.4 [email protected]:richardcochran/linuxptp.git
cd linuxptp
make❗️For the sake of brevity, the command ptp4l will always be a shortcut for the ptp4l executable located in the directory where you installed linuxptp, e.g. /home/user/libraries/linuxptp/ptp4l.
Devices connected to the same network and using PTP decide which one is the Master clock with a priority parameter, located in a configuration file. Copy the file located in linuxptp/configs/default.cfg to your desired location, for example norlab_robot/scripts/config/ptp4l.conf.
Open the file and change the value of the clockClass parameter:
clockClass 256 -> 128
The lower number means a higher priority. Since we will not modify the clockClass value for any other devices, this computer will become the Master device.
ℹ️ Always check your sensor's documentation for the exact priority number it uses. While most sensors use the default value of 256, manufacturers sometimes have different ideas.
With the configuration file updated and saved, we can launch our ptp4l instance with
# hardware-timestamping
ptp4l -i <your_interface> -m -l 7 -f norlab_robot/scripts/config/ptp4l.confUpdate the interface value to match your computer. The -m -l 7 flags are for debugging. Finally, we specify that we want to use the configuration file we just modified. If your interface supports only software-clock add the -S flag:
# software-timestamping
ptp4l -i <your_interface> -m -l 7 -f norlab_robot/scripts/config/ptp4l.conf -SWhile the ptp4l command manages the PTP protocol on our specified network interface, it typically uses the hardware-clock soldered on that interface. We also need a way to synchronize this hardware-clock with our Linux system time.
ℹ️ You can skip this section if you only use the
software-clockoption.
phc2sys is another utility that we installed as part of the linuxptp package.
It is a program that synchronizes two or more clocks, in our case, the system time and the hardware-clock on the ethernet interface.
Again, from now on, the command phc2sys refers to the phc2sys executable located in the directory where you installed Linux PHP, e.g., /home/user/libraries/linuxptp/phc2sys.
Run
phc2sys -w -s CLOCK_REALTIME -c <your_interface> -R 100
to synchronize time between the source system clock CLOCK_REALTIME and the client clock on your ethernet interface.
The -w flag makes the service wait for ptp4l.
The synchronization will run at 100 Hz. Now, our time synchronization diagram looks something like this:
Linux -> (phc2sys) -> ethernet interface -> (ptp4l) -> network
In the next section, we will add a client device connected to the network that uses the master's clock as a time reference.
Depending on what your client device is, you will need to take different steps to make time synchronization work on the other side.
In this case, follow all the steps you did for the Master computer.
However, do not modify the clockClass value. Instead, change the clientOnly flag:
[global]
clientOnly 0 -> 1
and run ptp4l on your client computer with
ptp4l -i <your_interface> -m -l 7 -f norlab_robot_on_computer2/scripts/config/ptp4l.confIf you need to synchronize time between the hardware-clock on the ethernet interface and the Linux system time, use phc2sys again, you invert the source and client devices:
phc2sys -w -s <your_interface> -c CLOCK_REALTIME -R 100
Now, our network diagram may look like this:
Linux -> (phc2sys) -> ethernet interface -> (ptp4l) -> ethernet -> (ptp4l) -> ethernet interface -> (phc2sys) -> Linux
---------------------Master--------------------------- ---------------------Client------------------------
Depending on the sensor you use, you will need to activate PTP in either the sensor's web interface, configuration app, when flashing firmware etc. With the sensor being connected to our master computer, the time synchronization should work without any further tweaking. Most sensors also provide some kind of status and feedback on whether the time synchronization worked or not. Synchronizing a single sensor with our computer, the network diagram might look like this
Linux -> (phc2sys) -> ethernet interface -> (ptp4l) -> ethernet -> ??? -> sensor's internal clock
---------------------Master--------------------------- ------------Sensor------------
Having multiple devices that all need a consistent time source is a common scenario in robotics. For example, we might have a robot with multiple computers and multiple sensors. For simplicity, we will assume that all sensors supporting PTP are connected directly to the Master computer, e.g., there are no intermediate layers. In PTP terminology, these intermediate layers are called Boundary clocks.
ℹ️ Alternatively, you can also get a PTP-enabled switch and connect all devices to it. We received recommendations for this brand.
Following our previous advances, let's say we want to combine the case with a secondary computer and a sensor, both synchronized with the Master clock. Our desired network diagram might look like this:
---------------------Master--------------------------- ---------------------Client------------------------
Linux -> (phc2sys) -> ethernet interface1 -> (ptp4l) -> ethernet -> (ptp4l) -> ethernet interface -> (phc2sys) -> Linux
Linux -> (phc2sys) -> ethernet interface2 -> (ptp4l) -> ethernet -> ??? -> sensor's internal clock
------------Sensor------------
Easy, right? Well, unfortunately not, because we can't have multiple instances of ptp4l running at the same time.
Instead, we will need to go back to the Master ptp4l configuration file. At the end of the file, add all interfaces you want to use:
# Client computer
[ethernet interface1]
# Sensor
[ethernet interface2]
Then, run the ptp4l command, this time without explicitly specifying the interface name, as we do it in the config file instead.
# hardware-timestamping
ptp4l -m -l 7 -p /dev/ptp2 -f norlab_robot/scripts/config/ptp4l.confNotice the -p /dev/ptp2 flag. This tells ptp4l that all ethernet interfaces use the same hardware-clock.
Despite this, based on my experience, you will still need to run a phc2sys instance for each of your devices.
ℹ️ If you use
software-timestamp, no-pflag orphc2sysare needed.
Sometimes, your sensors require different PTP parameters. In that case, you can also add interface-specific parameters to the config file:
# radar has no extra config
[enp1s0f0]
# lidar has a different delay mechanism
[enp5s0]
delay_mechanism P2P
For a full ptp4l config file example, check the Warthog high-level file here.
When you are confident in your system's ability to synchronize time, you can set up all commands to start on startup with systemd services.
🆘 If you need some more help figuring out why your clocks don't sync, check the debugging section at the end of this guide.
Create a service file:
sudo touch /etc/systemd/system/ptp4l.serviceWith the following content:
[Unit]
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=ptp4l -m -l 7 -p /dev/ptp2 -f norlab_robot/scripts/config/ptp4l.conf
[Install]
WantedBy=default.target
‼️ Theptp4lcommand must be replaced by the full path to yourlinuxptpinstallation directory
Similarly, we will create a service file for each of our phc2sys instances:
sudo touch /etc/systemd/system/phc2sys_device1.serviceWith the following content:
[Unit]
After=ptp4l.service network-online.target
Wants=ptp4l.service network-online.target
[Service]
ExecStart=phc2sys -w -s CLOCK_REALTIME -c <your_interface1> -R 100
[Install]
WantedBy=default.target
‼️ Once againphc2syscommand must be the full path to the executable inside of thelinuxptpdirectory
Restart the systemd daemon and enable the newly created services with
sudo systemctl daemon-reload
sudo systemctl enable ptp4l phc2sys*
sudo systemctl restart ptp4l phc2sys*Now repeat the procedure on your client computers.
🎉🎉🎉 Congratulations, you successfully completed the PTP time synchronization guide!
Now that your robot is locally synchronized check the NTP and PPS guide to learn how to get a globally consistent time, either from the internet or a GNSS receiver.
You can also continue reading to learn more about leap seconds and why your devices (might) have a constant offset of about 38 seconds.
If you continue scrolling, you find some useful debugging tips and tricks.
But before that, here are some relevant links where you can find out more about this fascinating subject:
- Manpage ptp4l
- Manpage phc2sys
- Ouster has a good starter guide
- RedHat has a similar guide
- Stereolab's page on time sync
ptp4l with hardware timestamps announces the time on the PTP scale.
This scale is continuous and corresponds to the TAI scale (atomic).
The PTP hardware clock needs to be synchronized with the system clock, which is the role of the phc2sys utility.
However, system time uses the UTC standard, which is off from TAI by a few tens of seconds.
Here's the current offset value.
The whole difference is caused by leap seconds.
While the rate rotates at approximately 86400 SI seconds per day, it is actually slightly slower (86400.001, for example).
Due to this, the TAI time advances faster and occasionally a leap second must be added to the UTC timestamp to keep the difference to TAI under 0.9 seconds.
While this shouldn't influence you if all your computer use a recent linuxptp version, with some sensors, you might need to set a constant offset manually in your phc2sys command with the -O flag.
A good way to debug that the time synchronization works is to change the time on the Master device and watch how the client devices react. Before starting, make sure your device is unplugged from the internet so no internet-based time synchronization can take place while we do our tests. On your Master computer, open a terminal window with two panes. In one of the panes, run
watch -n 0.1 date +"%T.%6N"This will monitor the current system time in nanoseconds. Keep the other pane open for now.
If your other device is a client computer, open three panes in a new terminal window. Use them to launch:
sudo journalctl -u phc2sys.service -e -f
sudo journalctl -u ptp4l.service -e -f
watch -n 0.1 date +"%T.%6N"While the first two commands watch our two services in a live mode, the last one again reports the system time in nanoseconds.
Now, in the unused master computer pane, set the current time to a value that's slightly offset from now(), for example, one hour off:
# modify the date to be no more than 1 hour off the current system time
sudo date --set="2024-11-14 15:50:00.000"and observe what's going on on the other computer.
To sync the time back after connecting to the internet, use sudo systemctl restart ntpsec.
Similarly, one can debug any sensors with ROS drivers by running
ros2 topic echo <topic_name> --field=header.stampand verifying that the stamp time change follows the computer's clock.
Use Wireshark to check if there's any PTP communication at all on your interface. In the filter config, ip.addr == <your_source_ip> && ptp to specify the device. You should see packets coming from both devices.