Precision Time Protocol (PTP) and Packet Timestamping in Linux

---

Farnam Adelkhani - Sept. 2022 - [email protected]

---

• PTP is used for event ordering when working with multiple machines.
• With PTP the user can determine the ordering between logs and transactions.
• Ordering is based on timestamps and can be collected from any collection of machines.
• The timestamp is generated by the system clock of each machine.
• The clocks need to all be syncronized in frequency, phase and time for PTP to work.

---

PTP - Sub-microsecond accuracy on local networks.

• Standardized in three publications IEEE 1588-2002, IEEE 1588-2008, and IEEE 1588-2019

---

Links:
ptp4l(8) - Linux man page

• ptp4l [ -A | -E | -P ] [ -2 | -4 | -6 ] [ -H | -S | -L ] [ -f config ] [ -p phc-device ] [ -s ] [ -l print-level ] [ -q ] [ -v ] [ -i interface ] ...

Time Synchronization in modular collaborative robots

• The article above shows how using a tiny and inexpensive AMD Zynq 7000 1-based SoM module you can obtain a) sub-microsecond clock synchronization accuracy, b) ROS 2.0 timestamping accuracies below 100 microseconds and c) bounded end-to-end communication latencies between 1-2 milliseconds.

Introduction to PTP - PTP Basics
The Linux PTP Project
PTP Best Practices
Wireshark - Opensource Network Packet Analyzer
Precision Time Protocol Industry Profile (wiki)

Forums (Highly informative):
Experience with PTP (Precision Time Protocol) for mobile robots

Downloads:
linlinuxptp_3.1-2.1_arm64.deb Debian 11 Download

Hardware timestamping is called GTE 3, in the Xavier & Orin SOC’s as a feature of Jetson and DRIVE platforms.

---

Q: How are the system clocks syncronized?

NTP - Network time protocol
- Provides accuracy down to the millisecond, in the best case.
- Not good if events are occuring within milliseconds of each other.

Hierarchical, leader/follower architecture for clock distibution.
- Leader is the "Grandmaster" clock.
- Boundary and follower clocks are called "slaves".

PTP packets may be transmitted over Ethernet or UDP, over IPv4/IPv6, using multicast or unicast addresses.
- Depending on the publication used as reference, not all modes are available.

---

Leader clock
- Time source for the PTP network.
- Usually synchronized its clock to an external source (GNSS for example.
- Is an "ordinary clock", has a single PTP network connection.

Boundary clock
- Has multiple PTP network connections and relay accurate time:
- Keeps its clock synchronized against the leader.
- Acts as a clock source for the followers.
- Can become the leader if current leader disappears.
- Having a boundary clock is optional.

Follower clock
- Synchronize its clock to a leader (the boundary clock if there is one).
- May become the leader if the leader disappears.
- Is an "ordinary clock" (has a single PTP network connection).

---

PTP synchronization mechanism

The key piece of information to get is the offset between the leader and follower, such that the follower can adjust its local clock.
All of this computation will be done by the followers and not the leaders.
The leader will send sync packets, we want to compute when the sync packet will be sent and received.

Timestamps made on the leader side are sent to the followers by means of follow up packets (follow_up, delay_resp).

\begin{align*} d_{tf} &= t_1 - t_0 + \delta t \ d_{fl} &= t_3 - t_2 + \delta t \ \end{align*}

We assume the two trip times are equal, hence we have: \begin{align*} \delta t &= \frac{1}{2} (t_1 - t_0 + t_2 - t_3)\ \end{align*}

With this delta time the PTP implementation will be able to use an algo or filter to know what sort of adjustment to make on the local clock.
Should the frquency be changed, the phase adjusted? The key element for this calculation will be the delta between the two clocks.

---

PTP can work in two operating modes: 1-step and 2-step

1-step mode:

• Includes t₀ in the sync packet. There is no follow_up packet.
  • The difference here is on the leader's side. It requires a hardware enabled device to include t₀ in the sync packets.
  • All the followers should support both modes: there is no hardware requirement for receiving 1-step sync packets.

2-step mode: Is assumed to be processed in software.

Depending on what's available on our machine, we can select 1-step or 2-step modes.

---

The major takeaway should be the importance of the delta, which will ultimately be the parameter for us to calculate our clock correction for all the slave clocks in the network.

---

Ensuring timestamp accuracy

• Packet timestamps, when used for PTP, must be accurate as they play a critical role in the time offset computation.
• Ideally, a timestamp should be issued right at the exact time of transmission, when the packet leaves the device.
  • This is not actually possible, since the timestamp must occur before transmission.

Two possibilities to do this, in software or hardware...

Software timestamping:

Timestamping is done in the application or the kernel, far away from the actual transmission.
Uses the system clock, thus the errors and deltas are big:
- A lot can interfere: scheduling, queuing, interrupts...

If the timestamping is done in sofware, it will be done far

Hardware timestamping:

Timestamp is done in the hardware level.

• Can be done in the MAX
• In a PHY
• or using a dedicated controller.

A hardware solution would use a PTP hardware clock (PHC), included in the engine that is performing the timestamping.
Errors and deltas are much smaller with hardware timestamping.

• Timestamp occurs close to the actual transmission.
• The packet is already in the hardware.

Hardware solution is preferred as we can avoid any software related errors that are introduced with improper scheduling.

Two mechanisms are combined to provide support for offloading PTP packets timestamping:

• The S0_TIMESTAMPING socket option.
• The PTP hardware clock (PHC) infrastructure.

---

S0_TIMESTAMPING

Configured using setsockopt.
Generates timestamps on receoption, transmission, or both.
Works for streams and datagrams.

Supports multiple timestamp sources:

• SOF_TIMESTAMPING_RX_SOFTWARE timestamp is generated just after the network device driver hands the packet to the Rx stack.
• SOF_TIMESTAMPING_TX_SOFTWARE timestamp is generated in the network device driver, as close as possible to passing the packet to the hardware. Requires driver support and may not be available for all devices.
• SOF_TIMESTAMPING_RX_HARDWARE requires driver support.
• SOF_TIMESTAMPING_TX_HARDWARE requires driver support.
• and two other options not used for PTP applications:
  • SOF_TIMESTAMPING_TX_SCHED and SOF_TIMESTAMPING_TX_ACK

Hardware timestamping must be initialized for each device driver expected to be used.

Configuration passed using the SIOCSHWTSTAMP ioctl. Must choose a tx_type and an rx_filter.

• Possible values for tx_type:
  • HWTSTAMP_TX_OFF
  • HWTSTAMP_TX_ON: report timestamps through the socket error queue.
  • HWTSTAMP_TX_ONESTEP_SYNC: insert timestamps directly into sync packets.
  • HWTSTAMP_TX_ONESTEP_P2P: same as before but also insert timestamps into delay_resp packets.

Possible values for rx_filter:
- HWTSTAMP_FILTER_NONE
- HWTSTAMP_FILTER_ALL
- HWTSTAMP_FILTER_PTP_V2_L2_EVENT: PTP v2, Ethernet, all event packets.
- HWTSTAMP_FILTER_PTP_V2_L4_SYNC: PTP v2, UDP sync packets.
- For the full list, see include/uapi/linux/net_tstamp.h

---

ethtool -T eth0

Time stamping parameters capabilities for eth0:
- hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
- software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
- hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE)
- software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
- software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
- hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)

PTP Hardware Clock: 0

Hardware Transmit Timestamp Modes:
- off (HWTSTAMP_TX_OFF)
- on (HWTSTAMP_TX_ON)
- one-step-sync (HWTSTAMP_TX_ONESTEP_SYNC)

Hardware Receive Filter Modes:
- none (HWTSTAMP_FILTER_NONE)
- all (HWTSTAMP_FILTER_ALL)

---

In a networking Ethernet driver (MAC), implementing:

• get_ts_info in struct ethtool_ops

• ndo_do_ioctl in struct net_device_ops, for SIOCSHWTSTAMP and SIOCGHWTSTAMP.

• Filling hwtstamps in struct skbuff with Rx timestamps.

• Calling skb_tstamp_tx() when a Tx timestamp is reported by the hardware.

• In a networking PHY or other dedicated engines driver: implementing the struct mii_timestamper callbacks, in struct phy_device:

  • ts_info
  • hwtstamp
  • rxtstamp
  • txtstamp and calling skb_complete_tx_timestamp()

Both interfaces allow us to:

1. Report the timestamping capabilities (ts_info and get_ts_info).
2. Configure the mode to use (hwtstamp and ndo_do_ioctl).
3. Report Rx timestamps (rxtstamp and hwtstamps).
4. Report Tx timestamps (txtstamp/skb_complete_tx_timestamp() and skb_tstamp_tx).

• PHC are used by hardware engines to timestamp packets.

• The PHC must be synchronized.

• Described by struct ptp_clock_info, which embeds operation callbacks:

• gettimex64: reports the current time from the hardware clock by filling a timespec64 structure.

• settime64: sets the time on the hardware clock.

• adjfine: adjusts the frequency of the hardware clock, using an ”offset from nominal frequency in parts per million, but with a 16 bit binary fractional field”.

• adjtime: shifts the time of the hardware clock by an s64 delta.

• adjphase: adjusts the phase of the hardware clock by an s32 phase.

• The structure also contains a few parameters, including max_adj which defines the maximum frequency adjustment in parts per billion.

• At Bootlin, we had the opportunity to introduce PTP offloading support for some hardware engines.

• For the Microsemi Ocelot network switch:

• drivers/net/ethernet/mscc/

• For the Microsemi VSC85xx PHYs:

• drivers/net/phy/mscc/mscc_ptp.c

• The Linux PTP project is an implementation of the Precision Time Protocol according to IEEE-1588, for Linux.

• Maintained by Richard Cochran, who also maintains PTP support in Linux.

• Provides a reliable implementation of PTP for Linux, and correctly uses the kernel interfaces for PHC and timestamping.

• Provides a few utilities including ptp4l and phc2sys.

Implementation of PTP, for ordinary and boundary clocks.

• Can use software or hardware timestamping.
• Can perform PTP operations on top of UDP (IPv4/IPv6) and Ethernet.
• Can optionally use a configuration file.

ptp4l -i eth0 -H -2 -m

selected /dev/ptp4 as PTP clock
port 1: INITIALIZING to LISTENING on INIT_COMPLETE  
port 0: INITIALIZING to LISTENING on INIT_COMPLETE  
port 1: new foreign master 7e7618.fffe.b52b26-1  
selected best master clock 7e7618.fffe.b52b26  
port 1: LISTENING to UNCALIBRATED on RS_SLAVE  
master offset 1949 s0 freq -552 path delay 2807  
master offset 1953 s2 freq -548 path delay 2807  
port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED  
master offset 1974 s2 freq +1426 path delay 2807  
master offset 629 s2 freq +673 path delay 2807  

• Synchronizes two (or more) clocks.
• Typically used to keep the system clock in sync with the PHC.
• When using hardware timestamping, ptp4l adjusts the PHC and then phc2sys adjusts the system clock.
• When using software timestamping, phc2sys isn’t used; the system clock is directly adjusted by ptp4l.

---

PTP offloading support in Linux

What can be done in the Linux kernel to provide PTP offloading to user space applications?

• More info: https://www.kernel.org/doc/html/v5.4/driver-api/ptp.html

What are Pcaps?

• Packet capture is a networking practice involving the interception of data packets travelling over a network.

ptp4l is an implementation of the Precision Time Protocol (PTP) according to IEEE standard 1588 for Linux.

sudo ptp4l -i enp4s10f6 -m -H -s

enp0s10f1:
en => ethernet
p0 => bus number (0)
s10 => slot number (10)
f1 => function number (1)

---