PTP Quickstart Guide

There are many configurations for a PTP network, this quick start guide aims to cover the basics by using Ubuntu 18.04 as an example. It provides configuration settings for a commercial PTP grandmaster clock and also provides directions on setting up a Linux computer (Ubuntu 18.04) to function as a PTP grandmaster.

The linuxptp project provides a suite of PTP tools that can be used to serve as a PTP master clock for a local network of sensors.

Assumptions

  • Command line Linux knowledge (e.g., package management, command line familiarity, etc.).

  • Ethernet interfaces that support hardware timestamping.

  • Ubuntu 18.04 is assumed for this tutorial, but any modern distribution should suffice.

  • Knowledge of systemd service configuration and management.

  • Familiarity with Linux permissions.

Physical Network Setup

Ensure the Ouster sensor is connected to the PTP master clock with at most one network switch. Ideally the sensor should be connected directly to the PTP grandmaster. Alternatively, a simple layer-2 gigabit Ethernet switch will suffice. Multiple switches are not recommended and will add unnecessary jitter.

Third Party Grandmaster Clock

A dedicated grandmaster clock should be used for the highest absolute accuracy often with a GPS receiver.

It must be configured with the following parameters which match the linuxptp client defaults:

  • Transport: UDP IPv4

  • Delay Mechanism: E2E

  • Sync Mode: Two-Step

  • Announce Interval: 1 - sent every 2 seconds

  • Sync Interval: 0 - sent every 1 second

  • Delay Request Interval: 0 - sent every 1 second

For more settings, review the port_data_set field returned from the sensor’s HTTP /time/ptp interface.

Linux PTP Grandmaster Clock

An alternative to an external grandmaster PTP clock is to run a local Linux PTP master clock if accuracy allows. This is often implemented on a vehicle computer that interfaces directly with the lidar sensors.

This section outlines how to configure a master clock.

Example Network Setup

This section assumes the following network setup as it has elements of a local master clock and the option for an upstream PTP time source.

+-------------------------------------+
| Ubuntu 18.04 System                 |
| * 2x Intel i210 Ethernet Interfaces |
| * Linux PTP service                 |
|                                     |
|      eno1                 eno2      |
+-------+---------------------+-------+
        |                     |
+-------+-------+    +--------+------+
| Trimble GM100 |    |               + +
|  GPS -> PTP   |    |  Ouster OS1   | |
|  grandmaster  |    |               | |
|  (optional)   |    |               | |
+---------------+    +---------------- |
                      +--------------- +

The focus is on configuring the Linux PTP service to serve a common clock to all the downstream Ouster OS1 sensors using the Linux system time from the Ubuntu host machine.

Optionally, a grandmaster clock can be added to discipline the system time of the Linux host.

Installing Necessary Packages

Several packages are needed for PTP functionality and verification:

  • linuxptp - Linux PTP package with the following components:

    • ptp4l daemon to manage hardware and participate as a PTP node

    • phc2sys to synchronize the Ethernet controller’s hardware clock to the Linux system clock or shared memory region

    • pmc to query the PTP nodes on the network.

  • chrony - A NTP and PTP time synchronization daemon. It can be configured to listen to both NTP time sources via the Internet and a PTP master clock such as one provided by a GPS with PTP support. This will validate the time configuration makes sense given multiple time sources.

  • ethtool - A tool to query the hardware and driver capabilities of a given Ethernet interface.

$ sudo apt update
...
Reading package lists... Done
Building dependency tree
Reading state information... Done

$ sudo apt install linuxptp chrony ethtool
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  chrony ethtool linuxptp
0 upgraded, 3 newly installed, 0 to remove and 29 not upgraded.
Need to get 430 kB of archives.
After this operation, 1,319 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 ethtool amd64 1:4.15-0ubuntu1 [114 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 linuxptp amd64 1.8-1 [112 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 chrony amd64 3.2-4ubuntu4.2 [203 kB]
Fetched 430 kB in 1s (495 kB/s)
Selecting previously unselected package ethtool.
(Reading database ... 117835 files and directories currently installed.)
Preparing to unpack .../ethtool_1%3a4.15-0ubuntu1_amd64.deb ...
Unpacking ethtool (1:4.15-0ubuntu1) ...
Selecting previously unselected package linuxptp.
Preparing to unpack .../linuxptp_1.8-1_amd64.deb ...
Unpacking linuxptp (1.8-1) ...
Selecting previously unselected package chrony.
Preparing to unpack .../chrony_3.2-4ubuntu4.2_amd64.deb ...
Unpacking chrony (3.2-4ubuntu4.2) ...
Setting up linuxptp (1.8-1) ...
Processing triggers for ureadahead (0.100.0-20) ...
ureadahead will be reprofiled on next reboot
Setting up chrony (3.2-4ubuntu4.2) ...
Processing triggers for systemd (237-3ubuntu10.13) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up ethtool (1:4.15-0ubuntu1) ...

Ethernet Hardware Timestamp Verification

Identify the Ethernet interface to be used on the client (Linux) machine,

e.g., eno1. Run the ethtool utility and query this network interface for supported capabilities.

Output of ethtool -T for a functioning Intel i210 Ethernet interface:

$ sudo ethtool -T eno1
Time stamping parameters for eno1:
Capabilities:
        hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
        software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
        hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
        software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
        software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
        hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
        off                   (HWTSTAMP_TX_OFF)
        on                    (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
        none                  (HWTSTAMP_FILTER_NONE)
        all                   (HWTSTAMP_FILTER_ALL)

Configurations

Configuring ptp4l for Multiple Ports

On a Linux system with multiple Ethernet ports (i.e. Intel i210) ptp4l needs to be configured to support all of them.

boundary_clock_jbod 1
[eno1]
[eno2]

Note

Add the above required modification at the end of the existing file. Deleting or editing the default settings section of the ptp41.conf file will result in an error.

The default systemd service file for Ubuntu 18.04 attempts to use the eth0 address on the command line. Override systemd service file so that the configuration file is used instead of hard coded in the service file.

Create a systemd drop-in directory to override the system service file:

$ sudo mkdir -p /etc/systemd/system/ptp4l.service.d

Create a file at /etc/systemd/system/ptp4l.service.d/override.conf with the following contents:

[Service]
ExecStart=
ExecStart=/usr/sbin/ptp4l -f /etc/linuxptp/ptp4l.conf

Restart the ptp4l service so the change takes effect:

$ sudo systemctl daemon-reload
$ sudo systemctl restart ptp4l
$ sudo systemctl status ptp4l
* ptp4l.service - Precision Time Protocol (PTP) service
   Loaded: loaded (/lib/systemd/system/ptp4l.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/ptp4l.service.d
           └─override.conf
   Active: active (running) since Wed 2019-03-13 14:38:57 PDT; 3s ago
     Docs: man:ptp4l
 Main PID: 25783 (ptp4l)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/ptp4l.service
           └─25783 /usr/sbin/ptp4l -f /etc/linuxptp/ptp4l.conf

Mar 13 14:38:57 leadlizard ptp4l[25783]: [590188.756] port 1: INITIALIZING to LISTENING on INITIALIZE
Mar 13 14:38:57 leadlizard ptp4l[25783]: [590188.756] driver changed our HWTSTAMP options
Mar 13 14:38:57 leadlizard ptp4l[25783]: [590188.756] tx_type   1 not 1
Mar 13 14:38:57 leadlizard ptp4l[25783]: [590188.756] rx_filter 1 not 12
Mar 13 14:38:57 leadlizard ptp4l[25783]: [590188.756] port 2: INITIALIZING to LISTENING on INITIALIZE
Mar 13 14:38:57 leadlizard ptp4l[25783]: [590188.757] port 0: INITIALIZING to LISTENING on INITIALIZE
Mar 13 14:38:57 leadlizard ptp4l[25783]: [590188.757] port 1: link up
Mar 13 14:38:57 leadlizard ptp4l[25783]: [590188.757] port 2: link down
Mar 13 14:38:57 leadlizard ptp4l[25783]: [590188.757] port 2: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
Mar 13 14:38:58 leadlizard ptp4l[25783]: [590189.360] port 1: new foreign master 001747.fffe.700038-1

The above systemctl status ptp4l console output shows systemd correctly reading the override file created earlier before starting several seconds after the restart command.

The log output shows that a grandmaster clock has been discovered on port 1 (eno1) and port 2 (eno2) is currently disconnected and in the faulty state as expected. In the test network a Trimble Thunderbolt PTP GM100 Grandmaster Clock is attached on eno1.

Logs can be monitored (i.e. followed) like so:

$ journalctl -f -u ptp4l
-- Logs begin at Fri 2018-11-30 06:40:50 PST. --
Mar 13 14:51:37 leadlizard ptp4l[25783]: [590948.224] master offset    -17 s2 freq  -25963 path delay     14183
Mar 13 14:51:38 leadlizard ptp4l[25783]: [590949.224] master offset    -13 s2 freq  -25964 path delay     14183
Mar 13 14:51:39 leadlizard ptp4l[25783]: [590950.225] master offset     35 s2 freq  -25920 path delay     14192
Mar 13 14:51:40 leadlizard ptp4l[25783]: [590951.225] master offset    -59 s2 freq  -26003 path delay     14201
Mar 13 14:51:41 leadlizard ptp4l[25783]: [590952.225] master offset    -24 s2 freq  -25986 path delay     14201
Mar 13 14:51:42 leadlizard ptp4l[25783]: [590953.225] master offset    -39 s2 freq  -26008 path delay     14201
Mar 13 14:51:43 leadlizard ptp4l[25783]: [590954.225] master offset     53 s2 freq  -25928 path delay     14201
Mar 13 14:51:44 leadlizard ptp4l[25783]: [590955.226] master offset    -85 s2 freq  -26050 path delay     14207
Mar 13 14:51:45 leadlizard ptp4l[25783]: [590956.226] master offset    127 s2 freq  -25863 path delay     14207
Mar 13 14:51:46 leadlizard ptp4l[25783]: [590957.226] master offset      9 s2 freq  -25943 path delay     14208
Mar 13 14:51:47 leadlizard ptp4l[25783]: [590958.226] master offset    -23 s2 freq  -25973 path delay     14208
Mar 13 14:51:48 leadlizard ptp4l[25783]: [590959.226] master offset    -61 s2 freq  -26018 path delay     14190
Mar 13 14:51:49 leadlizard ptp4l[25783]: [590960.226] master offset     69 s2 freq  -25906 path delay     14190
Mar 13 14:51:50 leadlizard ptp4l[25783]: [590961.226] master offset    -73 s2 freq  -26027 path delay     14202
Mar 13 14:51:51 leadlizard ptp4l[25783]: [590962.226] master offset     19 s2 freq  -25957 path delay     14202
Mar 13 14:51:52 leadlizard ptp4l[25783]: [590963.226] master offset    147 s2 freq  -25823 path delay     14202
...

Configuring ptp4l as a Local Master Clock

The IEEE-1588 Best Master Clock Algorithm (BMCA) will select a grandmaster clock based on a number of masters. In most networks there should be only a single master. In the example network the Ubuntu machine will be configured with a non-default clockClass so its operation qualifies it to win the BMCA.

Replace the default value with a lower clock class (higher priority) and restart linuxptp. Edit /etc/linuxptp/ptp4l.conf and comment out the default clockClass value and insert a line setting it 128.

#clockClass     248
clockClass      128

Restart ptp4l so the configuration change takes effect.

$ sudo systemctl restart ptp4l

This will configure ptp4l to advertise a master clock on eno2 as a clock that will win the BMCA for an Ouster OS1 sensor.

However, the ptp4l service is only advertising the Ethernet controller’s PTP hardware clock, not the Linux system time as is often expected.

Configuring phc2sys to Synchronize the System Time to the PTP Clock

To synchronize the Linux system time to the PTP hardware clock the phc2sys utility needs to be run. The following configuration will tell phc2sys to take the Linux CLOCK_REALTIME and write that time to the PTP hardware clock in the Ethernet controller for eno2. These interfaces are then connected to PTP slaves such as Ouster OS1 sensors.

Create a systemd drop-in directory to override the system service file:

$ sudo mkdir -p /etc/systemd/system/phc2sys.service.d

Create a file at /etc/systemd/system/phc2sys.service.d/override.conf with the following contents:

[Service]
ExecStart=
ExecStart=/usr/sbin/phc2sys -w -s CLOCK_REALTIME -c eno2

Note

If multiple interfaces need to be synchronized from CLOCK_REALTIME then multiple instances of the phc2sys service need to be run as it only accepts a single slave (i.e. -c) argument.

Restart the phc2sys service so the change takes effect:

$ sudo systemctl daemon-reload
$ sudo systemctl restart phc2sys
$ sudo systemctl status phc2sys

Configuring Chrony to Set System Clock Using PTP

An upstream PTP grandmaster clock (e.g., a GPS disciplined PTP clock) can be used to set the system time if precise absolute time is needed for sensor data.

Chrony is a Linux time service that can read from NTP and PTP and set the Linux system time using the most accurate source available. With a properly functioning PTP grandmaster the PTP time source will be selected and the error from the public time servers can be reviewed.

The following phc2shm service will synchronize the time from eno1 (where the external grandmaster is attached) to the system clock.

Create a file named /etc/systemd/system/phc2shm.service with the following contents:

# /etc/systemd/system/phc2shm.service
[Unit]
Description=Synchronize PTP hardware clock (PHC) to NTP SHM
Documentation=man:phc2sys
After=ntpdate.service
Requires=ptp4l.service
After=ptp4l.service

[Service]
Type=simple
ExecStart=/usr/sbin/phc2sys -s eno1 -E ntpshm -w

[Install]
WantedBy=multi-user.target

Then start the newly created service and check that it started.

$ sudo systemctl start phc2shm
$ sudo systemctl status phc2shm

Add the PTP time source to the chrony configuration which will read the shared memory region managed by the phc2shm service created above.

Append the following to the /etc/chrony/chrony.conf file:

refclock SHM 0 poll 1 refid ptp

Restart chrony so the updated configuration file takes effect:

$ sudo systemctl restart chrony

After waiting a minute for the clock to synchronize, review the chrony client timing accuracy:

$ chronyc tracking
Reference ID    : 70747000 (ptp)
Stratum         : 1
Ref time (UTC)  : Thu Mar 14 02:22:58 2019
System time     : 0.000000298 seconds slow of NTP time
Last offset     : -0.000000579 seconds
RMS offset      : 0.001319735 seconds
Frequency       : 0.502 ppm slow
Residual freq   : -0.028 ppm
Skew            : 0.577 ppm
Root delay      : 0.000000001 seconds
Root dispersion : 0.000003448 seconds
Update interval : 2.0 seconds
Leap status     : Normal

$ chronyc sources -v
210 Number of sources = 9

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#* ptp                           0   1   377     1    +27ns[  +34ns] +/-  932ns
^- chilipepper.canonical.com     2   6   377    61   -482us[ -482us] +/-   99ms
^- pugot.canonical.com           2   6   377    62   -498us[ -498us] +/-  112ms
^- golem.canonical.com           2   6   337    59   -467us[ -468us] +/-   95ms
^- alphyn.canonical.com          2   6   377    58   +957us[ +957us] +/-   95ms
^- legacy13.chi1.ntfo.org        3   6   377    62    -10ms[  -10ms] +/-  178ms
^- tesla.selinc.com              2   6   377   128   +429us[ +514us] +/-   42ms
^- io.crash-override.org         2   6   377    59   +441us[ +441us] +/-   58ms
^- hadb2.smatwebdesign.com       3   6   377    58  +1364us[+1364us] +/-   99ms

Note that the Reference ID matches the ptp reference ID from the chrony.conf file and that the sources output shows the ptp reference ID as selected (signified by the * state in the second column). Additionally, the NTP time sources show a small relative error to the high accuracy PTP time source.

In this case the PTP grandmaster is properly functioning.

If this error is large, chrony will select the NTP time sources and mark the PTP time source as invalid. This typically signifies that something is mis-configured with the PTP grandmaster upstream of this device or the linuxptp configuration.

Verifying Operation

If the PTP grandmaster was just set up and configured, it’s recommended to power cycle the sensor. The sensor will then jump to the correct time instead of slowly easing in the time adjustment which will take time if the grandmaster initially set the sensor to the wrong time.

Sensor PTP Sync Verification

The sensor can be queried for the state of its local PTP service through the GET /api/v1/time/ptp request.

JSON response fields to check:

  • parent_data_set.grandmaster_identity should list the identity of the local grandmaster

  • port_data_set.port_state should be SLAVE

  • time_status_np.gm_present should be true

  • time_status_np.master_offset which is given in nanoseconds, should be less than 250000. This equates to 250 microseconds.

PTP Example JSON Response

{
  "profile": "default",
  "parent_data_set":
  {
    "grandmaster_identity": "001747.fffe.700038",
    "parent_port_identity": "ac1f6b.fffe.1db84e-2",
    "parent_stats": 0,
    "gm_clock_class": 6,
    "observed_parent_clock_phase_change_rate": 2147483647,
    "gm_clock_accuracy": 33,
    "gm_offset_scaled_log_variance": 65535,
    "grandmaster_priority1": 128,
    "grandmaster_priority2": 128,
    "observed_parent_offset_scaled_log_variance": 65535
  },
  "current_data_set":
  {
    "steps_removed": 1,
    "offset_from_master": 61355,
    "mean_path_delay": 117977.0
  },
  "port_data_set":
  {
    "port_state": "SLAVE",
    "peer_mean_path_delay": 0,
    "log_min_delay_req_interval": 0,
    "port_identity": "bc0fa7.fffe.c48254-1",
    "log_sync_interval": 0,
    "log_announce_interval": 1,
    "delay_mechanism": 1,
    "log_min_pdelay_req_interval": 0,
    "announce_receipt_timeout": 3,
    "version_number": 2
  },
  "time_status_np":
  {
    "gm_time_base_indicator": 0,
    "gm_identity": "001747.fffe.700038",
    "cumulative_scaled_rate_offset": 0,
    "scaled_last_gm_phase_change": 0,
    "ingress_time": 0,
    "master_offset": 61355,
    "last_gm_phase_change": "0x0000'0000000000000000.0000",
    "gm_present": true
  },
  "time_properties_data_set":
  {
    "frequency_traceable": 0,
    "leap61": 0,
    "time_traceable": 0,
    "current_utc_offset": 37,
    "leap59": 0,
    "current_utc_offset_valid": 0,
    "time_source": 160,
    "ptp_timescale": 1
  }
}

LinuxPTP PMC Tool

The sensor will respond to PTP management messages. The linuxptp pmc (see man pmc) utility can be used to query all PTP devices on the local network.

On the Linux host for the pmc utility to communicate with then run the following command:

$ sudo pmc 'get PARENT_DATA_SET' 'get CURRENT_DATA_SET' 'get PORT_DATA_SET' 'get TIME_STATUS_NP' -i eno2
sending: GET PARENT_DATA_SET
sending: GET CURRENT_DATA_SET
sending: GET PORT_DATA_SET
sending: GET TIME_STATUS_NP
        bc0fa7.fffe.c48254-1 seq 0 RESPONSE MANAGEMENT PARENT_DATA_SET
                parentPortIdentity                    ac1f6b.fffe.1db84e-2
                parentStats                           0
                observedParentOffsetScaledLogVariance 0xffff
                observedParentClockPhaseChangeRate    0x7fffffff
                grandmasterPriority1                  128
                gm.ClockClass                         6
                gm.ClockAccuracy                      0x21
                gm.OffsetScaledLogVariance            0x4e5d
                grandmasterPriority2                  128
                grandmasterIdentity                   001747.fffe.700038
        bc0fa7.fffe.c48254-1 seq 1 RESPONSE MANAGEMENT CURRENT_DATA_SET
                stepsRemoved     2
                offsetFromMaster 61355.0
                meanPathDelay    117977.0
        bc0fa7.fffe.c48254-1 seq 2 RESPONSE MANAGEMENT PORT_DATA_SET
                portIdentity            bc0fa7.fffe.c48254-1
                portState               SLAVE
                logMinDelayReqInterval  0
                peerMeanPathDelay       0
                logAnnounceInterval     1
                announceReceiptTimeout  3
                logSyncInterval         0
                delayMechanism          1
                logMinPdelayReqInterval 0
                versionNumber           2
        bc0fa7.fffe.c48254-1 seq 3 RESPONSE MANAGEMENT TIME_STATUS_NP
                master_offset              61355
                ingress_time               0
                cumulativeScaledRateOffset +0.000000000
                scaledLastGmPhaseChange    0
                gmTimeBaseIndicator        0
                lastGmPhaseChange          0x0000'0000000000000000.0000
                gmPresent                  true
                gmIdentity                 001747.fffe.700038

Tested Grandmaster Clocks

  • Trimble Thunderbolt PTP GM100 Grandmaster Clock
    • Firmware version: 20161111-0.1.4.0, November 11 2016 15:58:25

    • PTP configuration:

      > get ptp eth0
                   Enabled : Yes
                  Clock ID : 001747.fffe.700038-1
                   Profile : 1588
             Domain number : 0
        Transport protocol : IPV4
                   IP Mode : Multicast
           Delay Mechanism : E2E
                 Sync Mode : Two-Step
               Clock Class : 6
                Priority 1 : 128
                Priority 2 : 128
             Multicast TTL : 0
             Sync interval : 0
          Del Req interval : 0
             Ann. interval : 1
      Ann. receipt timeout : 3
      
  • Ubuntu 18.04 + Linux PTP as a master clock
    • Intel i210 Ethernet interface

    • PCI hardware identifiers: 8086:1533 (rev 03)

  • Ubuntu 18.04 kernel package: linux-image-4.18.0-16-generic

  • Ubuntu 18.04 linuxptp package: linuxptp-1.8-1