Distributed Perception

Distribution Perception is a configuration of perception that allows for the distribution of perception processing across multiple machines. This configuration is useful for the following applications:

  • Splitting the processing of perception across multiple machines

  • Enabling more networking options (Wi-Fi and less performant LAN’s) because the bandwidth of a node is much smaller than a lidar

  • Skips the clustering steps, since an aggregator listens to the output.

    • This is helpful because:

      • The cluster steps are a multi-threaded process, scaling with sensors, and consumes a lot of CPU.

      • Whereas, the processing of objects run on the aggregator is primarily single-threaded.

  • Allows for the duplication of sensor data to multiple machines. Ouster Gemini Detect does not support having multiple node instances connected to the same sensor.

Abbreviations:

  • A node is a perception instance, which contains at least a perception and web-gui instance. In the default installation, a node will have these two containers running.

  • An aggregator is a perception instance that is configured to receive data from nodes. An aggregator listens to the perception output from a node, while not changing any of the node's state.

Note

Important Notes to consider:

  • An aggregator instantiating a connection with a node will not affect the node’s state. The aggregator only listens to the node’s output, and does not set anything.

  • The perception output of the node may be slightly different than that of the aggregator. Each system is running its own process, which is the output of the objects.

  • Moving sensors between nodes: Remove the sensor from the node, and then add it to the new node. It is not recommended to add a sensor before removing it from the original node.

  • There will be a slight delay between the node and the aggregator. The aggregator is receiving the data from the node over a network, and there will be a delay between the node and the aggregator.

  • There is a setting in the message_aggregator called the buffer_duration. This specifies the amount of data to store within a buffer before being processed. A higher buffer will result in better time-synchronized results, however will introduce latency. The default value is 0 seconds, indicating that the data will be processed as soon as it is received from all sources.

License Requirements:

  • An aggregator’s connection to a node does not consume or require a license.

  • An aggregator can still have sensors attached to it, and those sensors will consume licenses.

Add a Node

  • IMPORTANT: Nodes must have publish_cluster_points set to true. Go to the setting of any node before you add it and change that to true.

  • IP address: The IP address of the node. This is the same IP address that is used to connect to the node’s web-gui and must be accessible from the aggregator. Note: There is no discovery for nodes, user must know the IP address of the node that is being added.

  • ID name: Is a user defined name for the node and is used to identify the node in the GUI. The alert system will also use this name to identify the node.

  • Username: The username to connect to the node. This is nginx basic authentication, and is the same username that is used to connect to the node’s web-gui. This by default will be ouster.

  • Password: This is the password to connect to the node. This is the password that is associated with the basic authentication that is used to connect to the node’s web-gui.

_images/node.png

Adding a Node

Please refer to the required information to add a Node:

  • Perform the alignment on the node to get its aligned cloud.

  • Open the setup page of the aggregator’s web-gui.

  • Click Add new source by hostname, and type in the ip address of the node. Click New Node to the other details of the node.

  • Fill in the other info for the node: ID name, username (default: Ouster), and password.

  • Click Add Node to add the node to the GUI.

Note

Failure Cases

  • The aggregator cannot reach the node’s IP address. The aggregator must be able to reach the node’s IP address to add the node.

  • The IP address does not belong to a node.

  • The node’s credentials are incorrect.

  • The node’s ID is not unique, or does not meet the ID requirement

Please make sure you have followed the above steps correctly and verify IP address, if the issue persists please contact Ouster Support.

Specifying a PORT Number

Ouster Gemini Detect assumes that the node's web-gui is running on port 443. If the user has changed the port within the compose file to be a different port, the user can specify a different port number using the swagger interface.

To specify a different port number, navigate to https://<machine-hostname>/perception/api/v1/swagger/ui, and go to the Node Management section. Click POST /perception/api/v1/node, and then click Try it out.

  • Example: Acceptable request body for a node that is running on port 8000, at IP address 10.0.0.2:

{
  "id": "node_1",
  "ip_address": "10.0.0.2",
  "username": "ouster",
  "password": "password",
  "port": 8000
}

Node ID Requirement

  • The node ID must be unique. If a node with the same ID is added, the node will not be added.

  • The node ID must be alphanumeric, and can contain spaces ( ), underscores (_), dashes (-), and periods (.).

    Any spaces will be converted to underscores (_).

Note

Important Considerations:

  • The node’s credentials are only used for initial connection. The credentials are not stored, and are not used to connect to the node after the node is added.

  • mDNS hostnames are not supported. The node must be added by IP address.

  • The username and password are the basic authentication credentials that are used to connect to the node’s web-gui. These credentials may not be the same as the credentials that are used to connect to the aggregator’s web-gui.

  • After initial connection, the node’s username and password can be changed without affecting the aggregator’s connection to the node.

Transforming a Node

The process of transforming a node is the same as transforming a sensor. When maneuvering the node, all the clouds will move together. Automatic alignment is possible with node clouds. To begin processing clusters and background clouds from a node, the transform must be set.

  • Moving the clouds on the Node will also move the clouds on the Aggregator. It will not occur the other way around.

  • To reduce the bandwidth of the raw clouds from the nodes, there exists an optional downsampling to reduce data throughput in the settings (/server/raw_cloud_publisher/downsample_factor).

Remove a Node

If the node is no longer needed or if a node was added incorrectly, the node can be removed from the aggregator.

Removing a node will remove the node from the aggregator’s GUI, and will stop the aggregator from listening to the node’s output.

Steps to remove a Node

  • Open the setup page of the aggregator’s web-gui.

  • Select the node that you want to in the left panel.

  • Click Remove Node to remove the node from the aggregator.

Credentials will need to be re-entered if the same node is added again.

Frequently Asked Questions (FAQs)

  • EventRecorder: The EventRecorder will work with distributed perception.

  • LidarHub: The LidarHub will operate on the aggregator’s output and not the node’s output. The alerts, zones, diagnostics, and telemetry will be based on the aggregator’s output, and not the node’s.

  • What is not forwarded to the aggregator?

    The node is not forwarding the following things to the aggregator:

    • Alerts

    • Zones

    • Diagnostics

    • Telemetry

    • Licensing

  • GET /perception/api/v1/sensor: The endpoint returns the sensors of the current system, and the sensors of the nodes. The sensors from the nodes will be populated with the node_id field.

Unsupported Features

  • Recording. Recording will only record the data of connected sensors, but not connected nodes.

  • Changing of the frame_rate without a restart. The frame_rate of a node is set when the node is started, and cannot be changed without restarting the node. To change the frame rate, set the frame rate in the node’s config file, and then restart the node.

  • Node to node to aggregator connections. A node cannot connect to another node, and then have the second node connect to the aggregator. The aggregator can only connect to nodes, and nodes can only connect to the aggregator.

  • Node discovery. The user must know the IP address of the node to add the node to the aggregator.

Possible Alerts

  • Failed to connect to raw frame websocket server: This error indicates that the aggregator’s websocket client was unable to connect to the node’s server which publishes the raw, complete point clouds.

  • Failed to connect to perception websocket server: This error indicates that the aggregator’s websocket client was unable to connect to the node’s server which publishes the perception output, which is the cluster data and the background data.

  • Unable to connect to node (REST server unresponsive): This error indicates that the aggregator’s REST client was unable to connect to the node’s REST server. This is the server that is used to access state information of the node, such as the node’s transforms and frame_rate.

  • Node authentication failed (remove and re-add node): This error indicates that the aggregator was unable to authenticate with the node. This error can occur if the node’s system was hard reset. The resolution to this error is to remove the node from the aggregator, and then re-add the node.

  • Connection to node’s raw websocket timed out: This error indicates that the aggregator’s websocket client timed out its connection to the node’s raw websocket server. This error can occur if the network connection between the aggregator and the node is unstable.

  • Connection to node’s perception websocket timed out: This error indicates that the aggregator’s websocket client timed out its connection to the node’s perception websocket server. This error can occur if the network connection between the aggregator and the node is unstable.

  • Node closed connection to raw websocket: This error indicates that the node closed its connection to the aggregator’s raw websocket server. This error can occur if the node’s system was shutdown.

  • Node closed connection to perception websocket: This error indicates that the node closed its connection to the aggregator’s perception websocket server. This error can occur if the node’s system was shutdown.

  • 2 or more nodes have the same hostname: This error indicates that 2 or more nodes have the same hostname. There is nothing stopping 2 or more nodes from having the same hostname, but this can cause confusion when trying to identify a node.