Troubleshooting

Accessing logs

The last 7 days of Gemini Detect logs can be accessed by clicking the View Logs button in the Diagnostics tab.

The logs page lists all log files from the last 7 days. Files follow the format {service}-{day_of_week}.log. Where the service is one of the various docker containers which make up the Gemini Detect system. If logs are being supplied to Ouster for support, please download a log from each service on the day of the week the issue occurred. For example, if the issue occurred on a Tuesday, download the logs for each service with the format {service}-tue.log. Simply clicking the filename will download the log file.

Note

Logs are stored in UTC time and this should be taken into account when determining which logs to download.

Other stored data can be accessed using the data directory at: https://<base_url>/data

Alert Data

Ouster Gemini Detect exposes alerts to the user to communicate when the system is in an error state or when an aspect of the system should be investigated. The user can query the alerts through the RESTful interface as well as view them through the web page. The underlying format of the alerts is in JSON and is very similar to the format defined in the Ouster Firmware User Manual.

The table below lists the alerts in Detect along with their alert code, description, and guidance on what could be happening to cause it.

Alert list
Alert Code	Name	Severity	Description/Possible causes	Recommended action
0x2000001	Lidar timeout	CRITICAL	The system is not receiving data from a lidar that’s been configured. Possible causes are the UDP destination has been changed, the network infrastructure has been severed, or the lidar hardware has failed.	Investigate the state of the lidar by navigating to the IP address directly (If the lidar is directly connected to the catalyst, use the sensor proxy feature in the UI). Check the UDP destination address and confirm it’s pointing to the Catalyst. The lidar’s serial number will be in this alert’s source information.
0x2000002	Dropping lidar data	WARNING	The system is not keeping up with lidar data it’s ingesting	Check to make sure your compute device is within the recommended specifications for the number of lidars in your setup.
0x2000003	Dropping cluster	WARNING	The system is not able to process clusters fast enough	This is most likely an issue with the number of moving objects the system is trying to track in the scene. Check to see if this number is correlated with a high volume of moving objects
0x2000004	Cluster delta over threshold	WARNING	The timestamps of data from all lidar sources are over the configurable threshold. This will likely cause issues with tracking since the system is attempting to process data from multiple lidars which are unsynchronized.	Make sure all sensors have a stable fast connection to the Catalyst. If this does not resolve the issue, contact customer support about putting the connected sensors in phase lock.
0x2000005	Websocket client behind	WARNING	A client connected to the websocket server is not keeping up with the data behind sent. The system is dropping data as a result	The source information field of this alert will contain the IP address of the client which is not keeping up. Check the network bandwidth between the Catalyst and this client
0x2000006	TCP client behind	WARNING	A client connected to the TCP server is not keeping up with the data behind sent. The system is dropping data as a result	The source information field of this alert will contain the IP address of the client which is not keeping up. Check the network bandwidth between the Catalyst and this client
0x2000007	Invalid packet source	ERROR	The system is receiving data from an an expected IP address on a network port configured to receive lidar data	The cause of this alert is most likely a configuration issue with one of the lidars. This occurs when a lidar was previously configured to send it’s data to a Catalyst unit but the Catalyst unit is expecting data from another lidar now on that port. Navigate to the IP addresses of all lidars on the network and confirm any unused lidars are not sending their data to the Catalyst
0x2000008	Lidar packet mismatch	WARNING	The system is receiving data of unexpected size on a port it’s expecting lidar data on. This is usually configuration issue related to UDP profiles. There is likely a difference between the UDP profile Detect is expecting and the UDP profile configured on the lidar.	The cause of this alert is most likely a configuration issue with one of the lidars. Navigate to the IP address of the browser of the lidar in source info and confirm the UDP profile is the one Detect is configured for (Default is RNG15_RFL8_NIR8)
0x2000009	Object Pipeline slow	WARNING	The object pipeline is not operating at the desired frequency (10 Hz or 20 Hz)	The edge computer is not keeping up with the number of objects. It’s recommended to look into if there are other processes on the edge computer or if the number of objects is higher than expected.
0x200000A	Unsupported sensor mode	CRITICAL	Detect is trying to process data from a lidar in an unsuppored mode	Check to make sure the lidar is in a supported mode
0x200000B	Sensor connection failed	CRITICAL	Detect failed trying to connect to a sensor	Check the network connection between the edge computer and the lidar
0x2000016	Near sensor blockage	CRITICAL	An object has been detected which is blocking the lidar	Check for any obstructions which are close to the lidar and blocking its field of view
0x2000017	Sensor tilted	CRITICAL	Tilt in the sensor has been detected since the user last performed an alignment	Check the lidar mount to make sure it hasn’t moved from the desired position and orientation.
0x2000018	License issue	CRITICAL	A licensing error has been found. See the perception logs for more details	Make sure your license has not expired. If you have a valid license, check the perception logs and send them to support@ouster.io
0x2000019	Incomplete frame	ERROR	Detect has calculated that the number of incomplete frames is over the threshold.	Validate that the network connection between the lidar and the edge computer is operational and supports the bandwidth requirements of the lidar

Detect also forwards alerts from the lidar. We distinguish between lidar alerts and Detect’s alert with the 4th byte (0x01XXXXXX indicates a lidar alert, 0x02XXXXXX indicates a Detect alert). See the Ouster Firmware User Manual for a description of the lidar alerts.

Detect alerts go from RESET to ACTIVE when the condition for the alert is satisfied. Once the condition becomes false, the alert state goes from ACTIVE to LOGGED. Users can query the ACTIVE, LOGGED, or all alerts with the endpoints defined in the diagnostic section.

The table below describes the information returned when querying alerts from Detect.

Alert fields
Field	Type	Description
active	boolean	Whether the condition enabling this alert is current true
active_count	integer	How many times this alert has been true since it first occurred
alert_code	string	Hexidecimal identifier for the alert type. See table above
category	string	Module where the alert exists
first_occurred	integer	Timestamp when alert first occurred, in microseconds since Jan. 1, 1970
last_occurred	integer	Timestamp when alert last occurred, in microseconds since Jan. 1, 1970
source_info	string	String identifier for alert instance. For example, in a multi-lidar setup we can have alerts for different instances processing two different lidars. This field distinguishes between those instances such that we can have more than one active alert with the same alert_code. In the case where we want to distinguish between lidars, this field would be the serial number of the lidar
id	integer	Unique identifier for all alerts generated from the system over all time. Two alerts occurring at different times with the same alert code will have different id’s.
level	string	Severity of the alert. Can be WARNING, ERROR, or CRITICAL
msg	string	Short description
msg_verbose	string	Long description

Consumers can receive this data through web requests to /perception/api/v1/alerts or by configuring a publisher with a “diagnostics” source (See JSON Data Streams w/Down-sampling, Batching & Field Mapping).

The following code sample shows alert data in JSON format

"alerts": [
  {
    "active": true,
    "category": "SHOT_LIMITING",
    "cursor": 2833,
    "alert_code": "0x0100000f",
    "level": "WARNING",
    "source_info": "OS1-992144000616",
    "id": 13962805723535194000,
    "msg": "Shot limiting mode is active. Laser power is partially attenuated; please see user guide for heat sinking requirements.",
    "msg_verbose": "Shot limiting has started.",
    "first_occurred": 0,
    "last_occurred": 0,
    "active_count": 0
  },
  {
    "active": true,
    "category": "UDP_TRANSMISSION",
    "cursor": 2822,
    "alert_code": "0x01000016",
    "level": "WARNING",
    "source_info": "OS1-992144000616",
    "id": 11061546793406626000,
    "msg": "Could not send lidar data UDP packet to host; check that network is up.",
    "msg_verbose": "Failed to send lidar UDP data to destination host 10.0.0.39:34642",
    "first_occurred": 0,
    "last_occurred": 0,
    "active_count": 0
  }
]