Module ouster.sdk.osf

Copyright (c) 2021, Ouster, Inc. All rights reserved.

API to work with OSF files

Low-Level API

Reading

Reader for OSF files

class Reader(self: ouster.sdk.osf._osf.Reader, file: str)

Reader is a main entry point to get any info out of OSF file.

chunks(self: ouster.sdk.osf._osf.Reader) → Iterator

Creates an iterator to reads chunks as they appear in a file.

property end_ts

End timestamp (ns) - the highest message timestamp present in the file

property has_message_idx

Whether OSF contains the message counts that are needed for ts_by_message_idx() (message counts was added a bit later to the OSF core, so this function will be obsolete over time)

property has_stream_info

Whether StreamingInfo metadata is available (i.e. reading messages by timestamp and streams can be performed)

messages(*args, **kwargs)

Overloaded function.

  1. messages(self: ouster.sdk.osf._osf.Reader) -> Iterator

    Creates an iterator to reads messages in default STREAMING layout.

  2. messages(self: ouster.sdk.osf._osf.Reader, start_ts: int, end_ts: int) -> Iterator

    Read messages in [start_ts, end_ts] timestamp range (inclusive)

  3. messages(self: ouster.sdk.osf._osf.Reader, stream_ids: List[int] = []) -> Iterator

    Read messages from only specified [<stream_ids>] list

  4. messages(self: ouster.sdk.osf._osf.Reader, stream_ids: List[int], start_ts: int, end_ts: int) -> Iterator

    Read messages in [start_ts, end_ts] timestamp range (inclusive) of a specified <stream_ids> list

property meta_store

Returns the metadata store that gives an access to all metadata entries in the file.

property metadata_id

Data id string

property start_ts

Start timestamp (ns) - the lowest message timestamp present in the file

ts_by_message_idx(self: ouster.sdk.osf._osf.Reader, stream_id: int, message_idx: int) → object

Find the timestamp of the message by its index and stream_id.

Requires the OSF with message_counts inside, i.e. has_message_idx() is True, otherwise return value is always None.

MessageRef wrapper for a message

class MessageRef

Thin message wrapper for underlying StampedMessage object.

Provides the decode() function that resolves the underlying message bytes into the object according to the message type.

Underlying message memory is not copied and reference to mmap’ed file object until the decode() is called.

property buffer

Returns encoded message byte array

decode(self: ouster.sdk.osf._osf.MessageRef) → object

Decodes the underlying object and returns it.

Currently supports only two stream types with a corresponding object types:

  • LidarScan - frame of lidar data (client.LidarScan)

property id

Message id which is a stream_id and point to the metadata entry that describes the stream

of(self: ouster.sdk.osf._osf.MessageRef, msg_stream: object) → bool

Checks whether the message belongs to the msg_stream type

property ts

Message timestamp (ns)

ChunkRef wrapper for a chunk

class ChunkRef

Thin chunk wrapper for underlying Chunk object.

Provides chunk properties accessor and messages iterator.

Underlying chunk memory is not copied and reference to mmap’ed file object.

__len__(self: ouster.sdk.osf._osf.ChunkRef) → int

Number of messages in the chunk

property end_ts

The highest timestamp (ns) of the messages in the chunk

property offset

Offset to the first byte of the chunk

property start_ts

The lowest timestamp (ns) of the messages in the chunk

property valid

True if chunk content is a valid buffer and CRC checksum is correct

MetadataStore for metadata entries

class MetadataStore(self: ouster.sdk.osf._osf.MetadataStore)

Stores metadata entries of the file.

One of information about available sensors, it’s configuration, available streams, chunks layout method, etc.

find(self: ouster.sdk.osf._osf.MetadataStore, meta_type: object) → Dict[int, ouster::osf::MetadataEntry]

Get all metadata entries of the specified meta_type

get(self: ouster.sdk.osf._osf.MetadataStore, meta_type: object) → ouster::osf::MetadataEntry

Get the first metadata entry of the specified meta_type

items(self: ouster.sdk.osf._osf.MetadataStore) → Iterator

Key/Value pairs of metadata entries

MetadataEntry base class for all metadatas

class MetadataEntry(self: ouster.sdk.osf._osf.MetadataEntry)

Single OSF metadata entry.

It’s typed and has corresponding encoding/decoding functions to the underlying bytes representation (buffer()/from_buffer())

property buffer

Encodes (serialize) metadata entry to a stored byte array

static from_buffer(buf: List[int], type_str: str) → ouster.sdk.osf._osf.MetadataEntry

Decodes (deserialize) metadata entry buffer to a typed object

property id

Id of the metadata entry (unique for a file)

of(self: ouster.sdk.osf._osf.MetadataEntry, meta_obj_type: object) → bool

Checks whether metadata entry is of particular type

It’s just: self.type == meta_obj_type.type_id

property static_type

Static type, C++ compile time (in Python use type_id of concrete type objects instead)

property type

Type of the metadata entry (use this)

Writing OSF files

Writer to create OSF file

class Writer(*args, **kwargs)

Simple writer interface for OSF file

All jobs are done with MetadataStore for adding metadata entries and stream interfaces that encodes messages and passes them to internal chunks writer.

Overloaded function.

  1. __init__(self: ouster.sdk.osf._osf.Writer, file_name: str, chunk_size: int = 0) -> None

    Creates a Writer with specified chunk_size.

    Default chunk_size is 2 MB.

  2. __init__(self: ouster.sdk.osf._osf.Writer, filename: str, info: ouster.sdk.client._client.SensorInfo, field_types: Dict[ouster.sdk.client._client.ChanField, object] = {}, chunk_size: int = 0) -> None

    Creates a Writer with deafault STREAMING layout chunks writer.

    Using default chunk_size of 2MB.

    Args:

    filename (str): The filename to output to. info (sensor_info): The sensor info vector to use for a multi stream OSF

    file.

    chunk_size (int): The chunk size in bytes to use for the OSF file. This arg

    is optional, and if not provided the default value of 2MB is used. If the current chunk being written exceeds the chunk_size, a new chunk will be started on the next call to save. This allows an application to tune the number of messages (e.g. lidar scans) per chunk, which affects the granularity of the message index stored in the StreamingInfo in the file metadata. A smaller chunk_size means more messages are indexed and a larger number of index entries. A more granular index allows for more precise seeking at the slight expense of a larger file.

    field_types (Dict[ChanField, FieldDType]): The fields from scans to

    actually save into the OSF. If not provided uses the fields from the first saved lidar scan for each stream. This parameter is optional.

  3. __init__(self: ouster.sdk.osf._osf.Writer, filename: str, info: List[ouster.sdk.client._client.SensorInfo], field_types: Dict[ouster.sdk.client._client.ChanField, object] = {}, chunk_size: int = 0) -> None

    Creates a Writer with specified chunk_size.

    Default chunk_size is 2MB.

    Args:

    filename (str): The filename to output to. info (List[sensor_info]): The sensor info vector to use for a

    multi stream OSF file.

    field_types (Dict[ChanField, FieldDType]): The fields from scans to

    actually save into the OSF. If not provided uses the fields from the first saved lidar scan for each stream. This parameter is optional.

    chunk_size (int): The chunksize to use for the OSF file, this arg

    is optional.

add_metadata(self: ouster.sdk.osf._osf.Writer, m: object) → int

Add metadata entry to a file

add_sensor(self: ouster.sdk.osf._osf.Writer, info: ouster.sdk.client._client.SensorInfo, field_types: Dict[ouster.sdk.client._client.ChanField, object] = {}) → int

Add a sensor to the OSF file.

Parameters:

  • info (sensor_info) – Sensor to add.

  • field_types (Dict[ChanField, FieldDType]) – The fields from scans to actually save into the OSF. If not provided uses the fields from the first saved lidar scan for each stream. This parameter is optional.

Returns (int):

The stream index to use to write scans to this sensor.

close(self: ouster.sdk.osf._osf.Writer) → None

Finish OSF file and flush everything on disk.

filename(self: ouster.sdk.osf._osf.Writer) → str

Return the osf file name.

Returns (str):

The OSF filename.

is_closed(self: ouster.sdk.osf._osf.Writer) → bool

Return the closed status of the writer.

Returns (bool):

The closed status of the writer.

property meta_store

Returns the metadata store that gives an access to all metadata entries in the file.

metadata_id(self: ouster.sdk.osf._osf.Writer) → str

Return the metadata identifier string.

Returns (str):

The OSF metadata identifier string.

save(*args, **kwargs)

Overloaded function.

  1. save(self: ouster.sdk.osf._osf.Writer, stream_index: int, scan: ouster.sdk.client._client.LidarScan) -> None

    Save a lidar scan to the OSF file.

    Args:
    stream_index (int): The index of the corrosponding

    sensor_info to use.

    scan (LidarScan): The scan to save.

  2. save(self: ouster.sdk.osf._osf.Writer, stream_index: int, scan: ouster.sdk.client._client.LidarScan, ts: int) -> None

    Save a lidar scan to the OSF file.

    Args:
    stream_index (int): The index of the corresponding

    sensor_info to use.

    scan (LidarScan): The scan to save. ts (int): The timestamp to index the scan with.

  3. save(self: ouster.sdk.osf._osf.Writer, scan: List[ouster.sdk.client._client.LidarScan]) -> None

    Save a set of lidar scans to the OSF file.

    Args:
    scans (List[LidarScan]): The scans to save. This will correspond

    to the list of sensor_infos.

  4. save(self: ouster.sdk.osf._osf.Writer, stream_index: int, scan: ouster.sdk.client._client.LidarScan) -> None

    Save a lidar scan to the OSF file.

    Args:
    stream_index (int): The index of the corrosponding

    sensor_info to use.

    scan (LidarScan): The scan to save.

  5. save(self: ouster.sdk.osf._osf.Writer, scan: List[ouster.sdk.client._client.LidarScan]) -> None

    Save a set of lidar scans to the OSF file.

    Args:
    scans (List[LidarScan]): The scans to save. This will correspond

    to the list of sensor_infos.

save_message(*args, **kwargs)

Overloaded function.

  1. save_message(self: ouster.sdk.osf._osf.Writer, stream_id: int, ts: int, buffer: numpy.ndarray[numpy.uint8]) -> None

    Low-level save message routine.

    Directly saves the message buffer with id and ts (ns) without any further checks.

  2. save_message(self: ouster.sdk.osf._osf.Writer, stream_id: int, ts: int, buffer: buffer) -> None

    Low-level save message routine.

    Directly saves the message buffer with id and ts (ns) without any further checks.

sensor_info(*args, **kwargs)

Overloaded function.

  1. sensor_info(self: ouster.sdk.osf._osf.Writer) -> List[ouster.sdk.client._client.SensorInfo]

    Return the sensor info list.

    Returns (List[sensor_info]):

    The sensor info list.

  2. sensor_info(self: ouster.sdk.osf._osf.Writer, stream_index: int) -> ouster.sdk.client._client.SensorInfo

    Return the sensor info of the specifed stream_index.

    Args:
    stream_index (in): The index of the sensor to return

    info about.

    Returns (sensor_info):

    The correct sensor info

sensor_info_count(self: ouster.sdk.osf._osf.Writer) → int

Return the number of sensor_info objects.

Returns (int):

The number of sensor_info objects.

set_metadata_id(self: ouster.sdk.osf._osf.Writer, arg0: str) → None

Set the metadata identifier string.

class ChunksLayout(self: ouster.sdk.osf._osf.ChunksLayout, value: int)

Chunks Layout strategy in an OSF file

  • STANDARD layout - simple FIFO messages output to chunks

  • STREAMING layout - used for ordered by timestamp data replay with special chunks layout scheme, refer to RFC-0018 for details

Members:

STANDARD

STREAMING

STANDARD = <ChunksLayout.STANDARD: 0>
STREAMING = <ChunksLayout.STREAMING: 1>
static from_string(arg0: str) → ouster.sdk.osf._osf.ChunksLayout

Create OSF ChunksLayout from string.

property name
property value

Common metadata entries

LidarSensor Ouster sensor metadata (i.e. client.SensorInfo)

class LidarSensor(*args, **kwargs)

Ouster Lidar Sensor metadata with sensor intrinsics (i.e. SensorInfo/Metadata)

type_id static property is a LidarSensor metadata type identifier.

Overloaded function.

  1. __init__(self: ouster.sdk.osf._osf.LidarSensor, arg0: ouster.sdk.client._client.SensorInfo) -> None

Create from SensorInfo object

  1. __init__(self: ouster.sdk.osf._osf.LidarSensor, metadata_json: str) -> None

Create from metadata_json string representation

property info

SensorInfo stored

property metadata

metadata_json string stored

type_id = 'ouster/v1/os_sensor/LidarSensor'

StreamStats statistics per stream

class StreamStats

Statistics of a stream in STREAMING layout OSF files.

property end_ts

Highest timestamp (ns) of the stream messages

property message_avg_size

Average size (bytes) of a message in a stream

property message_count

Number of messages in a stream

property start_ts

Lowest timestamp (ns) of the stream messages

property stream_id

Id of a stream

StreamingInfo stream statisitcs

class StreamingInfo

Metadata entry that appears in STREAMING layout OSF files.

Establishes the chunk -> stream_id map as well as providing the statistics per stream.

type_id static property is a StreamingInfo metadata type identifier.

property chunks_info

Maps chunk to stream_id by chunk offset

property stream_stats

Statistics of messages in per stream

type_id = 'ouster/v1/streaming/StreamingInfo'

Common streams

LidarScanStream stream

class LidarScanStream

Stream of LidarScan objects from a sensor.

type_id static property is a LidarScanStream underlying metadata type identifier.

property meta

metadata entry to store LidarScanStream metadata in an OSF file

type_id = 'ouster/v1/os_sensor/LidarScanStream'

High-Level API

osf.Scans just read LidarScan objects from a file

class Scans(osf_file, *, cycle=False, sensor_id=0)[source]

An iterable stream of LidarScan read from OSF file (for the first available sensor).

Parameters:

  • osf_file (str) – OSF filename as scans source

  • cycle (bool) – repeat infinitely after iteration is finished is True

  • sensor_id (int) – id of the sensor which LidarScan stream data to read

  • type). ((i.e. id of the metadata entry with osf.LidarSensor) –

  • 0 (default) –

__iter__()[source]

Iterator that returns LidarScan objects.

Return type:

Iterator[LidarScan]

withTs()[source]

Iterator that returns tuple of (ts, LidarScan)

Where ts - is a timestamp (ns) of a LidarScan (usually as a timestamp of a first packet in a LidarScan)

Return type:

Iterator[Tuple[int, LidarScan]]

close()[source]

Release the underlying resource, if any.

Return type:

None

property metadata: SensorInfo

Return metadata of a Lidar Sensor used.

property is_live: bool

True if data obtained from the RUNNING sensor or as a stream from the socket

Returns:

True if data obtained from the RUNNING sensor or as a stream from the socket False if data is read from a stored media. Restarting an iter() means that

the data can be read again.

property is_seekable: bool

True for non-live sources, This property can be True regardless of scan source being indexed or not.

property is_indexed: bool

True for IndexedPcap and OSF scan sources, this property tells users whether the underlying source allows for random access of scans, see __getitem__.

property fields: Dict[ChanField, Type[unsignedinteger]]

Field types are present in the LidarScan objects on read from iterator

property scans_num: int | None

Number of scans available, in case of a live sensor or non-indexable scan source this method returns None