reader.h

ChunkValidity

enum class ouster::osf::ChunkValidity

Enumerator for dealing with chunk validity.

This is synthesized and thus does not have a reference in the Flat Buffer. Value is set in Reader::verify_chunk

Values:

enumerator UNKNOWN

Validity can not be ascertained.

enumerator VALID

Chunk is valid.

enumerator INVALID

Chunk is invalid.

ChunkState

struct ChunkState

The structure for representing chunk information and for forward iteration.

This struct is partially mapped to the Flat Buffer data. Flat Buffer Reference: fb/metadata.fbs :: ChunkOffset

Public Members

uint64_t offset

The current chunk’s offset from the begining of the chunks section.

Flat Buffer Reference: fb/metadata.fbs :: ChunkOffset :: offset

uint64_t next_offset

The next chunk’s offset for forward iteration. Should work like a linked list.

This is partially synthesized from the Flat Buffers. This will link up with the next chunks offset. Value is set in ChunksPile::link_stream_chunks Flat Buffer Reference: fb/metadata.fbs :: ChunkOffset :: offset

ts_t start_ts

The first timestamp in the chunk in ordinality.

Flat Buffer Reference: fb/metadata.fbs :: ChunkOffset :: start_ts

ts_t end_ts

The last timestamp in the chunk in ordinality.

Flat Buffer Reference: fb/metadata.fbs :: ChunkOffset :: end_ts

ChunkValidity status

The validity of the current chunk

This is synthesized and thus does not have a reference in the Flat Buffers. Value is set in Reader::verify_chunk

std::string ouster::osf::to_string(const ChunkState &chunk_state)

To String Functionality For ChunkState

Parameters:

chunk_state – [in] The data to get the string representation for

Returns:

The string representation

ChunkInfoNode

struct ChunkInfoNode

The structure for representing streaming information.

This struct is partially mapped to the Flat Buffer data.

Public Members

uint64_t offset

The chunk offset from the begining of the chunks section.

Flat Buffer Reference: fb/metadata.fbs :: ChunkOffset :: offset

uint64_t next_offset

The next chunk’s offset for forward iteration. Should work like a linked list.

This is partially synthesized from the Flat Buffers. This will link up with the next chunks offset. Value is set in ChunksPile::link_stream_chunks Flat Buffer Reference: fb/metadata.fbs :: ChunkOffset :: offset

uint32_t stream_id

The stream this is associated with.

Flat Buffer Reference: fb/streaming/streaming_info.fbs :: ChunkInfo :: stream_id

uint32_t message_count

Total number of messages in a stream_id in the whole OSF file

Flat Buffer Reference: fb/streaming/streaming_info.fbs :: ChunkInfo :: message_count

uint32_t message_start_idx

The index of the start of the message.

Todo:

try to describe this better

This is partially synthesized from the Flat Buffers. Value is set in ChunksPile::link_stream_chunks Synthesized from Flat Buffer Reference: fb/metadata.fbs :: ChunkOffset :: message_count

std::string ouster::osf::to_string(const ChunkInfoNode &chunk_info)

To String Functionality For ChunkInfoNode

Parameters:

chunk_info – [in] The data to get the string representation format

Returns:

The string representation

ChunksPile

class ChunksPile

Chunks state map. Validity info and next offset.

Public Types

using StreamChunksMap = std::unordered_map<uint32_t, std::shared_ptr<std::vector<uint64_t>>>

stream_id to offset map.

Public Functions

ChunksPile()

Default blank constructor.

void add(uint64_t offset, ts_t start_ts, ts_t end_ts)

Add a new chunk to the ChunkPile.

Parameters:

• offset – [in] The offset for the chunk.

• start_ts – [in] The first timestamp in the chunk.

• end_ts – [in] The first timestamp in the chunk.

ChunkState *get(uint64_t offset)

Return the chunk associated with an offset.

Parameters:

offset – [in] The offset to return the chunk for.

Returns:

The chunk if found, or nullptr.

void add_info(uint64_t offset, uint32_t stream_id, uint32_t message_count)

Add a new streaming info to the ChunkPile.

Parameters:

• offset – [in] The offset for the chunk.

• stream_id – [in] The stream_id associated.

• message_count – [in] The number of messages.

ChunkInfoNode *get_info(uint64_t offset)

Return the streaming info associated with an offset.

Parameters:

offset – [in] The offset to return the streaming info for.

Returns:

The streaming info if found, or nullptr.

ChunkInfoNode *get_info_by_message_idx(uint32_t stream_id, uint32_t message_idx)

Return the streaming info associated with a message_idx.

Parameters:

• stream_id – [in] The stream to look for infos in.

• message_idx – [in] The specific message index to look for.

Returns:

The streaming info if found, or nullptr.

ChunkState *get_by_lower_bound_ts(uint32_t stream_id, const ts_t ts)

Return the chunk associated with a lower bound timestamp.

Parameters:

• stream_id – [in] The stream to look for chunks in.

• ts – [in] The lower bound for the chunk.

Returns:

The chunk if found, or nullptr.

ChunkState *next(uint64_t offset)

Return the next chunk identified by the offset.

Parameters:

offset – [in] The offset to return the next chunk for.

Returns:

The chunk if found, or nullptr.

ChunkState *next_by_stream(uint64_t offset)

Return the next chunk identified by the offset per stream.

Parameters:

offset – [in] The offset to return the next chunk for.

Returns:

The chunk if found, or nullptr.

ChunkState *first()

Return the first chunk.

Returns:

The chunk if found, or nullptr.

size_t size() const

Return the size of the chunk pile.

Returns:

The size of the chunk pile.

bool has_message_idx() const

Return if there is a message index.

Returns:

If there is a message index.

StreamChunksMap &stream_chunks()

Return the stream_id to chunk offset map.

Returns:

The stream_id to chunk offset map.

void link_stream_chunks()

Builds internal links between ChunkInfoNode per stream.

Throws:

• std::logic_error – exception on non increasing timestamps.

• std::logic_error – exception on non existent info.

ChunksIter

struct ChunksIter

Chunk forward iterator in order of offset.

Public Functions

ChunksIter()

Default construction that zeros out member variables.

ChunksIter(const ChunksIter &other)

Initialize from another ChunksIter object.

Parameters:

other – [in] The other ChunksIter object to initalize from.

ChunksIter &operator=(const ChunksIter &other) = default

Default assign operator.

Parameters:

other – [in] The other ChunksIter to assign to this.

const ChunkRef operator*() const

Return a ChunkRef object associated with this ChunksIter object.

Throws:

std::logic_error – Exception on end of iteration.

Returns:

The ChunkRef object associated with this ChunksIter object.

const std::unique_ptr<ChunkRef> operator->() const

Return a ChunkRef pointer associated with this ChunksIter object.

Returns:

The ChunkRef pointer associated with this ChunksIter object.

ChunksIter &operator++()

Increment the ChunksIter iterator and return *this.

Returns:

The current ChunksIter object.

bool operator==(const ChunksIter &other) const

Equality operator to compare two ChunksIter objects.

Parameters:

other – [in] The other object to compare.

Returns:

Whether the two ChunksIter objects are the same.

std::string to_string() const

To String Functionality For ChunksIter.

Returns:

The string representation

Related

bool operator!=(const ChunksIter &other) const

Equality operator to compare two ChunksIter objects.

==(const ChunksIter& other)

Parameters:

other – [in] The other object to compare.

Returns:

Whether the two ChunksIter objects are not the same.

ChunksRange

class ChunksRange

std iterator class for iterating through chunks.

Public Functions

ChunksIter begin() const

Begin function for std iterator support.

Returns:

A ChunksIter object for iteration.

ChunksIter end() const

End function for std iterator support.

Returns:

A ChunksIter object for signifying the end of iteration.

std::string to_string() const

To String Functionality For ChunksRange.

Returns:

The string representation.

Reader

class Reader

OSF Reader that simply reads sequentially messages from the OSF file.

Todo:

Add filtered reads, and other nice things…

Public Functions

Reader(OsfFile &osf_file)

Creates reader from OSF file resource.

Parameters:

osf_file – [in] The OsfFile object to use to read from.

Reader(const std::string &file)

Creates reader from OSF file name.

Parameters:

file – [in] The OSF file path to read from.

MessagesStreamingRange messages()

Reads the messages from the first OSF chunk in sequental order till the end. Doesn’t support RandomAccess.

Throws:

std::logic_error – Exception on not having sensor_info.

Returns:

The MessageStreamingRange object to iterate through the messages.

MessagesStreamingRange messages(const ts_t start_ts, const ts_t end_ts)

Reads the messages from the first OSF chunk in sequental order till the end. Doesn’t support RandomAccess.

Throws:

std::logic_error – Exception on not having sensor_info.

Parameters:

• start_ts – [in] Specify the start of the timestamps that should be iterated through.

• end_ts – [in] Specify the end of the timestamps that should be iterated through.

Returns:

The MessageStreamingRange object to iterate through the messages.

MessagesStreamingRange messages(const std::vector<uint32_t> &stream_ids)

Reads the messages from the first OSF chunk in sequental order till the end. Doesn’t support RandomAccess.

Throws:

std::logic_error – Exception on not having sensor_info.

Parameters:

stream_ids – [in] Filter the message iteration to specific streams.

Returns:

The MessageStreamingRange object to iterate through the messages.

MessagesStreamingRange messages(const std::vector<uint32_t> &stream_ids, const ts_t start_ts, const ts_t end_ts)

Reads the messages from the first OSF chunk in sequental order till the end. Doesn’t support RandomAccess.

Throws:

std::logic_error – Exception on not having sensor_info.

Parameters:

• start_ts – [in] Specify the start of the timestamps that should be iterated through.

• end_ts – [in] Specify the end of the timestamps that should be iterated through.

• stream_ids – [in] Filter the message iteration to specific streams.

Returns:

The MessageStreamingRange object to iterate through the messages.

nonstd::optional<ts_t> ts_by_message_idx(uint32_t stream_id, uint32_t message_idx)

Find the timestamp of the message by its index and stream_id.

Requires the OSF with message_counts inside, i.e. has_message_idx() return True, otherwise return value is always empty (nullopt).

Throws:

std::logic_error – Exception on not having sensor_info.

Parameters:

• stream_id – [in] stream id on which the message_idx search is performed

• message_idx – [in] the message index (i.e. rank/number) to search for

Returns:

message timestamp that corresponds to the message_idx in the stream_id

bool has_message_idx() const

Whether OSF contains the message counts that are needed for ts_by_message_idx()

Message counts was added a bit later to the OSF core (ChunkInfo struct), so this function will be obsolete over time.

Returns:

Whether OSF contains the message counts that are needed for ts_by_message_idx()

ChunksRange chunks()

Reads chunks and returns the iterator to valid chunks only. NOTE: Every chunk is read in full and validated. (i.e. it’s not just iterator over chunks index)

Returns:

The iterator to valid chunks only.

std::string metadata_id() const

Return the metadata id.

Returns:

The metadata id.

ts_t start_ts() const

Return the lowest timestamp in the ChunksIter.

Returns:

The lowest timestamp in the ChunksIter.

ts_t end_ts() const

Return the highest timestamp in the ChunksIter.

Returns:

The highest timestamp in the ChunksIter.

const MetadataStore &meta_store() const

Return all metadata entries as a MetadataStore

Returns:

All of the metadata entries as a MetadataStore.

bool has_stream_info() const

If the chunks can be read by stream and in non-decreasing timestamp order.

Returns:

The chunks can be read by stream and timestamps are sane.

MessageRef

class MessageRef

Thin interface class that holds the pointer to the message and reconstructs underlying data to the corresponding object type given the Stream type.

Public Functions

MessageRef(const uint8_t *buf, const MetadataStore &meta_provider)

The only way to create the MessageRef is to point to the corresponding byte buffer of the message in OSF file.

Parameters:

• buf – [in] The buffer to use to make a MessageRef object.

• meta_provider – [in] The metadata store that is used in types reconstruction

MessageRef(const uint8_t *buf, const MetadataStore &meta_provider, std::shared_ptr<std::vector<uint8_t>> chunk_buf)

The only way to create the MessageRef is to point to the corresponding byte buffer of the message in OSF file.

Parameters:

• buf – [in] The buffer to use to make a MessageRef object.

• meta_provider – [in] The metadata store that is used in types reconstruction

• chunk_buf – [inout] The pre-existing chunk buffer to use.

uint32_t id() const

Get the message stream id.

Returns:

The message stream id.

ts_t ts() const

Get the timestamp of the message.

Returns:

The timestamp of the message.

const uint8_t *buf() const

Todo:

[pb] Type of the stored data (meta of the stream?)

Get the pointer to the underlying data.

Returns:

The pointer to the underlying data.

std::string to_string() const

Debug string representation.

Returns:

The string representation of a MessageRef.

template<typename Stream>
inline bool is() const

Checks whether the message belongs to the specified Stream type.

Template Parameters:

Stream – The cpp data type to check against.

Returns:

If the current MessageRef is of type [Stream].

bool is(const std::string &type_str) const

Checks whether the message belongs to the specified Strean type.

Parameters:

type_str – [in] The data type in string form to check against.

Returns:

If the current MessageRef is of type type_str.

template<typename Stream>
inline std::unique_ptr<typename Stream::obj_type> decode_msg() const

Reconstructs the underlying data to the class (copies data).

Template Parameters:

Stream – The type of the target data.

Returns:

A smart pointer to the new object.

std::vector<uint8_t> buffer() const

Get the underlying raw message byte vector.

Returns:

Return the underlying raw message byte vector.

bool operator==(const MessageRef &other) const

Check if two MessageRefs are equal.

Parameters:

other – [in] The other MessageRef to check against.

Returns:

If the two MessageRefs are equal.

bool operator!=(const MessageRef &other) const

Check if two MessageRefs are not equal.

Parameters:

other – [in] The other MessageRef to check against.

Returns:

If the two MessageRefs are not equal.

ChunkRef

class ChunkRef

Thin interface class that holds the pointer to the chunk and hides the messages reading routines. It expects that Chunk was “verified” before creating a ChunkRef.

Public Functions

ChunkRef()

Default ChunkRef constructor that just zeros the internal fields.

ChunkRef(const uint64_t offset, Reader *reader)

Parameters:

• offset – [in] The offset into the chunk array for the specified chunk.

• reader – [in] The reader object to use for reading.

bool operator==(const ChunkRef &other) const

Check if two ChunkRefs are equal.

Parameters:

other – [in] The other ChunkRef to check against.

Returns:

If the two ChunkRef are equal.

bool operator!=(const ChunkRef &other) const

Check if two ChunkRefs are not equal.

Parameters:

other – [in] The other ChunkRef to check against.

Returns:

If the two ChunkRef are not equal.

const ChunkState *state() const

Get the ChunkState for the chunk associated with this ChunkRef.

Returns:

The ChunkState associated with this ChunkRef.

const ChunkInfoNode *info() const

Get the ChunkInfoNode for the chunk associated with this ChunkRef.

Returns:

The ChunkInfoNode associated with this ChunkRef.

MessagesChunkIter begin() const

Begin function for std iterator support.

Returns:

A MessagesChunkIter object for iteration.

MessagesChunkIter end() const

End function for std iterator support.

Returns:

A MessagesChunkIter object for signifying the end of iteration.

const MessageRef operator[](size_t msg_idx) const

Get the message at a specific index.

Todo:

Simplify this and any other instance of this

Parameters:

msg_idx – [in] The message index to get.

Returns:

The resulting message.

std::unique_ptr<const MessageRef> messages(size_t msg_idx) const

Get the message smart pointer at a specific index.

Todo:

Simplify this and any other instance of this

Parameters:

msg_idx – [in] The message index to get.

Returns:

The resulting message smart pointer, returns nullptr if non existent.

std::string to_string() const

Debug string representation.

Returns:

The string representation of a ChunkRef.

uint64_t offset() const

Return the chunk offset in the larger flatbuffer array.

Returns:

The chunk offset in the larger flatbuffer array.

size_t size() const

Returns the summation of the sizes of the chunks messages

Returns:

The summation of the sizes of the chunks messages, 0 on chunk invalidity.

bool valid() const

Get the validity of the chunk.

Returns:

The validity of the chunk.

Related

ChunkState *state()

Get the ChunkState for the chunk associated with this ChunkRef.

Returns:

The ChunkState associated with this ChunkRef.

ChunkInfoNode *info()

Get the ChunkInfoNode for the chunk associated with this ChunkRef.

Returns:

The ChunkInfoNode associated with this ChunkRef.

ts_t start_ts() const

The lowest timestamp in the chunk. A shortcut for state()->start_ts

ts_t end_ts() const

The highest timestamp in the chunk. A shortcut for state()->end_ts

MessagesChunkIter

struct MessagesChunkIter

Convenient iterator class to go over all of the messages in a chunk.

Public Functions

MessagesChunkIter()

Default MessagesChunkIter constructor that just zeros the internal fields.

MessagesChunkIter(const MessagesChunkIter &other)

Initialize the MessagesChunkIter from another MessageChunkIter object.

Parameters:

other – [in] The other MessagesChunkIter to initalize from.

MessagesChunkIter &operator=(const MessagesChunkIter &other) = default

Default assignment operation.

Parameters:

other – [in] The other MessageChunkIter to assign to.

const MessageRef operator*() const

Gets the current ChunkRef via value.

Returns:

The current ChunkRef value.

std::unique_ptr<const MessageRef> operator->() const

Gets the current ChunkRef via smart pointer.

Returns:

The current ChunkRef smart pointer.

MessagesChunkIter &operator++()

Advance to the next message in the chunk.

Returns:

*this

MessagesChunkIter operator++(int)

Advance to the next message in the chunk.

Returns:

*this

MessagesChunkIter &operator--()

Regress to the previous message in the chunk.

Returns:

*this

MessagesChunkIter operator--(int)

Regress to the previous message in the chunk.

Returns:

*this

bool operator==(const MessagesChunkIter &other) const

Check if two MessagesChunkIter are equal.

Parameters:

other – [in] The other MessagesChunkIter to check against.

Returns:

If the two MessagesChunkIter are equal.

bool operator!=(const MessagesChunkIter &other) const

Check if two MessagesChunkIter are not equal.

Parameters:

other – [in] The other MessagesChunkIter to check against.

Returns:

If the two MessagesChunkIter are not equal.

std::string to_string() const

Debug string representation.

Returns:

The string representation of a MessagesChunkIter.

MessagesStreamingRange

class MessagesStreamingRange

Public Functions

MessagesStreamingIter begin() const

Begin function for std iterator support.

Returns:

A MessagesStreamingIter object for iteration.

MessagesStreamingIter end() const

End function for std iterator support.

Returns:

A MessagesStreamingIter object for signifying the end of iteration.

std::string to_string() const

Debug string representation.

Returns:

The string representation of a MessagesStreamingRange.

MessagesStreamingIter

struct MessagesStreamingIter

Iterator over all messages in Streaming Layout order for specified timestamp range.

Public Functions

MessagesStreamingIter()

Default MessagesStreamingIter constructor that just zeros the internal fields.

MessagesStreamingIter(const MessagesStreamingIter &other)

Initialize the MessagesStreamingIter from another MessagesStreamingIter object.

Parameters:

other – [in] The other MessagesStreamingIter to initalize from.

MessagesStreamingIter &operator=(const MessagesStreamingIter &other) = default

Default assignment operation.

Parameters:

other – [in] The other MessagesStreamingIter to assign to.

const MessageRef operator*() const

Gets the current MessageRef via value.

Returns:

The current MessageRef value.

std::unique_ptr<const MessageRef> operator->() const

Gets the current MessageRef via smart pointer.

Returns:

The current MessageRef smart pointer.

MessagesStreamingIter &operator++()

Advance to the next message.

Returns:

*this

MessagesStreamingIter operator++(int)

Advance to the next message.

Returns:

*this

bool operator==(const MessagesStreamingIter &other) const

Check if two MessagesStreamingIter are equal.

Parameters:

other – [in] The other MessagesStreamingIter to check against.

Returns:

If the two MessagesStreamingIter are equal.

bool operator!=(const MessagesStreamingIter &other) const

Check if two MessagesStreamingIter are not equal.

Parameters:

other – [in] The other MessagesStreamingIter to check against.

Returns:

If the two MessagesStreamingIter are not equal.

std::string to_string() const

Debug string representation.

Returns:

The string representation of a MessagesStreamingIter.

struct greater_chunk_type

Comparison struct used for determining which chunk is greater.

Public Functions

bool operator()(const opened_chunk_type &a, const opened_chunk_type &b)

Comparison operator used for determining if the first is greater than the second. The comparison is based on the timestamps.

Parameters:

• a – [in] The first chunk to compare.

• b – [in] The second chunk to compare.

Returns:

If the first chunk is greater than the second chunk.