vg::io::ProtobufEmitter< T > Class Template Reference

#include <protobuf_emitter.hpp>

Public Types
using	group_listener_t = std::function< void(int64_t, int64_t)>
using	message_listener_t = std::function< void(const T &)>

Public Member Functions
	ProtobufEmitter (std::ostream &out, bool compress=true, size_t max_group_size=1000)
	~ProtobufEmitter ()
	Destructor that finishes the file. More...
	ProtobufEmitter (const ProtobufEmitter &other)=delete
ProtobufEmitter &	operator= (const ProtobufEmitter &other)=delete
	ProtobufEmitter (ProtobufEmitter &&other)=default
ProtobufEmitter &	operator= (ProtobufEmitter &&other)=default
void	write (T &&item)
void	write_many (vector< T > &&ordered_items)
void	write_copy (const T &item)
void	on_group (group_listener_t &&listener)
void	on_message (message_listener_t &&listener)
	Add an event listener that will be called every time a message is emitted. More...
void	emit_group ()
void	flush ()

Private Member Functions
void	handle (bool ok)
	Make sure the given Protobuf-library bool return value is true, and fail otherwise with a message. More...

Private Attributes
mutex	out_mutex
MessageEmitter	message_emitter
	We wrap a MessageEmitter that handles tagged message IO. More...
string	tag
	And a single precomputed copy of the tag string to use. More...
list< group_listener_t >	group_handlers
vector< message_listener_t >	message_handlers
	These we invoke ourselves per message. More...

Detailed Description

template<typename T>
class vg::io::ProtobufEmitter< T >

Class that wraps an output stream and allows emitting groups of Protobuf objects to it, with internal buffering. Handles finishing the file on its own, and allows tracking of BGZF virtual offsets within a non-seekable stream (as long as the entire stream is controlled by one instance). Cannot be copied, but can be moved.

Can call callbacks with the groups emitted and their virtual offsets, for indexing purposes.

Note that the callbacks may be called by the ProtobufEmitter's destructor, so anything they reference needs to outlive the ProtobufEmitter.

Writes compressed VPKG data by default.

May be more efficient than repeated write/write_buffered calls because a single BGZF stream can be used.

Thread-safe to call into. Serialization is done before locking. If a particular order is needed between objects, use the multi-object write functions. Listeners will be called inside the lock, so only one will be in progress at a time.

Member Typedef Documentation

group_listener_t

template<typename T >

using vg::io::ProtobufEmitter< T >::group_listener_t = std::function<void(int64_t, int64_t)>

Define a type for group emission event listeners. The arguments are the start virtual offset and the past-end virtual offset.

message_listener_t

template<typename T >

using vg::io::ProtobufEmitter< T >::message_listener_t = std::function<void(const T&)>

Define a type for message emission event listeners. This gets called for every message we emit, and then the group listeners get called for the whole group.

Constructor & Destructor Documentation

ProtobufEmitter() [1/3]

template<typename T >

vg::io::ProtobufEmitter< T >::ProtobufEmitter	(	std::ostream &	out,
		bool	compress = true,
		size_t	max_group_size = 1000
	)

Constructor. Writes type-tagged Protobuf data to the given output stream. If compress is true, data will be BGZF-compressed. The maximum number of Protobuf messages in a tagged group is controlled by max_group_size.

~ProtobufEmitter()

template<typename T >

vg::io::ProtobufEmitter< T >::~ProtobufEmitter

Destructor that finishes the file.

ProtobufEmitter() [2/3]

template<typename T >

vg::io::ProtobufEmitter< T >::ProtobufEmitter ( const ProtobufEmitter< T > &  other )

delete

ProtobufEmitter() [3/3]

template<typename T >

vg::io::ProtobufEmitter< T >::ProtobufEmitter ( ProtobufEmitter< T > &&  other )

default

Member Function Documentation

emit_group()

template<typename T >

auto vg::io::ProtobufEmitter< T >::emit_group

Actually write out everything in the buffer. Doesn't actually flush the underlying streams to disk. Assumes that no more than one group's worth of items are in the buffer.

flush()

template<typename T >

auto vg::io::ProtobufEmitter< T >::flush

Write out anything in the buffer, and flush the backing BGZF and the backing stream. After this function is called, a complete BGZF block has been output (unless another thead has written something).

handle()

template<typename T >

auto vg::io::ProtobufEmitter< T >::handle ( bool  ok )

private

Make sure the given Protobuf-library bool return value is true, and fail otherwise with a message.

on_group()

template<typename T >

auto vg::io::ProtobufEmitter< T >::on_group

(

group_listener_t &&

listener

)

Add an event listener that listens for emitted groups. The listener will be called with the start virtual offset, and the past-end virtual offset. Moves the function passed in. Anything the function uses by reference must outlive this object!

on_message()

template<typename T >

auto vg::io::ProtobufEmitter< T >::on_message

(

message_listener_t &&

listener

)

Add an event listener that will be called every time a message is emitted.

operator=() [1/2]

template<typename T >

ProtobufEmitter& vg::io::ProtobufEmitter< T >::operator= ( const ProtobufEmitter< T > &  other )

delete

operator=() [2/2]

template<typename T >

ProtobufEmitter& vg::io::ProtobufEmitter< T >::operator= ( ProtobufEmitter< T > &&  other )

default

write()

template<typename T >

auto vg::io::ProtobufEmitter< T >::write

(

T &&

item

)

Emit the given item. TODO: May not really be any more efficient. We serialize to string right away in either case.

write_copy()

template<typename T >

auto vg::io::ProtobufEmitter< T >::write_copy

(

const T &

item

)

Emit a copy of the given item. To use when you have something you can't move.

write_many()

template<typename T >

auto vg::io::ProtobufEmitter< T >::write_many

(

vector< T > &&

ordered_items

)

Emit the given collection of items in order, with no other intervening items between them.

Member Data Documentation

group_handlers

template<typename T >

list<group_listener_t> vg::io::ProtobufEmitter< T >::group_handlers

private

And all the group handler functions. These need to never move; they are captured by reference to listeners in our MessageEmitter.

message_emitter

template<typename T >

MessageEmitter vg::io::ProtobufEmitter< T >::message_emitter

private

We wrap a MessageEmitter that handles tagged message IO.

message_handlers

template<typename T >

vector<message_listener_t> vg::io::ProtobufEmitter< T >::message_handlers

private

These we invoke ourselves per message.

out_mutex

template<typename T >

mutex vg::io::ProtobufEmitter< T >::out_mutex

private

Mutex to controll access to the backing MessageEmitter. Also needs to control access to the listener lists.

tag

template<typename T >

string vg::io::ProtobufEmitter< T >::tag

private

And a single precomputed copy of the tag string to use.

---

The documentation for this class was generated from the following file:

• deps/libvgio/include/vg/io/protobuf_emitter.hpp