Skip to main content
Version: latest

Overview

This page describes how to generally use any of the Fluvio clients. Each client has some differences but these are the general rules about them.

For more details on each available client, see the following:

Official API Clients​

Rust SDK
Python SDK
NodeJS SDK

Experimental API Clients​

Java SDK
Java SDK
Java SDK

Connect to Fluvio​

The first thing you want to do to use a Fluvio client is connect to the Fluvio cluster.

Producer​

Once you've got a connection handler, you will want to create a producer for a given topic.

The producer could be created with the following configurations: max_request_size, batch_size, compression, linger and partitioner.

These configurations control the behavior of the producer in the following way:

  • max_request_size: Maximum number of bytes that the producer can send in a single request. If the record is larger than the max request size, the producer drops the record and returns an error. Defaults to 1048576 bytes.
  • batch_size: Maximum number of bytes accumulated by the records before sending the batch. If the record is larger than the batch size, the producer will split the records and send them in multiple batches. Defaults to 16384 bytes.
  • compression: Compression algorithm used by the producer to compress each batch before sending to the SPU. Supported compression algorithms are none, gzip, snappy and lz4.
  • linger: The maximum time to wait to accumulate records before sending the batch. Defaults to 100 ms.
  • partitioner: custom class/struct that assigns the partition to each record that needs to be send. Defaults to Siphash Round Robin partitioner.

Sending​

When sending into a stream, the general send will take a key and a value. The key is optional. For clients which don't have Option as a feature, this is simply an empty array.

Depending on the client, these can be string or an array of bytes.

Depending on the producer configuration, a send call will not send immediately the record to the SPU. flush is used to immediately send all the queued records in the producer batches. Producers should flush before terminating to ensure that all records are sent properly.

Consumer​

Similar to a producing, once you've got a connection, you'll need to create a consumer for a given topic.

Streams​

Once you've got a consumer, you can create a stream given an offset and listen for new items.

Most of our clients support idiomatic ways of iterating over the items in the stream:

This functionality has not been implemented for the java client yet.

Offsets​

An offset is used when a stream is created to request the stream to start at N items from the beginning/end of the stream.

Records​

Each item in a stream is a Record.

Our clients differ a little bit on this but a Record is a wrapper around array of bytes with accessor methods of key or value.

In the python, node and java clients, we have to-string convenience methods.

Timestamps​

Fluvio Records contain timestamp information. As of Fluvio 0.9.25, the timestamp of each record is set by Fluvio Producer on creation, previously the timestamp fields were uninitialized. This information is available to the consumer using the respective API call timestamp().