Aedis 1.0.0  
High level Redis client
Documentation

Useful links: Reference, Changelog and Benchmarks.

Overview

Aedis is a high-level Redis client library built on top of Asio. Some of its distinctive features are

  • Support for the latest version of the Redis communication protocol RESP3.
  • First class support for STL containers and C++ built-in types.
  • Serialization and deserialization of your own data types.
  • Healthy checks, back pressure and low latency.
  • Hides most of the low level asynchronous operations away from the user.

Let us have a look a some code snippets

Async

The code below sends a ping command to Redis and quits (see intro.cpp)

int main()
{
net::io_context ioc;
connection db{ioc};
request req;
req.push("PING");
req.push("QUIT");
std::tuple<std::string, aedis::ignore> resp;
db.async_run(req, adapt(resp), net::detached);
ioc.run();
std::cout << std::get<0>(resp) << std::endl;
}
A high level connection to Redis.
Definition: connection.hpp:45
auto async_run(CompletionToken token=CompletionToken{})
Starts communication with the Redis server asynchronously.
Definition: connection.hpp:271
auto adapt() noexcept
Creates an adapter that ignores responses.
Definition: adapt.hpp:140

The connection class maintains a healthy connection with Redis over which users can execute their commands, without any need of queuing. For example, to execute more than one request

int main()
{
...
net::io_context ioc;
connection db{ioc};
db.async_exec(req1, adapt(resp1), handler1);
db.async_exec(req2, adapt(resp2), handler2);
db.async_exec(req3, adapt(resp3), handler3);
db.async_run(net::detached);
ioc.run();
...
}
auto async_exec(resp3::request const &req, Adapter adapter=adapt(), CompletionToken token=CompletionToken{})
Executes a command on the redis server asynchronously.
Definition: connection.hpp:335

The connection::async_exec functions above can be called from different places in the code without knowing about each other, see for example echo_server.cpp. Server-side pushes are supported on the same connection where commands are executed, a typical subscriber will look like (see subscriber.cpp)

net::awaitable<void> reader(std::shared_ptr<connection> db)
{
for (std::vector<node_type> resp;;) {
co_await db->async_receive_event(adapt(resp));
// Use resp and clear it.
resp.clear();
}
}

Sync

The connection class offers only an asynchronous API. Synchronous communications with redis is provided by the aedis::sync wrapper class. (see intro_sync.cpp)

int main()
{
net::io_context ioc{1};
auto work = net::make_work_guard(ioc);
std::thread t1{[&]() { ioc.run(); }};
sync<connection> conn{work.get_executor()};
std::thread t2{[&]() { boost::system::error_code ec; conn.run(ec); }};
request req;
req.push("PING");
req.push("QUIT");
std::tuple<std::string, aedis::ignore> resp;
conn.exec(req, adapt(resp));
std::cout << "Response: " << std::get<0>(resp) << std::endl;
work.reset();
t1.join();
t2.join();
}

Installation

To install and use Aedis you will need

  • Boost 1.78 or greater.
  • C++17. Some examples require C++20 with coroutine support.
  • Redis 6 or higher. Optionally also redis-cli and Redis Sentinel.

For a simple installation run

$ git clone --branch v1.0.0 https://github.com/mzimbres/aedis.git
$ cd aedis
# Option 1: Direct compilation.
$ g++ -std=c++17 -pthread examples/intro.cpp -I./include -I/path/boost_1_79_0/include/
# Option 2: Use cmake.
$ BOOST_ROOT=/opt/boost_1_79_0/ cmake -DCMAKE_CXX_FLAGS=-std=c++20 .
Note
CMake support is still experimental.

For a proper full installation on the system run

# Download and unpack the latest release
$ wget https://github.com/mzimbres/aedis/releases/download/v1.0.0/aedis-1.0.0.tar.gz
$ tar -xzvf aedis-1.0.0.tar.gz
# Configure, build and install
$ CXXFLAGS="-std=c++17" ./configure --prefix=/opt/aedis-1.0.0 --with-boost=/opt/boost_1_78_0
$ sudo make install

To build examples and tests

$ make

Using Aedis

When writing you own applications include the following header

#include <aedis/src.hpp>

in no more than one source file in your applications.

Supported compilers

Aedis has been tested with the following compilers

  • Tested with gcc: 12, 11.
  • Tested with clang: 14, 13, 11.

Developers

To generate the build system clone the repository and run

# git clone https://github.com/mzimbres/aedis.git
$ autoreconf -i

After that we get a configure script that can be run as explained above, for example, to build with a compiler other that the system compiler with coverage support run

$ CXX=clang++-14 \
CXXFLAGS="-g -std=c++20 -Wall -Wextra --coverage -fkeep-inline-functions -fkeep-static-functions" \
LDFLAGS="--coverage" \
./configure --with-boost=/opt/boost_1_79_0

To generate release tarballs run

$ make distcheck

Requests

Redis requests are composed of one of more Redis commands (in Redis documentation they are called pipelines). For example

request req;
// Command with variable length of arguments.
req.push("SET", "key", "some value", value, "EX", "2");
// Pushes a list.
std::list<std::string> list
{"channel1", "channel2", "channel3"};
req.push_range("SUBSCRIBE", list);
// Same as above but as an iterator range.
req.push_range2("SUBSCRIBE", std::cbegin(list), std::cend(list));
// Pushes a map.
std::map<std::string, mystruct> map
{ {"key1", "value1"}
, {"key2", "value2"}
, {"key3", "value3"}};
req.push_range("HSET", "key", map);

Sending a request to Redis is then peformed with the following function

co_await db->async_exec(req, adapt(resp));

Serialization

The push and push_range functions above work with integers e.g. int and std::string out of the box. To send your own data type defined a to_bulk function like this

struct mystruct {
// Example struct.
};
void to_bulk(std::string& to, mystruct const& obj)
{
std::string dummy = "Dummy serializaiton string.";
aedis::resp3::to_bulk(to, dummy);
}
void to_bulk(Request &to, boost::string_view data)
Adds a bulk to the request.
Definition: request.hpp:49

Once to_bulk is defined and accessible over ADL mystruct can be passed to the request

request req;
std::map<std::string, mystruct> map {...};
req.push_range("HSET", "key", map);

Example serialization.cpp shows how store json string in Redis.

Responses

To read responses effectively, users must know their RESP3 type, this can be found in the Redis documentation for each command (https://redis.io/commands). For example

Command RESP3 type Documentation
lpush Number https://redis.io/commands/lpush
lrange Array https://redis.io/commands/lrange
set Simple-string, null or blob-string https://redis.io/commands/set
get Blob-string https://redis.io/commands/get
smembers Set https://redis.io/commands/smembers
hgetall Map https://redis.io/commands/hgetall

Once the RESP3 type of a given response is known we can choose a proper C++ data structure to receive it in. Fortunately, this is a simple task for most types. The table below summarises the options

RESP3 type Possible C++ type Type
Simple-string std::string Simple
Simple-error std::string Simple
Blob-string std::string, std::vector Simple
Blob-error std::string, std::vector Simple
Number long long, int, std::size_t, std::string Simple
Double double, std::string Simple
Null boost::optional<T> Simple
Array std::vector, std::list, std::array, std::deque Aggregate
Map std::vector, std::map, std::unordered_map Aggregate
Set std::vector, std::set, std::unordered_set Aggregate
Push std::vector, std::map, std::unordered_map Aggregate

For example

request req;
req.push("HELLO", 3);
req.push_range("RPUSH", "key1", vec);
req.push_range("HSET", "key2", map);
req.push("LRANGE", "key3", 0, -1);
req.push("HGETALL", "key4");
req.push("QUIT");
std::tuple<
aedis::ignore, // hello
int, // rpush
int, // hset
std::vector<T>, // lrange
std::map<U, V>, // hgetall
std::string // quit
> resp;
co_await db->async_exec(req, adapt(resp));
adapter::detail::ignore ignore
Tag used to ignore responses.
Definition: adapt.hpp:35

The tag aedis::ignore can be used to ignore individual elements in the responses. If the intention is to ignore the response to all commands in the request use adapt()

co_await db->async_exec(req, adapt());

Responses that contain nested aggregates or heterogeneous data types will be given special treatment later in The general case. As of this writing, not all RESP3 types are used by the Redis server, which means in practice users will be concerned with a reduced subset of the RESP3 specification.

Optional

It is not uncommon for apps to access keys that do not exist or that have already expired in the Redis server, to deal with these cases Aedis provides support for boost::optional. To use it, wrap your type around boost::optional like this

boost::optional<std::unordered_map<T, U>> resp;
co_await db->async_exec(req, adapt(resp));

Everything else stays the same.

Transactions

To read the response to transactions we have to observe that Redis queues the commands as they arrive and sends the responses back to the user as an array, in the response to the exec command. For example, to read the response to the this request

db.send("MULTI");
db.send("GET", "key1");
db.send("LRANGE", "key2", 0, -1);
db.send("HGETALL", "key3");
db.send("EXEC");

use the following response type

using boost::optional;
using exec_resp_type =
std::tuple<
optional<std::string>, // get
optional<std::vector<std::string>>, // lrange
optional<std::map<std::string, std::string>> // hgetall
>;
std::tuple<
ignore, // multi
ignore, // get
ignore, // lrange
ignore, // hgetall
exec_resp_type, // exec
> resp;
co_await db->async_exec(req, adapt(resp));

Note that above we are not ignoring the response to the commands themselves but whether they have been successfully queued. For a complete example see containers.cpp.

Deserialization

As mentioned in Serialization, it is common to serialize data before sending it to Redis e.g. to json strings. For performance and convenience reasons, we may also want to deserialize it directly in its final data structure. Aedis supports this use case by calling a user provided from_bulk function while parsing the response. For example

void from_bulk(mystruct& obj, char const* p, std::size_t size, boost::system::error_code& ec)
{
// Deserializes p into obj.
}

After that, you can start receiving data efficiently in the desired types e.g. mystruct, std::map<std::string, mystruct> etc.

The general case

There are cases where responses to Redis commands won't fit in the model presented above, some examples are

  • Commands (like set) whose responses don't have a fixed RESP3 type. Expecting an int and receiving a blob-string will result in error.
  • RESP3 aggregates that contain nested aggregates can't be read in STL containers.
  • Transactions with a dynamic number of commands can't be read in a std::tuple.

To deal with these cases Aedis provides the resp3::node type, that is the most general form of an element in a response, be it a simple RESP3 type or an aggregate. It is defined like this

template <class String>
struct node {
// The RESP3 type of the data in this node.
type data_type;
// The number of elements of an aggregate (or 1 for simple data).
std::size_t aggregate_size;
// The depth of this node in the response tree.
std::size_t depth;
// The actual data. For aggregate types this is always empty.
String value;
};
type
RESP3 data types.
Definition: type.hpp:23

Any response to a Redis command can be received in a std::vector<node<std::string>>. The vector can be seen as a pre-order view of the response tree (https://en.wikipedia.org/wiki/Tree_traversal#Pre-order,_NLR). Using it is no different than using other types

// Receives any RESP3 simple data type.
node<std::string> resp;
co_await db->async_exec(req, adapt(resp));
// Receives any RESP3 simple or aggregate data type.
std::vector<node<std::string>> resp;
co_await db->async_exec(req, adapt(resp));

For example, suppose we want to retrieve a hash data structure from Redis with HGETALL, some of the options are

  • std::vector<node<std::string>: Works always.
  • std::vector<std::string>: Efficient and flat, all elements as string.
  • std::map<std::string, std::string>: Efficient if you need the data as a std::map
  • std::map<U, V>: Efficient if you are storing serialized data. Avoids temporaries and requires from_bulk for U and V.

In addition to the above users can also use unordered versions of the containers. The same reasoning also applies to sets e.g. SMEMBERS.

Examples

The examples listed below cover most use cases presented in the documentation above.

Why Aedis

At the time of this writing there are seventeen Redis clients listed in the official list. With so many clients available it is not unlikely that users are asking themselves why yet another one. In this section I will try to compare Aedis with the most popular clients and why we need Aedis. Notice however that this is ongoing work as comparing client objectively is difficult and time consuming.

The most popular client at the moment of this writing ranked by github stars is

Before we start it is worth mentioning some of the things it does not support

  • RESP3. Without RESP3 is impossible to support some important Redis features like client side caching, among other things.
  • Coroutines.
  • Reading responses directly in user data structures avoiding temporaries.
  • Error handling with error-code and exception overloads.
  • Healthy checks.

The remaining points will be addressed individually.

redis-plus-plus

Let us first have a look at what sending a command a pipeline and a transaction look like

auto redis = Redis("tcp://127.0.0.1:6379");
// Send commands
redis.set("key", "val");
auto val = redis.get("key"); // val is of type OptionalString.
if (val)
std::cout << *val << std::endl;
// Sending pipelines
auto pipe = redis.pipeline();
auto pipe_replies = pipe.set("key", "value")
.get("key")
.rename("key", "new-key")
.rpush("list", {"a", "b", "c"})
.lrange("list", 0, -1)
.exec();
// Parse reply with reply type and index.
auto set_cmd_result = pipe_replies.get<bool>(0);
// ...
// Sending a transaction
auto tx = redis.transaction();
auto tx_replies = tx.incr("num0")
.incr("num1")
.mget({"num0", "num1"})
.exec();
auto incr_result0 = tx_replies.get<long long>(0);
// ...

Some of the problems with this API are

  • Heterogeneous treatment of commands, pipelines and transaction. This makes auto-pipelining impossible.
  • Any Api that sends individual commands has a very restricted scope of usability and should be avoided for performance reasons.
  • The API imposes exceptions on users, no error-code overload is provided.
  • No way to reuse the buffer for new calls to e.g. redis.get in order to avoid further dynamic memory allocations.
  • Error handling of resolve and connection not clear.

According to the documentation, pipelines in redis-plus-plus have the following characteristics

NOTE: By default, creating a Pipeline object is NOT cheap, since it creates a new connection.

This is clearly a downside of the API as pipelines should be the default way of communicating and not an exception, paying such a high price for each pipeline imposes a severe cost in performance. Transactions also suffer from the very same problem.

NOTE: Creating a Transaction object is NOT cheap, since it creates a new connection.

In Aedis there is no difference between sending one command, a pipeline or a transaction because requests are decoupled from the IO objects.

redis-plus-plus also supports async interface, however, async support for Transaction and Subscriber is still on the way.

The async interface depends on third-party event library, and so far, only libuv is supported.

Async code in redis-plus-plus looks like the following

auto async_redis = AsyncRedis(opts, pool_opts);
Future<string> ping_res = async_redis.ping();
cout << ping_res.get() << endl;

As the reader can see, the async interface is based on futures which is also known to have a bad performance. The biggest problem however with this async design is that it makes it impossible to write asynchronous programs correctly since it starts an async operation on every command sent instead of enqueueing a message and triggering a write when it can be sent. It is also not clear how are pipelines realised with the design (if at all).

Acknowledgement

Some people that were helpful in the development of Aedis

  • Richard Hodges (madmongo1): For helping me with Asio and the design of asynchronous programs in general.
  • Vinícius dos Santos Oliveira (vinipsmaker): For useful discussion about how Aedis consumes buffers in the read operation (among other things).
  • Petr Dannhofer (Eddie-cz): For helping me understand how the AUTH and HELLO command can influence each other.