I am happy to announce that we have completed the migration from C++ to Rust, and have released a stable version (v1.4.0) that is entirely written in Rust. 🤩
It was not an easy journey. After six weeks of coding, we encountered numerous regressions and changes in behavior. I needed to release two alpha and two beta versions with production testing to clean up the database. Now, it is finally ready!
This article provides an introduction to ReductStore and explains how to use the Reduct C++ SDK to subscribe to data from the database.
To subscribe to new records, we should use a continuous query, which has been supported by ReductStore since version v1.4. We can use the following Docker command to run it:
docker pull reduct/store:latest
docker run -p 8383:8383 reduct/store:latest
Now, we need to install the Reduct Client SDK for C++. Please refer to these instructions.
I previously wrote about my plan to rewrite ReductStore in Rust. I am happy to announce that the migration is now complete. It was a very interesting experience, and I would like to share it here.
I wrote ReductStore in C++20, utilizing coroutines and ranges.
For the HTTP frontend, I used uWebSockets as an HTTP server and its event loop for coroutines.
The storage engine was implemented from scratch.
I used Protobuf as a JSON and binary serializer in both the HTTP frontend and the storage engine. Many of the structures were shared between the two.
I managed dependencies with Conan and used CMake as a build system.
Codebase about 20k lines with unit tests.
I have been developing in C++ and Python for about five years. Mostly, I write services for data acquisition, processing, and storage. I like OOP, design patterns, and my C++ smells Java. However, I avoid using exceptions and follow the RAII approach.
In Rust, I have only written the "Hello, World" example.
I work on the project in my spare time 10-20 hours a week.
Initially, I planned to use the cxx.rs library and rewrite the project in small steps by wrapping Rust code and integrating it into C++.
[SPOILER] I wasn't able to handle this...
Before we dive into the importance of data reduction for edge computing, it is important to define both terms. Data reduction refers to the process of reducing the amount of data that needs to be transmitted or stored, while still maintaining the necessary information and level of accuracy. This can be achieved through techniques such as compression, aggregation, and filtering.
Initially, I chose to use C++ for the early editions of ReductStore because of my experience with the language. This allowed me to quickly create a functional time series database for binary data. However, as our platform expanded to include Windows and MacOS, I found myself struggling to manage the C++ infrastructure as the codebase grew. This made it difficult for me to focus on enhancing the product's functionality and unique features, as I had to ensure compatibility across multiple platforms while managing numerous dependencies.
Hey, I've released version 0.8.0 of Reduct CLI, the Python package for managing data stored in ReductStore. This release includes two new features that will be particularly helpful for our public datasets hosted on ReductStore, where metadata can be used to provide important context for the data.
The rcli export folder command now has a new option --with-metadata, which exports meta information about a
record and its labels into a JSON file along with the content of the record.
The rcli export commands now accept integers for --start and --stop options, which means you can specify a time
interval in Unix time or use it as a range of record IDs if you don't care about time:
# Export records with ID from 0 to 5
rcli export folder instance/bucket ./export_path --start 0 --stop 5
I hope these new features make it even easier to work with ReductStore and manage your data. As always, if you have any questions or feedback, please don't hesitate to reach out!
Thanks for using ReductStore!
The ReductStore Project hosts the free "cats" dataset, which contains about 10K photos of cats in JPEG format with eye coordinates, a month, and ears as labels. In this example, we can learn how to download the dataset from the ReductStore instance and draw the features using the OpenCV library.
First, we need to install the ReductStore Client SDK for Python to download the photos and labels. We also need the OpenCV Python library for drawing features and Pillow to display images in the Jupyter environment:
pip install reduct-py opencv-python Pillow
To retrieve data we need the URL of the ReductStore instance, the bucket name, where we store our datasets and an API token with read access, so that we can connect to the database by using the Client class:
from reduct import Client, Bucket
HOST = "https://play.reduct.store"
API_TOKEN = "dataset-read-eab13e4f5f2df1e64363806443eea7ba83406ce701d49378d2f54cfbf02850f5"
BUCKET = "datasets"
client = Client(HOST, api_token=API_TOKEN)
bucket: Bucket = await client.get_bucket(BUCKET)
Using snap, users can now easily install and manage ReductStore on various Linux distributions.
Snap is a universal package manager developed by Canonical, the company behind Ubuntu Linux. It allows developers to package their applications and dependencies into a single package that can be installed on any Linux distribution that supports snap, without worrying about different packaging formats and dependency conflicts.
We are excited to announce the release of ReductStore Client SDK for C++ v1.3.0! This release includes support for the ReductStore API v1.3.0 with labels and content type.
Hey everyone,
I'm happy to announce that we have released Reduct CLI client v0.7.0 with some minor improvements and bug fixes. We started using the tool in real applications and faced some problems exporting data from a ReductStore instance when the connection is slow and we have many entries to download asynchronously.
First of all, it wasn't very convenient to count all needed entries in the rcli export command. Now we can use
wildcards:
rcli export folder instance/bucket ./export_path --entries=sensor-*