ReductStore Bucket Guide

Buckets are the primary storage unit in ReductStore. They are used to group data entries and define the storage settings for the data. This guide will cover the concepts of buckets, their settings, and operations like creating, browsing, changing settings, and removing buckets.

Concepts

A bucket is a container used for data storage and serves as a logical grouping of data. Each bucket can contain multiple entries, with each entry representing a time-series dataset composed of time-stamped records. If you're familiar with databases, think of a bucket as a database, and entries as tables.

Given the nature of time-series data, ReductStore partitions data into blocks. Each block, stored in a separate file, contains a set of records. There are limits to the block size and the number of records it can hold. When a block reaches its limit, a new block is created to store additional data. This method allows ReductStore to store and access data efficiently. You can read more about blocks and records in the How Does It Work? document.

Unlike other blob storage or file systems, ReductStore has a flat hierarchy. It doesn't use folders or directories. Instead, all data is stored in entries, which are grouped by buckets.

Bucket Settings

Each bucket has settings that determine how data is stored and accessed. These settings include:

• Quota Type: You can choose between NONE, which means there is no quota, and FIFO (First In, First Out), where the oldest data is removed to make space for new data. The default setting is NONE.
• Quota Size: This represents the quota size in bytes. Only enforced with FIFO Quota.
• Max Block Size: This is the maximum allowable size of a block, in bytes. The default size is 64MB.
• Maximal Number of Records: This is the maximum number of records that can be contained in a block. The default number is 1024.

Quota Type

The quota type determines the method of quota enforcement and can be set to either NONE, FIFO or HARD.

NONE implies no quota, allowing data to be stored without any restrictions.

FIFO, short for first-in-first-out, enforces the quota by deleting the oldest block to accommodate new data once the quota is reached. ReductStore ensures that the data does not surpass the quota size, refusing to store new data if there's insufficient room.

HARD is a strict quota type that prevents data from being stored once the quota is reached. This setting is useful when you want to limit the amount of data stored in a bucket but don't want to delete old data.

Quota Size

The quota size defines the maximum size of the bucket in bytes. It's ignored if the quota type is set to NONE.

Max Block Size

The max block size determines the maximum size of a block in bytes. Once a block reaches this limit, a new block is created for storing additional data. A record's size isn't restricted by the block size. If a record exceeds the block size, it will be stored in the current block, and a new block will be created for the next record.

The default value is set at 64MB. Typically, there is no need to alter this unless your records exceed the default size. ReductStore pre-allocates the block size during the creation of a new block. As such, it improves performance when multiple records can be stored in a single block. We recommend to keep the block size large enough to store at least 1024 records.

Maximal Number of Records

The maximal number of records refers to the maximum number of records that can be stored in a block. When a block reaches this capacity, a new block is created for additional data. The default value is 1024. Generally, you don't need to adjust this unless you have numerous small records and an excess of blocks, which could impact search performance.

To optimize search performance, bear in mind that record searching consists of two steps:

• Locating the block that contains the record. This search time is O(log(n)), where n represents the number of blocks.
• Finding the record within the block. This search time is O(m), with m representing the number of records in the block.

If there are too many blocks, the time spent searching for a block could exceed the time spent searching for the record within the block. In such cases, consider increasing the maximal number of records to reduce the total number of blocks.

Bucket Operations

Here you will find examples of how to create, browse, change settings, and remove buckets using the ReductStore SDKs, CLI client, Web Console, and REST API. Pay attention that all the examples are written for a local ReductStore instance available at http://127.0.0.1:8383 with API token my-token.

For more information on how to set up a local ReductStore instance, refer to the Getting Started guide.

Creating a Bucket

A bucket can be created using the SDKs, CLI client, Web Console, or REST API. The bucket name must be unique within the store, and a client must have full access permission if the authentication is enabled. Provisioning a bucket with environment variables is also possible. Refer to the example below:

CLI

reduct-cli alias add local -L http://localhost:8383 -t "my-token"
reduct-cli bucket create local/my-bucket --quota-type FIFO --quota-size 1GB

Web Console

Steps to create a bucket using the Web Console:

1. Open the Web Console at http://127.0.0.1:8383 in your browser.
2. Enter the API token if the authorization is enabled.
3. Click on the "Buckets" tab in the left sidebar.
4. Click on the plus icon in the top right corner to create a new bucket:

5. In the "Add a new bucket" dialog, enter the bucket name and settings.
6. Click on the "Create" button:

Python

from reduct import Client, BucketSettings, QuotaType


async def create_bucket():
    # Create a client with the base URL and API token
    async with Client("http://localhost:8383", api_token="my-token") as client:
        # Create a bucket with the name "my-bucket" and a FIFO quota of 1GB
        settings = BucketSettings(
            quota_type=QuotaType.FIFO,
            quota_size=1000_000_000,
        )
        bucket = await client.create_bucket("my-bucket", settings, exist_ok=True)
        assert bucket.name == "my-bucket"


if __name__ == "__main__":
    import asyncio

    asyncio.run(create_bucket())

JavaScript

import { Client, QuotaType } from "reduct-js";
import assert from "node:assert";

// Create a new client with the server URL and an API token
const client = new Client("http://127.0.0.1:8383", { apiToken: "my-token" });

// Create a bucket with the name "my-bucket" and a FIFO quota of 1GB
let settings = {
  quotaType: QuotaType.FIFO,
  quotaSize: 1000_000_000n,
};
const bucket = await client.getOrCreateBucket("bucket", settings);
assert(bucket.name === "bucket");

Rust

use reduct_rs::{QuotaType, ReductClient, ReductError};
use tokio;

#[tokio::main]
async fn main() -> Result<(), ReductError> {
    // Create a new client with the API URL and API token
    let client = ReductClient::builder()
        .url("http://127.0.0.1:8383")
        .api_token("my-token")
        .build();

    // Create a bucket with the name "my-bucket" and a FIFO quota of 1GB
    let bucket = client
        .create_bucket("my-bucket")
        .quota_type(QuotaType::FIFO)
        .quota_size(1_000_000_000)
        .exist_ok(true)
        .send()
        .await?;

    assert_eq!(bucket.name(), "my-bucket");
    Ok(())
}

C++

#include <reduct/client.h>
#include <iostream>
#include <cassert>

using reduct::IBucket;
using reduct::IClient;
using reduct::Error;

int main() {
    // Create a client with the server URL
    auto client = IClient::Build("http://127.0.0.1:8383", {
            .api_token = "my-token"
    });

    // Create a bucket with the name "my-bucket" and a FIFO quota of 1GB
    IBucket::Settings settings;
    settings.quota_type = IBucket::QuotaType::kFifo;
    settings.quota_size = 1'000'000'000;
    auto [bucket, create_err] = client->GetOrCreateBucket("my-bucket", settings);

    assert(create_err == Error::kOk);
    assert(bucket->GetInfo().result.name == "my-bucket");
}

cURL

#!/bin/bash
set -e -x

API_PATH="http://127.0.0.1:8383/api/v1"
AUTH_HEADER="Authorization: Bearer my-token"

curl -X POST \
  -d '{"quota_type":"FIFO", "quota_size":1000000000}' \
  -H "${AUTH_HEADER}" \
  -a "${API_PATH}"/b/my_data

Provisioning

version: "3"
services:
  reductstore:
    image: reduct/store:latest
    ports:
      - "8383:8383"
    volumes:
      - ./data:/data
    environment:
      - RS_API_TOKEN=my-api-token
      - RS_BUCKET_1_NAME=my-bucket
      - RS_BUCKET_1_QUOTA_TYPE=FIFO
      - RS_BUCKET_1_QUOTA_SIZE=1GB

Browse Buckets

You might want to view the list of buckets in your store or see the details of a specific bucket. This can be done by using the SDKs, CLI client, Web Console, or REST API. For listing all buckets, a client must have a valid access token if the authorization is enabled.

CLI

reduct-cli alias add local -L http://localhost:8383 -t "my-token"
reduct-cli bucket ls local --full

# Output:
#| Name              | Entries | Size     | Oldest record (UTC)      | Latest record (UTC)      |
#|-------------------|---------|----------|--------------------------|--------------------------|
#| example-bucket    | 3       | 30.7 GB  | 1970-01-01T00:00:00.000Z | 1970-01-01T00:00:00.060Z |


reduct-cli bucket show local/example-bucket --full

# Output:
#Name:                datasets                       Quota Type:         NONE
#Entries:             3                              Quota Size:         0 B
#Size:                30.7 GB                        Max. Block Size:    64.0 MB
#Oldest Record (UTC): 1970-01-01T00:00:00.000Z       Max. Block Records: 256
#Latest Record (UTC): 1970-01-01T00:00:00.060Z
#
#| Name           | Records | Blocks | Size    | Oldest Record (UTC)      | Latest Record (UTC)      |
#|----------------|---------|--------|---------|--------------------------|--------------------------|
#| cats           | 9993    | 40     | 2.0 GB  | 1970-01-01T00:00:00.000Z | 1970-01-01T00:00:00.016Z |
#| imdb           | 46641   | 459    | 28.7 GB | 1970-01-01T00:00:00.000Z | 1970-01-01T00:00:00.046Z |
#| mnist_training | 60000   | 236    | 17.5 MB | 1970-01-01T00:00:00.000Z | 1970-01-01T00:00:00.060Z |

Web Console

Steps to browse buckets using the Web Console:

1. Open the Web Console at http://127.0.0.1:8383 in your browser.
2. Enter the API token if the authorization is enabled.
3. Click on the "Buckets" tab in the left sidebar.
4. You will see a list of all buckets in the store.
5. Click on a specific bucket to view its details:

6. You can see the bucket settings by clicking on the cog icon(⚙️) in the bucket panel:

Python

from typing import List

from reduct import Client, Bucket, BucketInfo


async def browse_buckets():
    # Create a client with the base URL and API token
    async with Client("http://localhost:8383", api_token="my-token") as client:
        # Browse all buckets and print their information
        buckets: List[BucketInfo] = await client.list()
        for info in buckets:
            print(f"Bucket: {info.name}")
            print(f"Size: {info.size}")
            print(f"Entry Count: {info.entry_count}")
            print(f"Oldest Record: {info.oldest_record}")
            print(f"Latest Record: {info.latest_record}")

        # Get information about a specific bucket
        bucket: Bucket = await client.get_bucket("example-bucket")
        info = await bucket.get_full_info()
        print(f"Bucket settings: {info.settings}")

        for entry in info.entries:
            print(f"Entry: {entry.name}")
            print(f"Size: {entry.size}")
            print(f"Oldest Record: {entry.oldest_record}")
            print(f"Latest Record: {entry.latest_record}")


if __name__ == "__main__":
    import asyncio

    asyncio.run(browse_buckets())

JavaScript

import { Client } from "reduct-js";

// Create a new client with the server URL and an API token
const client = new Client("http://127.0.0.1:8383", { apiToken: "my-token" });

// Browse the list of buckets and print their information
for (const info of await client.getBucketList()) {
  console.log(`Bucket: ${info.name}`);
  console.log(`Size: ${info.size}`);
  console.log(`Entry Count: ${info.entryCount}`);
  console.log(`Oldest Record: ${info.oldestRecord}`);
  console.log(`Latest Record: ${info.latestRecord}`);
}

// Get information about a specific bucket
const bucket = await client.getBucket("example-bucket");
const settings = await bucket.getSettings();
console.log(`Settings: ${settings}`);
for (const info of await bucket.getEntryList()) {
  console.log(`Entry: ${info.name}`);
  console.log(`Size: ${info.size}`);
  console.log(`Oldest Record: ${info.oldestRecord}`);
  console.log(`Latest Record: ${info.latestRecord}`);
}

Rust

use reduct_rs::{Bucket, ReductClient, ReductError};

#[tokio::main]
async fn main() -> Result<(), ReductError> {
    // Create a new client with the API URL and API token
    let client = ReductClient::builder()
        .url("http://127.0.0.1:8383")
        .api_token("my-token")
        .build();

    // Browse all buckets and print their information
    let buckets = client.bucket_list().await?.buckets;
    for info in buckets {
        println!("Bucket: {}", info.name);
        println!("Size: {}", info.size);
        println!("Entry count: {}", info.entry_count);
        println!("Oldest Record: {:?}", info.oldest_record);
        println!("Latest Record: {:?}", info.latest_record);
    }

    // Get information about a specific bucket
    let bucket: Bucket = client.get_bucket("example-bucket").await?;
    let settings = bucket.settings().await?;
    println!("Settings: {:?}", settings);

    for entry in bucket.entries().await? {
        println!("Entry: {:?}", entry.name);
        println!("Size: {:?}", entry.size);
        println!("Oldest Record: {:?}", entry.oldest_record);
        println!("Latest Record: {:?}", entry.latest_record);
    }

    Ok(())
}

C++

#include <reduct/client.h>
#include <iostream>
#include <cassert>

using reduct::IBucket;
using reduct::IClient;
using reduct::Error;

std::string PrintTime(std::chrono::system_clock::time_point time) {
    auto now_time_t = std::chrono::system_clock::to_time_t(time);
    auto now_tm = std::localtime(&now_time_t);
    char buf[32];
    std::strftime(buf, sizeof(buf), "%Y-%m-%d %H:%M:%S", now_tm);
    return buf;
}

int main() {
    // Create a client with the server URL
    auto client = IClient::Build("http://127.0.0.1:8383", {
            .api_token = "my-token"
    });

    //  Browse all buckets and print their information
    auto [list, list_err] = client->GetBucketList();
    assert(list_err == Error::kOk);

    for (const auto &bucket: list) {
        std::cout << "Bucket: " << bucket.name << std::endl;
        std::cout << "Size: " << bucket.size << std::endl;
        std::cout << "Entry count: " << bucket.entry_count << std::endl;
        std::cout << "Latest Record: " << PrintTime(bucket.latest_record) << std::endl;
        std::cout << "Oldest Record: " << PrintTime(bucket.oldest_record) << std::endl;
    }

    // Get information about a specific bucket
    auto [bucket, get_err] = client->GetBucket("example-bucket");
    assert(get_err == Error::kOk);

    std::cout << "Bucket Settings: " << bucket->GetSettings().result << std::endl;
    for (const auto &entry: bucket->GetEntryList().result) {
        std::cout << "Entry: " << entry.name << std::endl;
        std::cout << "Size: " << entry.size << std::endl;
        std::cout << "Latest Record: " << PrintTime(entry.latest_record) << std::endl;
        std::cout << "Oldest Record: " << PrintTime(entry.oldest_record) << std::endl;
    }

    return 0;
}

cURL

#!/bin/bash
set -e -x

API_PATH="http://127.0.0.1:8383/api/v1"
AUTH_HEADER="Authorization: Bearer my-token"

printf "Browse all buckets:\n"
curl -X GET \
  -H "${AUTH_HEADER}" \
  -a "${API_PATH}"/list

printf "\nBrowse a specific bucket:\n"
curl -X GET \
  -H "${AUTH_HEADER}" \
  -a "${API_PATH}"/b/my-bucket

Changing Bucket Settings

Bucket settings can be modified using the SDKs, CLI client, Web Console, or REST API. A client must have full access permission to change the settings if the authorization is enabled. The settings that can be modified include the quota type, quota size, max block size, and maximal number of records. You cannot change the bucket name.

CLI

reduct-cli alias add local -L http://localhost:8383 -t "my-token"
# Change the quota size of the bucket
reduct-cli bucket update local/example-bucket --quota-size 10GB

Web Console

Steps to change bucket setting using the Web Console:

1. Open the Web Console at http://127.0.0.1:8383 in your browser.
2. Enter the API token if the authorization is enabled.
3. Click on the "Buckets" tab in the left sidebar.
4. You will see a list of all buckets in the store.
5. Click on a specific bucket in the list:

6. Open the bucket settings by clicking on cog icon(⚙️) in the bucket panel:

7. Modify the settings as needed and click on the Update button:

Python

from reduct import Client, BucketSettings, QuotaType


async def change_bucket():
    # Create a client with the base URL and API token
    async with Client("http://localhost:8383", api_token="my-token") as client:
        # Get an existing bucket
        bucket = await client.get_bucket("example-bucket")

        # Change the quota size of the bucket to 5GB
        new_settings = BucketSettings(quota_size=5000_000_000)
        await bucket.set_settings(new_settings)

        assert (await bucket.get_settings()).quota_size == 5000_000_000


if __name__ == "__main__":
    import asyncio

    asyncio.run(change_bucket())

JavaScript

import { Client } from "reduct-js";
import assert from "node:assert";

// Create a new client with the server URL and an API token
const client = new Client("http://127.0.0.1:8383", { apiToken: "my-token" });

// Get an existing bucket
const bucket = await client.getBucket("example-bucket");

// Change the quota size of the bucket to 5GB
let new_settings = {
  quotaSize: 5000_000_000n,
};
await bucket.setSettings(new_settings);

assert((await bucket.getSettings()).quotaSize === 5000_000_000n);

Rust

use reduct_rs::{BucketSettings, ReductClient, ReductError};
use tokio;

#[tokio::main]
async fn main() -> Result<(), ReductError> {
    // Create a new client with the API URL and API token
    let client = ReductClient::builder()
        .url("http://127.0.0.1:8383")
        .api_token("my-token")
        .build();

    // Get an existing bucket
    let bucket = client.get_bucket("example-bucket").await?;

    // Change the quota size of the bucket to 5GB
    let new_settings = BucketSettings {
        quota_size: Some(5_000_000_000),
        ..BucketSettings::default()
    };
    bucket.set_settings(new_settings).await?;

    assert_eq!(bucket.settings().await?.quota_size, Some(5_000_000_000));
    Ok(())
}

C++

#include <reduct/client.h>
#include <iostream>
#include <cassert>

using reduct::IBucket;
using reduct::IClient;
using reduct::Error;

int main() {
    // Create a client with the server URL
    auto client = IClient::Build("http://127.0.0.1:8383", {
            .api_token = "my-token"
    });

    // Get an existing bucket
    auto [bucket, get_err] = client->GetBucket("example-bucket");
    assert(get_err == Error::kOk);

    // Change the quota size of the bucket to 5GB
    IBucket::Settings new_settings;
    new_settings.quota_size = 5'000'000'000;
    auto change_err = bucket->UpdateSettings(new_settings);

    assert(change_err == Error::kOk);
    assert(bucket->GetSettings().result.quota_size == 5'000'000'000);
}

cURL

#!/bin/bash
set -e -x

API_PATH="http://127.0.0.1:8383/api/v1"
AUTH_HEADER="Authorization: Bearer my-token"

# Change the quota size of the bucket
curl -X UPDATE \
  -d '{"quota_size":5000000000}' \
  -H "${AUTH_HEADER}" \
  -a "${API_PATH}"/b/my_data

note

You can't change settings for provisioned buckets. You should unprovision it first.

Removing a Bucket

A bucket can be deleted using the SDKs, CLI client, Web Console, or REST API. A client must have full access permission to delete a bucket if the authorization is enabled.

CLI

reduct-cli alias add local -L http://localhost:8383 -t "my-token"
# Remove the bucket without confirmation
reduct-cli bucket rm local/bucket-to-remove -y

Web Console

Steps to remove a bucket using the Web Console:

1. Open the Web Console at http://127.0.0.1:8383 in your browser.
2. Enter the API token if the authorization is enabled.
3. Click on the "Buckets" tab in the left sidebar.
4. You will see a list of all buckets in the store.
5. Click on a specific bucket in the list:

6. Click on the Trash icon in the bucket panel:

7. Confirm the deletion by typing the bucket name and clicking on the Remove button:

Python

from reduct import Client, ReductError


async def remove_bucket():
    # Create a client with the base URL and API token
    async with Client("http://127.0.0.1:8383", api_token="my-token") as client:
        # Remove the bucket with the name "bucket-to-remove"

        bucket = await client.get_bucket("bucket-to-remove")
        await bucket.remove()

        # Check that the bucket no longer exists
        try:
            await client.get_bucket("bucket-to-remove")
        except ReductError as e:
            # The bucket should not exist anymore
            assert e.status_code == 404


if __name__ == "__main__":
    import asyncio

    asyncio.run(remove_bucket())

JavaScript

import { Client } from "reduct-js";
import assert from "node:assert";

// Create a new client with the server URL and an API token
const client = new Client("http://127.0.0.1:8383", { apiToken: "my-token" });

// Remove the bucket with the name "bucket-to-remove"
const bucket = await client.getBucket("bucket-to-remove");
await bucket.remove();

// Check that the bucket no longer exists
try {
  await client.getBucket("bucket-to-remove");
} catch (e) {
  assert(e.status === 404);
}

Rust

use reduct_rs::{ErrorCode, ReductClient, ReductError};
use tokio;

#[tokio::main]
async fn main() -> Result<(), ReductError> {
    // Create a new client with the API URL and API token
    let client = ReductClient::builder()
        .url("http://127.0.0.1:8383")
        .api_token("my-token")
        .build();

    // Remove the bucket "bucket-to-remove"
    let bucket = client.get_bucket("bucket-to-remove").await?;
    bucket.remove().await?;

    // Check that the bucket no longer exists
    let bucket = client.get_bucket("bucket-to-remove").await;
    assert_eq!(bucket.err().unwrap().status, ErrorCode::NotFound);
    Ok(())
}

C++

#include <reduct/client.h>
#include <iostream>
#include <cassert>

using reduct::IBucket;
using reduct::IClient;
using reduct::Error;

int main() {
    // Create a client with the server URL
    auto client = IClient::Build("http://127.0.0.1:8383", {
            .api_token = "my-token"
    });

    // Remove the bucket with the name "bucket-to-remove"
    auto [bucket, get_err] = client->GetBucket("bucket-to-remove");
    assert(get_err == Error::kOk);
    auto remove_err = bucket->Remove();
    assert(remove_err == Error::kOk);

    // Check that the bucket no longer exists
    auto [_, get_err2] = client->GetBucket("bucket-to-remove");
    assert(get_err2.code == 404);
}

cURL

#!/bin/bash
set -e -x

API_PATH="http://127.0.0.1:8383/api/v1"
AUTH_HEADER="Authorization: Bearer my-token"

curl -X DELETE \
  -H "${AUTH_HEADER}" \
  -a "${API_PATH}"/b/bucket-to-remove

warning

Removing a bucket will also remove all the data stored in it. This action is irreversible.

note

You can't remove a provisioned bucket. You should unprovision it first.