Migrating from MinIO to Garage on a Raspberry Pi 5

I recently migrated my self-hosted S3-compatible storage from MinIO to Garage.

While MinIO is the industry leader for enterprise object storage, I needed something lighter for my homelab to serve Paperless-ngx (thousands of small files) and Audiobookshelf (large media files). My choice fell on Garage (v1.0+), an open-source distributed object store tailored for self-hosting.

Running a 4-node cluster on a single Raspberry Pi 5 with USB-connected HDDs presented unique challenges. Out of the box, write throughput flatlined at ~30 MB/s, which felt wrong for the hardware I had.

This post shows the migration and gives some idea how to setup Garage with Docker Compose on a Raspberry Pi 5, including how I verified the performance to break the 100 MB/s barrier.

The Hardware Setup

Host: Raspberry Pi 5 (8GB RAM)
Storage: 4x Mechanical HDDs (USB 3.0), formatted with XFS.
Orchestration: Docker Compose

The goal: Run a 4-node Garage cluster (one node per physical disk) on this single host to emulate a distributed system with data replication.

The Migration: Why Garage?

MinIO is fantastic, but its hardware requirements (especially regarding distinct drives for erasure coding) can be tricky in smaller setups. Apart from that, MinIO’s focus has shifted entirely towards enterprise customers. Long-time users have noticed a gradual decline in the usability of the free version: useful administration features in the mc client were steadily removed or locked behind paywalls over the years.

The recent move to a “Source Only Distribution” model clarifies the situation: MinIO no longer provides pre-compiled binary releases for the community version. While self-compiling or waiting for third-party maintainers (like Debian) to package software is standard procedure in the Linux world, it may add significant friction for homelab users who simply want a working, officially supported Docker container.

Garage fills this gap perfectly. It distributes ready-to-use Docker images, focuses on running efficiently on commodity hardware, and handles failures gracefully in mesh networks without the enterprise complexity.

Architecture Note:

I initially tested moving the Garage metadata database (LMDB) to the Pi’s NVMe SSD, assuming it would reduce latency. However, benchmarks showed no measurable difference for my workload (20k+ files). Garage uses LMDB, which memory-maps the database directly into the RAM. Since the Raspberry Pi 5 has ample RAM, the Linux Page Cache handles metadata access so efficiently that the mechanical drives were not the bottleneck. To avoid a Single Point of Failure, I kept both data and metadata on the HDDs.

Performance Verification (Breaking the 30 MB/s Limit)

After setting up the cluster, single-client uploads were stuck at 30 MB/s.

Since dd tests confirmed my USB drives could write at 106 MB/s individually, the bottleneck was clearly not the USB bus.

The Culprit: CPU & Parallelism

On a Raspberry Pi 5, calculating Blake3 hashes and ChaCha20 encryption for a single stream saturates one CPU core. Standard tools like mc (MinIO Client) often upload files sequentially, hitting this single-core limit.

To saturate the hardware, you must force parallelism. I didn’t need to tune the server; I just needed to test it correctly.

Benchmark 1: Parallel Write (Saturating the HDD)

I used rclone to benchmark uploads from a RAM disk (to exclude client-side bottlenecks), forcing 4 concurrent threads to utilize all Pi cores.

# Generate 1GB file in RAM
dd if=/dev/zero of=/dev/shm/1G.bin bs=1M count=1024

# Upload with forced concurrency (4 threads)
rclone copy /dev/shm/1G.bin garage:bench_test/ \
  --transfers 4 \
  --checkers 4 \
  --s3-upload-concurrency 4 \
  --progress

Result:

102.4 MiB/s (~107 MB/s)

I hit the physical write limit of the hard drive. By using all 4 cores of the Pi, I tripled the throughput compared to the default settings.

Benchmark 2: Parallel Read (Distributed Read)

This is where the distributed nature of Garage shines.

# Download to RAM, forcing 4 streams for a single file
rclone copy garage:bench_test/1G.bin /dev/shm/ \
  --transfers 4 \
  --multi-thread-streams 4 \
  --progress

Result:

204.8 MiB/s (~215 MB/s)

Since the data is striped across multiple disks, the Pi reads from multiple USB drives simultaneously, decrypts on multiple cores, and delivers double the speed of a single HDD.

My Configuration

Garage Node Configuration

Since all nodes run on the same host using the host network, we cannot use standard ports for everyone. We must manually assign distinct ports for the S3 API and the RPC (Inter-node communication).

Here is the configuration for Node 1 (/opt/docker/garage/config/node1.toml).

metadata_dir = "/var/lib/garage/meta"
data_dir = "/data"
db_engine = "lmdb"

# Replication factor is usually handled via 'garage layout', 
# but defining it here doesn't hurt.
replication_factor = 2

[rpc]
# The port for inter-node communication.
# Node 1: 3901 | Node 2: 3921 | Node 3: 3931 | Node 4: 3941
bind_addr = "[::]:3901"
# How other nodes reach this node (Localhost is fine here)
public_addr = "127.0.0.1:3901"

[s3_api]
# The port Nginx talks to.
# Node 1: 3900 | Node 2: 3920 | Node 3: 3930 | Node 4: 3940
api_bind_addr = "[::]:3900"
s3_region = "garage"
root_domain = ".s3.local"

[s3_web]
bind_addr = "[::]:3902" # Web website hosting (optional)
root_domain = ".web.local"

[k2v]
api_bind_addr = "[::]:3904" # Key-Value store (optional)

[admin]
api_bind_addr = "[::]:3903" # Admin API (optional)
metrics_token = "YOUR_METRICS_TOKEN" # Optional for Prometheus

Configuration for Other Nodes:

For Nodes 2, 3, and 4, you simply increment the ports to avoid conflicts on the host network interface.

Setting	Node 1	Node 2	Node 3	Node 4
RPC Bind	3901	3921	3931	3941
S3 API	3900	3920	3930	3940

Bootstrapping the Cluster:

To tell the nodes about each other initially, you can add a bootstrap_peers entry to all config files, or simply connect them once via CLI after startup:

docker compose exec node1 garage layout assign -z 1 -c 1TB 127.0.0.1:3901
docker compose exec node2 garage layout assign -z 1 -c 1TB 127.0.0.1:3921
docker compose exec node3 garage layout assign -z 1 -c 1TB 127.0.0.1:3931
docker compose exec node4 garage layout assign -z 1 -c 1TB 127.0.0.1:3941

# Connect them
docker compose exec node1 garage node connect 127.0.0.1:3921
docker compose exec node1 garage node connect 127.0.0.1:3931
docker compose exec node1 garage node connect 127.0.0.1:3941

Docker Compose

Here is the docker-compose.yml snippet to get this running efficiently. Note the use of network_mode: "host" to avoid Docker proxy overhead.

services:
  # --- NODE 1 ---
  # Garage is a masterless architecture. Node 1 is just the first peer.
  node1:
    image: dxflrs/garage:v1.0.1
    container_name: garage-1
    restart: unless-stopped
    # 'host' mode is critical for performance to bypass Docker NAT overhead
    # and to allow nodes to communicate via localhost without complex port mapping.
    network_mode: "host"
    # IMPORTANT: Replace with your host user:group IDs (id -u : id -g)
    # to ensure the container can read/write to your mounted USB drives.
    user: "993:989"
    volumes:
      - /opt/docker/garage/config/node1.toml:/etc/garage.toml
      - /storage/garage/disk1/meta:/var/lib/garage/meta
      - /storage/garage/disk1/data:/data
    environment:
      # 'info' level on one node is usually sufficient for monitoring cluster state.
      - RUST_LOG=info
    healthcheck:
      test: ["CMD", "/garage", "status"]
      interval: 10s
      timeout: 5s
      retries: 5

  # --- NODE 2 ---
  node2:
    image: dxflrs/garage:v1.0.1
    container_name: garage-2
    restart: unless-stopped
    network_mode: "host"
    user: "993:989"
    volumes:
      - /opt/docker/garage/config/node2.toml:/etc/garage.toml
      - /storage/garage/disk2/meta:/var/lib/garage/meta
      - /storage/garage/disk2/data:/data
    environment:
      # Reduced log level to keep logs clean, since Node 1 logs cluster events.
      - RUST_LOG=warn

  # --- NODE 3 ---
  node3:
    image: dxflrs/garage:v1.0.1
    container_name: garage-3
    restart: unless-stopped
    network_mode: "host"
    user: "993:989"
    volumes:
      - /opt/docker/garage/config/node3.toml:/etc/garage.toml
      - /storage/garage/disk3/meta:/var/lib/garage/meta
      - /storage/garage/disk3/data:/data
    environment:
      - RUST_LOG=warn

  # --- NODE 4 ---
  node4:
    image: dxflrs/garage:v1.0.1
    container_name: garage-4
    restart: unless-stopped
    network_mode: "host"
    user: "993:989"
    volumes:
      - /opt/docker/garage/config/node4.toml:/etc/garage.toml
      - /storage/garage/disk4/meta:/var/lib/garage/meta
      - /storage/garage/disk4/data:/data
    environment:
      - RUST_LOG=warn

  # --- LOAD BALANCER ---
  lb:
    image: nginx:alpine
    container_name: garage-lb
    # Nginx also needs host mode to route traffic to localhost ports (3900, 3902, etc.)
    # defined in the upstream config.
    network_mode: "host"
    volumes:
      - /opt/docker/garage/config/nginx.conf:/etc/nginx/nginx.conf:ro
    restart: unless-stopped

Nginx Configuration

To ensure smooth streaming uploads, I use Nginx as a load balancer in front of the 4 nodes.

events {
    # Standard connection limit, sufficient for home/SOHO setups.
    worker_connections 1024;
}

http {
    # Core optimizations for file transfer performance
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    # Remove upload size limits (Critical for S3 object storage!)
    client_max_body_size 0;

    # Upstream definition pointing to your 4 Garage nodes
    upstream garage_s3 {
        # 'least_conn' routes traffic to the node with the fewest active connections.
        # This is vital for stability when one node is busy with background tasks
        # (e.g., synchronization or garbage collection).
        least_conn;

        server 127.0.0.1:3900;
        server 127.0.0.1:3920;
        server 127.0.0.1:3930;
        server 127.0.0.1:3940;

        # CRITICAL: Keepalive pool.
        # Keeps 64 idle connections open to Garage nodes to eliminate 
        # TCP handshake latency for subsequent requests.
        keepalive 64;
    }

    server {
        listen 3999;
        server_name localhost;

        # Standard headers required for correct proxying
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # HTTP 1.1 is mandatory for Keepalive connections
        proxy_http_version 1.1;
        proxy_set_header Connection ""; # Clears the 'close' header

        location / {
            # STREAMING UPLOAD (Crucial for Performance)
            # Disables request buffering. Nginx passes data immediately to Garage 
            # instead of storing it locally first. drastically reduces TTFB on uploads.
            proxy_request_buffering off;

            # STREAMING DOWNLOAD
            # Disabling response buffering saves RAM on the Pi during large downloads
            # and makes streaming media (e.g., Audiobookshelf) feel snappier.
            proxy_buffering off;

            # Increased timeouts to handle large file transfers or
            # temporary lag during node re-balancing.
            proxy_connect_timeout 300;
            proxy_send_timeout    300;
            proxy_read_timeout    300;
            send_timeout          300;

            proxy_pass http://garage_s3;
        }
    }
}

Key Management & Permissions

Unlike MinIO’s default “Root User,” Garage encourages explicit key management. Since I run Garage inside Docker, I use the exec command to manage keys via the CLI.

I implemented a two-tier security strategy:

System Key (Admin): One key with access to all buckets, used by my host system for maintenance scripts and backups.
Service Keys: Dedicated keys for each application (e.g., one for Paperless, one for Audiobookshelf), restricted strictly to their own buckets.

Here is how to set this up:

# 1. Create a global admin key
docker compose exec node1 garage key new --name system-admin
# (Save the Access Key ID and Secret Key!)

# 2. Create a restricted service key
docker compose exec node1 garage key new --name paperless-app

# 3. Create buckets
docker compose exec node1 garage bucket create paperless-bucket
docker compose exec node1 garage bucket create audiobooks-bucket

# 4. Assign permissions
# Give the admin key access to everything
docker compose exec node1 garage bucket allow paperless-bucket --read --write --key system-admin
docker compose exec node1 garage bucket allow audiobooks-bucket --read --write --key system-admin

# Give the app key access ONLY to its specific bucket
docker compose exec node1 garage bucket allow paperless-bucket --read --write --key paperless-app

Conclusion

Migrating to Garage on a Raspberry Pi 5 has made my services significantly snappier. Paperless-ngx loads documents instantly, and Audiobookshelf streams without buffering.

Key Takeaways:

Hardware is capable: The Pi 5 can push >200 MB/s if you use parallel transfers.
Metadata on HDD is fine: Thanks to LMDB and Linux caching, SSDs are not mandatory for metadata in this scenario.
Test properly: Don’t trust single-threaded tools (like scp or mc) to measure your system’s maximum capacity.