Distributed MinIO on AWS Lightsail: Multi-Node Setup

MinIO is a high-performance, S3-compatible object storage solution. This article provides a blueprint for deploying a distributed MinIO stack using Amazon Lightsail, covering the critical steps for multi-node setup, networking, and Systemd.

Note: The setup documented uses 2 disks on 2 nodes in a distributed configuration. This is considered the bare minimum and is not recommended for production environments where high availability is paramount. One should always check the official documentation for minimum node/disk requirements. The operating system used is Ubuntu 22.04 LTS.

1. Instance Provisioning and Networking

AWS Lightsail Setup

The total cost for this two-node distributed cluster (including required block storage) quickly totals around $60 USD/month, demonstrating that self-hosting multi-node storage requires careful capacity planning.

Hostname and DNS Configuration

A correct Hostname and consistent DNS resolution are critical for distributed services. The configuration involves setting the hostname and updating /etc/hosts to ensure all nodes can resolve each other.

/etc/hostname
de-aws-mio1.jeanbruenn.info

/etc/hosts
127.0.0.1       localhost de-aws-mio1.jeanbruenn.info

# External nodes/load balancers for distributed communication
84.XXX.X.XX    de-fc-mio1.jeanbruenn.info    de-fc-mio1
3.XXX.XXX.XXX   de-aws-mio1.jeanbruenn.info   de-aws-mio1

2. MinIO Installation and Disk Management

Disk Formatting and Mounting

Lightsail VMs expose block storage using names like /dev/nvme*. The disks are formatted with XFS and disk labels (e.g., minio1-disk1) are used to ensure consistent mounting via fstab.

cfdisk /dev/nvme1n1 (create 1 GPT partition)
mkfs.xfs /dev/nvme1n1p1 -L minio1-disk1

# /etc/fstab entry
LABEL=minio1-disk1 /srv/minio/disk1 xfs defaults,noatime 0 2

# Create mount points
mkdir -p /srv/minio/disk1
mkdir /srv/minio/disk2

Installation and Binaries

The MinIO server and client binaries are downloaded, given executable permissions, and symlinked for easy execution.

# Installation path and symlinking
mkdir -p /opt/minio/bin
wget https://dl.min.io/server/minio/release/linux-amd64/minio 
wget https://dl.minio/client/mc/release/linux-amd64/mc
chmod +x minio mc
ln -s /opt/minio/bin/minio /usr/bin/minio
ln -s /opt/minio/bin/mc /usr/bin/mc

3. Systemd and Cluster Configuration

The Configuration File: /etc/default/minio

This file defines the entire cluster topology, including the Erasure Coding configuration via the MINIO_VOLUMES endpoint and the load balancer URL.

# /etc/default/minio

# Distributed topology: MUST include all nodes and disks
MINIO_VOLUMES="https://de-aws-mio{1...2}.jeanbruenn.info:9000/srv/minio/disk{1...2}"

# Load balancer endpoint for API access
MINIO_SERVER_URL="https://minio-lb.jeanbruenn.info:9000"

# MinIO Console UI port
MINIO_CONSOLE_ADDRESS=":9002"

# Security Credentials
MINIO_ROOT_USER="xxx"
MINIO_ROOT_PASSWORD="xxxxxx"

Systemd Unit (Hardening)

The Systemd unit is configured for high availability and performance by setting resource limits and using an unprivileged user.

[Unit]
Description=MinIO
...
[Service]
User=minio
Group=minio
ProtectProc=invisible

EnvironmentFile=-/etc/default/minio
ExecStart=/usr/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
Restart=always

# High file descriptor limit for concurrent S3 connections
LimitNOFILE=1048576

# Allows unlimited threads for high concurrency
TasksMax=infinity
...

Post-Install Permissions and TLS Setup

The MinIO binary is granted the capability to bind to privileged ports (cap_net_bind_service) without running as root, which is a key security measure. Certificates from Certbot must be issued and then owned by the minio user.

setcap 'cap_net_bind_service=+ep' /opt/minio/bin/minio
mkdir -p /etc/minio/certs
chown minio:minio /etc/minio -R

4. Verification and Troubleshooting

Firewall and Startup Check

The final step is opening the MinIO API (9000) and Console (9002) ports in the Lightsail Firewall (a common source of initial failure).

After starting the service (systemctl start minio), the logs confirm the successful initialization of the distributed pool:

de-aws-mio2 minio[1664]: MinIO Object Storage Server
...
de-aws-mio2 minio[1664]: Status: 4 Online, 0 Offline.
de-aws-mio2 minio[1664]: S3-API: https://minio-lb.jeanbruenn.info:9000
de-aws-mio2 minio[1664]: Console: https://172.26.12.161:9002 https://127.0.0.1:9002

MinIO Client (mc) Setup and Cluster Health

The mc client is configured to access the cluster via the load balancer, which then provides a clear view of the entire distributed system status.

# Configure the client alias
mc alias set de-aws-mio https://minio-lb.jeanbruenn.info:9000 MINIO_ROOT_USER MINIO_ROOT_PASSWORD

# Example of mc verification output (Proof of Concept)
~# mc admin info de-aws-mio1
● de-aws-mio1.jeanbruenn.info:9000
Uptime: 2 minutes
Version: 2023-10-07T15:07:38Z
Network: 2/2 OK
Drives: 2/2 OK
Pool: 1

Errors and Solutions

The following errors were encountered during deployment, highlighting typical traps:

Error TypeLog Entry (Excerpt)Root Cause & Solution
Binding ErrorERROR Unable to start the server: listen tcp 18.XXX.XX.XXX:9000: bind: cannot assign requested addressLightsail NAT: The instance cannot bind to its Public IP internally. Solution: Ensure the hostname resolves to 127.0.0.1 in /etc/hosts for local services.
Server ValidationERROR Unable to validate passed arguments: host in server address should be this serverSolution: Temporarily use the instance’s hostname (e.g., de-aws-mio2.jeanbruenn.info) in MINIO_SERVER_URL until the primary load balancer is in place.
Distributed Quorum FailureError: Read failed. Insufficient number of drives onlineRoot Cause: A basic firewall failure. The MinIO nodes could not communicate on the private network ports (9000-9010), meaning the cluster could not reach its minimum quorum.
Certificate/TLS LoadERROR Unable to load the TLS configuration: HTTPS specified in endpoints, but no TLS certificate is foundSolution: Certificates cannot be symlinked from Let’s Encrypt. The certificates must be copied and owned by the minio user in /etc/minio/certs.

Sources / See Also

  • MinIO Documentation. MinIO Distributed Deployment Guide. https://min.io/docs/minio/linux/deployment/distributed-deployment/
  • MinIO Documentation. Install MinIO on Amazon EKS in 15 Minutes or Less. https://blog.min.io/install-minio-amazon-eks/
  • MinIO Documentation. How to Install MinIO in Distributed Mode on AWS EC2. https://blog.minio/install-minio-distributed-mode-aws-ec2/
  • MinIO Documentation. MinIO Troubleshooting Guide. https://min.io/docs/minio/linux/operations/install-deploy-manage/troubleshoot.html
  • AWS Lightsail Documentation. Networking and Firewall Rules. https://docs.aws.amazon.com/lightsail/latest/userguide/configuring-firewall-for-lightsail.html
  • Linux Security. Understanding capabilities and setcap. https://man7.org/linux/man-pages/man7/capabilities.7.html

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.