MinIO is a high-performance, S3-compatible object storage solution. This article provides a blueprint for deploying a distributed MinIO stack using Amazon Lightsail, covering the critical steps for multi-node setup, networking, and Systemd.
Note: The setup documented uses 2 disks on 2 nodes in a distributed configuration. This is considered the bare minimum and is not recommended for production environments where high availability is paramount. One should always check the official documentation for minimum node/disk requirements. The operating system used is Ubuntu 22.04 LTS.
1. Instance Provisioning and Networking
AWS Lightsail Setup
The total cost for this two-node distributed cluster (including required block storage) quickly totals around $60 USD/month, demonstrating that self-hosting multi-node storage requires careful capacity planning.
Hostname and DNS Configuration
A correct Hostname and consistent DNS resolution are critical for distributed services. The configuration involves setting the hostname and updating /etc/hosts to ensure all nodes can resolve each other.
/etc/hostname
de-aws-mio1.jeanbruenn.info
/etc/hosts
127.0.0.1 localhost de-aws-mio1.jeanbruenn.info
# External nodes/load balancers for distributed communication
84.XXX.X.XX de-fc-mio1.jeanbruenn.info de-fc-mio1
3.XXX.XXX.XXX de-aws-mio1.jeanbruenn.info de-aws-mio1
2. MinIO Installation and Disk Management
Disk Formatting and Mounting
Lightsail VMs expose block storage using names like /dev/nvme*. The disks are formatted with XFS and disk labels (e.g., minio1-disk1) are used to ensure consistent mounting via fstab.
cfdisk /dev/nvme1n1 (create 1 GPT partition)
mkfs.xfs /dev/nvme1n1p1 -L minio1-disk1
# /etc/fstab entry
LABEL=minio1-disk1 /srv/minio/disk1 xfs defaults,noatime 0 2
# Create mount points
mkdir -p /srv/minio/disk1
mkdir /srv/minio/disk2
Installation and Binaries
The MinIO server and client binaries are downloaded, given executable permissions, and symlinked for easy execution.
# Installation path and symlinking
mkdir -p /opt/minio/bin
wget https://dl.min.io/server/minio/release/linux-amd64/minio
wget https://dl.minio/client/mc/release/linux-amd64/mc
chmod +x minio mc
ln -s /opt/minio/bin/minio /usr/bin/minio
ln -s /opt/minio/bin/mc /usr/bin/mc
3. Systemd and Cluster Configuration
The Configuration File: /etc/default/minio
This file defines the entire cluster topology, including the Erasure Coding configuration via the MINIO_VOLUMES endpoint and the load balancer URL.
# /etc/default/minio
# Distributed topology: MUST include all nodes and disks
MINIO_VOLUMES="https://de-aws-mio{1...2}.jeanbruenn.info:9000/srv/minio/disk{1...2}"
# Load balancer endpoint for API access
MINIO_SERVER_URL="https://minio-lb.jeanbruenn.info:9000"
# MinIO Console UI port
MINIO_CONSOLE_ADDRESS=":9002"
# Security Credentials
MINIO_ROOT_USER="xxx"
MINIO_ROOT_PASSWORD="xxxxxx"
Systemd Unit (Hardening)
The Systemd unit is configured for high availability and performance by setting resource limits and using an unprivileged user.
[Unit]
Description=MinIO
...
[Service]
User=minio
Group=minio
ProtectProc=invisible
EnvironmentFile=-/etc/default/minio
ExecStart=/usr/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
Restart=always
# High file descriptor limit for concurrent S3 connections
LimitNOFILE=1048576
# Allows unlimited threads for high concurrency
TasksMax=infinity
...
Post-Install Permissions and TLS Setup
The MinIO binary is granted the capability to bind to privileged ports (cap_net_bind_service) without running as root, which is a key security measure. Certificates from Certbot must be issued and then owned by the minio user.
setcap 'cap_net_bind_service=+ep' /opt/minio/bin/minio
mkdir -p /etc/minio/certs
chown minio:minio /etc/minio -R
4. Verification and Troubleshooting
Firewall and Startup Check
The final step is opening the MinIO API (9000) and Console (9002) ports in the Lightsail Firewall (a common source of initial failure).
After starting the service (systemctl start minio), the logs confirm the successful initialization of the distributed pool:
de-aws-mio2 minio[1664]: MinIO Object Storage Server
...
de-aws-mio2 minio[1664]: Status: 4 Online, 0 Offline.
de-aws-mio2 minio[1664]: S3-API: https://minio-lb.jeanbruenn.info:9000
de-aws-mio2 minio[1664]: Console: https://172.26.12.161:9002 https://127.0.0.1:9002
MinIO Client (mc) Setup and Cluster Health
The mc client is configured to access the cluster via the load balancer, which then provides a clear view of the entire distributed system status.
# Configure the client alias
mc alias set de-aws-mio https://minio-lb.jeanbruenn.info:9000 MINIO_ROOT_USER MINIO_ROOT_PASSWORD
# Example of mc verification output (Proof of Concept)
~# mc admin info de-aws-mio1
● de-aws-mio1.jeanbruenn.info:9000
Uptime: 2 minutes
Version: 2023-10-07T15:07:38Z
Network: 2/2 OK
Drives: 2/2 OK
Pool: 1
Errors and Solutions
The following errors were encountered during deployment, highlighting typical traps:
| Error Type | Log Entry (Excerpt) | Root Cause & Solution |
| Binding Error | ERROR Unable to start the server: listen tcp 18.XXX.XX.XXX:9000: bind: cannot assign requested address | Lightsail NAT: The instance cannot bind to its Public IP internally. Solution: Ensure the hostname resolves to 127.0.0.1 in /etc/hosts for local services. |
| Server Validation | ERROR Unable to validate passed arguments: host in server address should be this server | Solution: Temporarily use the instance’s hostname (e.g., de-aws-mio2.jeanbruenn.info) in MINIO_SERVER_URL until the primary load balancer is in place. |
| Distributed Quorum Failure | Error: Read failed. Insufficient number of drives online | Root Cause: A basic firewall failure. The MinIO nodes could not communicate on the private network ports (9000-9010), meaning the cluster could not reach its minimum quorum. |
| Certificate/TLS Load | ERROR Unable to load the TLS configuration: HTTPS specified in endpoints, but no TLS certificate is found | Solution: Certificates cannot be symlinked from Let’s Encrypt. The certificates must be copied and owned by the minio user in /etc/minio/certs. |
Sources / See Also
- MinIO Documentation. MinIO Distributed Deployment Guide.
https://min.io/docs/minio/linux/deployment/distributed-deployment/ - MinIO Documentation. Install MinIO on Amazon EKS in 15 Minutes or Less.
https://blog.min.io/install-minio-amazon-eks/ - MinIO Documentation. How to Install MinIO in Distributed Mode on AWS EC2.
https://blog.minio/install-minio-distributed-mode-aws-ec2/ - MinIO Documentation. MinIO Troubleshooting Guide.
https://min.io/docs/minio/linux/operations/install-deploy-manage/troubleshoot.html - AWS Lightsail Documentation. Networking and Firewall Rules.
https://docs.aws.amazon.com/lightsail/latest/userguide/configuring-firewall-for-lightsail.html - Linux Security. Understanding capabilities and setcap.
https://man7.org/linux/man-pages/man7/capabilities.7.html