MinIO is a high performance, s3 compatible object storage. I’m using it for some years now. In this article I’ll show you how to setup your own MinIO stack using Amazon Lightsail.
Just a few words of precaution. I am using MinIO with 2 discs on 2 nodes in a distributed setup. This is NOT recommended. So while this might work for me – it might not work for you. Take a look at the minimum requirements for MinIO. Always be careful and backup your data.
Amazon Lightsail
Amazon Lightsail allows you to easily create virtual machines. This is the first time I am using it, so…
Login to your AWS account, in the search box enter Lightsail and click Lightsail or just directly navigate to https://lightsail.aws.amazon.com.
Click “create instance”. I pick Frankfurt as Region because I noticed that I have <2ms to my own servers from there. I select OS only. OS will be Linux/Unix. I picked the most current Ubuntu (22.04 LTS). Then I upload my SSH public key and pick a plan. By the way I don’t need automatic snapshots. For my setup 2 vcpus, 2 gb memory are enough. Just the storage capacity is not ideal. Enter a name for your instance (e.g. minio3 and minio4 in my case – though I renamed later to minio1 and minio2). And you’re done with the first step.
Now in the left menu
- click networking – create static ip – attach it to your instance and name it e.g. minio3-ip1.
- click storage – create disk (block storage). I need two disks with 80 GB each for now per virtual machine.
Let’s talk about costs. When I wrote this article, 2 vcpus and 2 gb memory will cost me 10 USD. 60 GB (OS) storage included. However. I need 160-200 GB storage. Adding this ontop will cost me 16-20 more USD. So we talk about 26-30 USD for one instance. For two instances I’m already at 60 USD/month which is currently circa 56 Euro/month. I’m using two Disks per virtual machine with 80 GB each.
In the end you should have something like:
Configure the linux systems
Set hosts and hostname
If you mix locations like I do you may come up with something like de-aws-mio1.yourdomain.tld and de-fc-mio1.yourdomain.tld. However. You might also just name them minio1, minio2 and so on.
Write down your prefered hostname into /etc/hostname and modify /etc/hosts accordingly. I would suggest you to put all minio Systems into /etc/hosts and make sure that your hostname is also listed in 127.0.0.1 e.g.:
/etc/hostname
de-aws-mio1.jeanbruenn.info
/etc/hosts
127.0.0.1 localhost de-aws-mio1.jeanbruenn.info
84.XXX.X.XX de-fc-mio1.jeanbruenn.info de-fc-mio1
84.XXX.XX.XX de-fc-mio2.jeanbruenn.info de-fc-mio2
3.XXX.XXX.XXX de-aws-mio1
18.XXX.XX.XXX de-aws-mio2.jeanbruenn.info de-aws-mio2I would suggest to you that this is also represented by your DNS configuration. In my stack this is:
nsupdate -l
zone jeanbruenn.info
update add de-fc-mio1.jeanbruenn.info 1200 IN A 84.XXX.X.XX
update add de-fc-mio2.jeanbruenn.info 1200 IN A 84.XXX.XX.XX
update add de-aws-mio1.jeanbruenn.info 1200 IN A 3.XXX.XXX.XXX
update add de-aws-mio2.jeanbruenn.info 1200 IN A 18.XXX.XX.XXX
sendThink a bit about the setup you’re creating. Maybe you also want to add loadbalancers per-site. In above example this could be de-aws.jeanbruenn.info and de-fc.jeanbruenn.info.
Install a few things and update the systems
Upgrade all MinIO nodes so that they’ll all have current version.
apt-get update
apt-get upgrade
apt-get dist-upgrade
rebootAlso install useful things…
apt-get install systemd-timesyncd dnsutils Format and attach the discs
While amazon lightsail shows the discs path as e.g. /dev/xvdg within the vm this does not seem to be the case. In my case the discs are /dev/nvme1n1 and /dev/nvme2n1. /dev/nvme0n1 is the operating system disc:
cfdisk /dev/nvme1n1 (create 1 GPT partition)
cfdisk /dev/nvme2n1 (create 1 GPT partition)
mkfs.xfs /dev/nvme1n1p1 -L minio1-disk1
mkfs.xfs /dev/nvme2n1p1 -L minio1-disk2Then add to /etc/fstab:
LABEL=minio1-disk1 /srv/minio/disk1 xfs defaults,noatime 0 2
LABEL=minio1-disk2 /srv/minio/disk2 xfs defaults,noatime 0 2Create the mountpoints:
mkdir -p /srv/minio/disk1
mkdir /srv/minio/disk2Install MinIO server and client
I download the server and client binary to /opt/minio/bin, symlink it from /usr/bin and put executable flags:
mkdir -p /opt/minio/bin
cd /opt/minio/bin
wget https://dl.min.io/server/minio/release/linux-amd64/minio
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x minio
chmod +x mc
cd /usr/bin/
ln -s /opt/minio/bin/minio
ln -s /opt/minio/bin/mcCreate a /etc/default/minio file with the following contents:
# environment variables can be found here:
# https://min.io/docs/minio/linux/reference/minio-server/minio-server.html
# Volume(s) to be used for MinIO server.
MINIO_VOLUMES="https://de-aws-mio{1...2}.jeanbruenn.info:9000/srv/minio/disk{1...2}"
# bind to a specific ADDRESS:PORT, ADDRESS can be an IP or hostname (default: ":9000")
MINIO_ADDRESS=":9000"
# bind to a specific ADDRESS:PORT for embedded Console UI, ADDRESS can be an IP or hostname
MINIO_CONSOLE_ADDRESS=":9002"
# Set to the URL of the load balancer for the MinIO deployment
MINIO_SERVER_URL="https://minio-lb.jeanbruenn.info:9000"
# more options
MINIO_OPTS="--config-dir /etc/minio --certs-dir /etc/minio/certs"
# access key / minio root user
MINIO_ROOT_USER="xxx"
# secret key / minio password
MINIO_ROOT_PASSWORD="xxxxxx"Add user and group:
groupadd -r minio
useradd -M -r -g minio minio
chown minio:minio /srv/minio -RThen create a systemd script:
[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/bin/minio
[Service]
WorkingDirectory=/srv
User=minio
Group=minio
ProtectProc=invisible
EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
# Let systemd restart this service always
Restart=always
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=1048576
# Specifies the maximum number of threads this process can create
TasksMax=infinity
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.targetReload systemd
systemctl daemon-reloadInstall certbot and issue a certificate for your new minio instance. I assume, you already made an entry in DNS for this purpose?
apt-get install certbot
certbot certonly --standalone -d de-aws-mio1.jeanbruenn.info --staple-ocsp -m yourmail@example.com --agree-tosAdd some permissions and you’re done
cd /opt/minio/bin/
setcap 'cap_net_bind_service=+ep' ./minio
mkdir -p /etc/minio/certs
chown minio:minio /etc/minio -R
cd /etc/minio/certs/Go to amazon lightsail. click your vm. there’s a tab networking. Click it and go to firewall. Now add:
CustomTCP – 9000 to 9010 – Any IPv4 address
do the same for the second vm.
Now you should be able to start both minio server instances using
systemctl start minioand you should see:
de-aws-mio2 systemd[1]: Starting MinIO…
de-aws-mio2 systemd[1]: Started MinIO.
de-aws-mio2 minio[1664]: Waiting for all MinIO sub-systems to be initialize…
de-aws-mio2 minio[1664]: Automatically configured API requests per node based on available memory on the system: 25
de-aws-mio2 minio[1664]: All MinIO sub-systems initialized successfully in 10.218698ms
de-aws-mio2 minio[1664]: MinIO Object Storage Server
de-aws-mio2 minio[1664]: Copyright: 2015-2023 MinIO, Inc.
de-aws-mio2 minio[1664]: License: GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html
de-aws-mio2 minio[1664]: Version: RELEASE.2023-10-07T15-07-38Z (go1.21.1 linux/amd64)
de-aws-mio2 minio[1664]: Status: 4 Online, 0 Offline.
de-aws-mio2 minio[1664]: S3-API: https://minio-lb.jeanbruenn.info:9000
de-aws-mio2 minio[1664]: Console: https://172.26.12.161:9002 https://127.0.0.1:9002
de-aws-mio2 minio[1664]: Documentation: https://min.io/docs/minio/linux/index.html
Mind that if you use SERVER_URL like I do in my configuration (/etc/defaults/minio) you will not be able to login to the MinIO GUI if that loadbalancer is not in-place. Temporary put the hostname into MINIO_SERVER_URL instead e.g.:
MINIO_SERVER_URL="https://de-aws-mio2.jeanbruenn.info:9000"Then you can access the GUI by the URL above with the port you used in CONSOLE_ADDRESS. Usually port 9002 (not 9000, yes).
If you use the naming scheme I wrote above, the server location would be de-aws and de-fc (de-aws for the MinIO Cluster at Amazon Lightsail and de-fc at my other ISP). So in this case: de-aws.
Setup the MinIO Client: mc
This is very easy. Just write:
mc alias set de-fc-mio https://address:9000 miniouser miniopasswordWhere address is the address to your loadbalancer (or the specific minio node), miniouser is the MINIO_ROOT_USER and miniopassword the MINIO_ROOT_PASSWORD. Once you’ve done that you can do:
~# mc admin info de-aws-mio1 ● de-aws-mio1.jeanbruenn.info:9000 Uptime: 2 minutes Version: 2023-10-07T15:07:38Z Network: 2/2 OK Drives: 2/2 OK Pool: 1● de-aws-mio2.jeanbruenn.info:9000
Uptime: 19 minutes
Version: 2023-10-07T15:07:38Z
Network: 2/2 OK
Drives: 2/2 OK
Pool: 1Pools:
1st, Erasure sets: 1, Drives per erasure set: 44 drives online, 0 drives offline
Updates of MinIO are also pretty cool by the way:
root@de-aws-mio1:~# mc update You are already running the most recent version of ‘mc’.root@de-aws-mio1:~# mc admin update de-aws-mio1 You are about to upgrade MinIO Server, please confirm [y/N]: y Server de-aws-mio1 already running the most recent version 2023-10-07T15:07:38Z of MinIO
See also
- Documentation, MinIO Object Storage for Linux. https://min.io/docs/minio/linux/index.html. Don’t miss the official documentation.
- Install MinIO on Amazon EKS in 15 Minutes or Less. https://blog.min.io/install-minio-amazon-eks/. While I used Amazon Lightsail they used Amazon Elastic Kubernetes Service.
- How to Install MinIO in Distributed Mode on AWS EC2. https://blog.min.io/install-minio-distributed-mode-aws-ec2/. Another method is to install MinIO in Amazon Elastic Compute Cloud.
Possible problems
ERROR Unable to validate passed arguments: host in server address should be this server
minio[3329]: ERROR Unable to validate passed arguments: host in server address should be this server
minio[3329]: > Please check --address parameter
minio[3329]: HINT:
minio[3329]: --address binds to a specific ADDRESS:PORT, ADDRESS can be an IPv4/IPv6 address or hostna>
minio[3329]: Examples: --address ':443'
minio[3329]: --address '172.16.34.31:9000'
minio[3329]: --address '[fe80::da00:a6c8:e3ae:ddd7]:9000'
systemd[1]: minio.service: Main process exited, code=exited, status=1/FAILUREThe amazon lightsail virtual machines come with a private IP and seem to be in a NAT’ed environment. So basically the static public IP is not added within the VM. The fix was to add this hostname to /etc/hosts for 127.0.0.1.
ERROR Unable to start the server: listen tcp 18.XXX.XX.XXX:9000: bind: cannot assign requested …
minio[3396]: ERROR Unable to start the server: listen tcp 18.XXX.XX.XXX:9000: bind: cannot assign requested ad>
systemd[1]: minio.service: Main process exited, code=exited, status=1/FAILURESame explanation as before. There’s a private IP used within lightsail so it can’t bind to the public IP which you likely used in /etc/hosts. Don’t list the hostname with the public IP in /etc/hosts.
ERROR Unable to start the server: HTTPS specified in endpoints, but no TLS certificate is found
minio[3460]: ERROR Unable to start the server: HTTPS specified in endpoints, but no TLS certificate is found o>
minio[3460]: > Please add TLS certificate or use HTTP endpoints only
minio[3460]: HINT:
minio[3460]: Refer to https://min.io/docs/minio/linux/operations/network-encryption.html for informati>
systemd[1]: minio.service: Main process exited, code=exited, status=1/FAILUREFirst I thought I was smart and I just symlink the certificates from /etc/letsencrypt/live… But no – It will not work like this. You need to put proper public.crt and private.key with proper permissions into the certs folder (in my case /etc/minio/certs)
ERROR Unable to load the TLS configuration: Unable to read the private key:
minio[3625]: ERROR Unable to load the TLS configuration: Unable to read the private key: open /etc/minio/certs>
minio[3625]: > Please check if the certificate has the proper owner and read permissions
systemd[1]: minio.service: Main process exited, code=exited, status=1/FAILUREJust like the previous error, this is about wrong permissions. If you used my guide, you can fix it using: chown minio:minio /etc/minio/certs -R
Error: Read failed. Insufficient number of drives online (*errors.errorString)
minio[5029]: Error: Read failed. Insufficient number of drives online (*errors.errorString)
minio[5029]: 8: internal/logger/logger.go:258:logger.LogIf()
minio[5029]: 7: cmd/prepare-storage.go:254:cmd.connectLoadInitFormats()
minio[5029]: 6: cmd/prepare-storage.go:312:cmd.waitForFormatErasure()
minio[5029]: 5: cmd/erasure-server-pool.go:104:cmd.newErasureServerPools()
minio[5029]: 4: cmd/server-main.go:964:cmd.newObjectLayer()
minio[5029]: 3: cmd/server-main.go:707:cmd.serverMain.func9()
minio[5029]: 2: cmd/server-main.go:426:cmd.bootstrapTrace()
minio[5029]: 1: cmd/server-main.go:705:cmd.serverMain()
minio[5029]: Waiting for a minimum of 2 drives to come online (elapsed 1m20s)One of my favorite errors. I really had trouble to find what’s going on. I believe, this error might surprise you for a lot of things. In this case…: Amazon Lightsail has a firewall which blocks everything not explicitly allowed ingoing… So, you might check that the communication between the minio instances through the minio ports works. Later I noticed the following message:
Error: Marking de-aws-mio1.jeanbruenn.info:9000 offline temporarily; caused by Post
minio[863]: Error: Marking de-aws-mio1.jeanbruenn.info:9000 offline temporarily; caused by Post "https://de-aws-mio1.jeanbruenn.info:9000/minio/storage/srv/minio/disk1/v50/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 3.XXX.XXX.XXX:9000: connect: connection refused (*fmt.wrapError)This happened when SSL was not working as explained in the previous error.