Category: Cloud Infrastructure

I experienced trouble with Nextcloud’s built-in S3 connector, as it would corrupt photos during auto-upload from the Android client. Since dedicated S3FS or Goofys were also not ideal, I decided on a reliable alternative: using rclone to manage the mounts. This strategy allows me to decouple the unreliable Nextcloud S3 implementation from the underlying object storage.

Note: While a native S3 implementation is preferable, this method abstracts the object storage as a block-like device. This compromise is necessary for stability.

Part I: Rclone Setup and S3 Configuration

1. Installation and Credentials

Installing rclone is straightforward. The challenge lies in secure credential management and defining the S3 endpoint.

apt-get install rclone
mkdir /etc/rclone

# Example rclone.conf for MinIO credentials
[minio]
type = s3
region = somewhere-over-the-rainbow
endpoint = http://127.0.0.1:9000
provider = Minio 
env_auth = false
access_key_id = ...
secret_access_key = ...
acl = bucket-owner-full-control

2. The Mounting Challenge

Mounting the entire Nextcloud data directory (/var/www/nextcloud/data) to S3 is suboptimal, as folders like appdata_... and updater often contain volatile data that should not reside on object storage. The cleanest solution is to mount the S3 bucket specifically to the user’s files directory (/var/www/nextcloud/data/user/files), which excludes the trashbin and appdata folders.

Part II: Multi-Instance Management with systemd

To manage mounts for multiple users or multiple S3 buckets without redundant service files, I implemented a systemd multi-instance unit template. This demonstrates efficient Configuration Management and scalable deployment.

Unit Definition (rclone@.service)

The unit template uses the %i placeholder, which corresponds to the username (e.g., rclone@jean).

[Unit]
Description=rclone - s3 mount for nextcloud %i data
Documentation=https://rclone.org/ man:rclone(1)
AssertPathExists=/etc/rclone/rclone-%i.conf
RequiresMountsFor=/var/www/nextcloud/data
Before=nginx.service
After=network-online.target
Wants=network-online.target

Note: The RequiresMountsFor and Before=nginx.service directives are crucial for guaranteeing that the mount point is ready before the webserver attempts to serve files.

Service Execution

The service uses the %i placeholder to reference the correct configuration and the user-specific mount path (/var/www/nextcloud/data/%i/files). The environment variable RCLONE_CONFIG ensures the service loads the correct credential file.

[Service]
Type=notify
Environment=RCLONE_CONFIG=/etc/rclone/rclone-%i.conf
ExecStart=/usr/bin/rclone \
    mount minio:nextcloud-%i /var/www/nextcloud/data/%i/files \
    --allow-other \
    --vfs-cache-mode writes \
    --log-level INFO \
    --log-file /var/log/rclone/rclone-%i.log \
    --umask 002
ExecStop=/bin/fusermount -uz /var/www/nextcloud/data/%i/files
Restart=on-failure
User=www-data
Group=www-data

Part III: Cache Tuning and Verification

VFS Cache and Performance Tuning

I am still experimenting with the optimal mount options, but these settings ensure stability and good performance. The VFS cache settings are critical for preventing corruption and managing the Object Storage paradigm.

   --allow-other \
   --vfs-cache-mode writes \  # Recommended for Nextcloud to handle writes reliably
   --umask 002 \
   --use-server-modtime \
   --transfers 16 \
   --vfs-fast-fingerprint \
   --vfs-cache-max-age 168h \
   --vfs-cache-max-size 15G

Verification and Observability

Verification involves ensuring the mount is persistent and checking the rclone status log.

systemctl enable rclone@jean
systemctl start rclone@jean

# Check status after a few hours
root@nc:~# systemctl status rclone@jean
...
     Active: active (running) since Sun 2024-04-07 04:30:55 CEST; 13h ago
       Docs: https://rclone.org/
             man:rclone(1)
   Main PID: 4105 (rclone)
     Status: "[17:53] vfs cache: objects 21 (was 21) in use 0, to upload 0, uploading 0, total size 895.986Mi (was 895.986Mi)"
...

The log confirms that the mount is active and the VFS cache is managing objects correctly.

Sources / See Also

1. Rclone Documentation. Mount Options and Usage (VFS Cache Modes). https://rclone.org/commands/rclone_mount/
2. Nextcloud Documentation. External Storage Configuration (S3 as Primary Storage). https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage/s3.html
3. FUSE Project Documentation. Understanding FUSE Filesystems and Permissions (allow-other, umask). https://github.com/libfuse/libfuse
4. systemd Documentation. Using Templates and Instances (rclone@.service). https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Templates
5. MinIO Documentation. Reference Guide for S3 Configuration and Endpoints. https://min.io/docs/minio/linux/reference/minio-cli/minio-mc-admin-config.html