MinIO is a fantastic Object Storage solution, and I intended to use my distributed MinIO system as the primary external storage for Nextcloud. This distributed setup, which uses Sidekick as a load balancer for seamless node access, proved functional but revealed a critical stability flaw, particularly with mobile uploads.
The Problem: Data Corruption During Mobile Auto-Uploads
After days of intense debugging—trying everything from disabling chunking and increasing timeouts to reviewing hundreds of GitHub issues—I isolated a critical flaw: Uploading large photos using the Nextcloud Android App consistently caused data corruption when Nextcloud was configured to use it’s built-in S3 External Storage connector.
My systematic troubleshooting confirmed the problem was not load balancing:
- I tested with Sidekick, NGINX, and direct access to a single upstream.
- The issue persisted across all load balancing configurations.
- Crucially, the problem was not reproducible when the storage was the local filesystem, or when external tools (
s3fs/goofys) were used.
This pointed directly to an incompatibility—likely an implementation bug involving Multipart Uploads or ETag handling—within Nextcloud’s S3 connector code.
The Workaround: Filesystem Abstraction
The only stable solution was to abandon the problematic S3 connector layer and mount the object storage as a local filesystem using tools like s3fs or goofys.
Insight: Using s3fs/goofys is not ideal; it forces object storage to act like a traditional filesystem, which incurs performance penalties. However, the stability gained by using the abstraction layer outweighed the performance cost in my setup.
Critical Configuration Notes
- Synchronization Pitfall: It is critically important to set the option “check for changes” to “never” in Nextcloud’s configuration when using S3. Otherwise, the background synchronization process will continuously run, checking the massive object storage backend.
- Stability Tests: I encountered conflicts when enabling Object Locking in MinIO in older Nextcloud versions. Thoroughly test features like Object Locking and Versioning before enabling them in a production environment.
See Also
- Nextcloud Documentation. Configuration of external storage (S3).
https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage/s3.html - MinIO Documentation. MinIO Distributed Deployment Guide.
https://min.io/docs/minio/linux/deployment/distributed-deployment/ - AWS S3 Documentation. Multipart Upload Overview.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-overview.html - GitHub Repository s3fs-fuse. Source repository for the workaround.
https://github.com/s3fs-fuse/s3fs-fuse