Rclone

Rclone is a command-line program for syncing and copying files to and from S3-compatible object storage. It’s the right tool for ad-hoc copies, scripted backups, and one-shot data migrations.

Exaba is validated against Rclone: full sync, copy, and mount workflows are part of routine integration testing. Rclone’s s3 backend includes a built-in Exaba provider, so there is no fork or custom build to install. See Rclone’s Exaba documentation for the upstream reference.

Configuration

Add an Exaba remote to your Rclone config (~/.config/rclone/rclone.conf):

[exaba]
type = s3
provider = Exaba
endpoint = https://your-exaba-endpoint.example.com
access_key_id = your-access-key
secret_access_key = your-secret-key
acl = private

The values that matter:

  • provider = Exaba selects Rclone’s built-in Exaba provider, which applies the right S3 behaviour for an Exaba endpoint.
  • endpoint: your Exaba cluster’s S3 endpoint URL.
  • access_key_id / secret_access_key: issued by Exaba IAM. Scope the underlying role to the buckets this remote will touch.
  • region: leave blank. Exaba does not require one.

Test the remote:

rclone lsd exaba:                  # list buckets
rclone ls exaba:my-bucket          # list objects

Common workflows

Sync a local directory to Exaba

rclone sync /local/path exaba:my-bucket/prefix \
    --transfers 32 \
    --checksum \
    --progress
  • --transfers 32 parallelises the upload; tune to saturate your uplink without overwhelming the source filesystem.
  • --checksum makes the sync idempotent: re-running picks up only changed files based on checksum, not modification time.

Copy between two S3-compatible targets

Rclone can copy directly between two S3 remotes without staging through local disk. This is the right pattern for migrations from AWS S3 to Exaba. See Migration from AWS S3: Pattern 1.

rclone copy s3-aws:source-bucket exaba:destination-bucket \
    --transfers 32 \
    --checksum

Mount a bucket as a filesystem

rclone mount exaba:my-bucket /mnt/exaba \
    --vfs-cache-mode full \
    --dir-cache-time 1m \
    --daemon

--vfs-cache-mode full is the right mode for general read/write workloads. For read-only or append-mostly workloads, writes suffices and uses less local disk.

Object Lock with Rclone

Rclone’s s3 backend supports Object Lock retention and Legal Hold through standard S3 headers, which Exaba implements at parity. The Compatibility Matrix: S3 protocol surface lists the validated Object Lock features.

For backup-target workloads driven by Rclone, the recommended pattern is to create the bucket with Object Lock enabled at creation time and rely on bucket-level default retention rather than setting retention per-object.

Performance notes

  • Multipart threshold: Rclone’s default chunk size is reasonable for most workloads. For very large objects (>100 GB), increase --s3-chunk-size 64M to reduce metadata overhead.
  • Concurrency: --transfers N controls parallel object transfers; --s3-upload-concurrency N controls parallel parts within a single multipart upload. Tune both for your specific source/destination/network.
  • Checksums: Rclone validates Exaba’s content-MD5 checksums on download. For end-to-end integrity, do not disable checksums (--no-checksum) or TLS verification (--no-check-certificate).

See also