Sync Files Directories from Different Cloud Storage with Rclone

Linux

Rclone is a command line program written in Go language, used to sync files and directories from different cloud storage providers such as: Amazon Drive, Amazon S3, Backblaze B2, Box, Ceph, DigitalOcean Spaces, Dropbox, FTP, Google Cloud Storage, Google Drive, etc.

As you see, it supports multiple platforms, which makes it a useful tool to sync your data between servers or to a private storage.

Rclone comes with the following features

  • MD5/SHA1 hash checks at all times for ensuring file integrity integrity.
  • Timestamps are preserved on files.
  • Partial syncs supported on a whole file basis.
  • Copy mode for new or changed files.
  • One way sync to make a directory identical.
  • Check mode – hash equality check.
  • Can sync to and from network, eg two different cloud accounts.
  • (Encryption) backend.
  • (Cache) backend.
  • (Union) backend.
  • Optional FUSE mount (rclone mount).

How to Install rclone in Linux Systems

The installation of rclone can be completed in two different ways. The easier one is using their installation script, by issuing the following command.

 

What this script does is to check the OS type on which it is ran and download the archive related to that OS. Then it extracts the archive and copies rclone binary to /usr/bin/rclone and gives 755 permissions on the file.

In the end, when the installation is complete, you should see the following line:

The second way to install rclone is by issuing the following commands.

Now copy the binary file and give it executable permissions.

Install rclone manpage.

How to Configure rclone in Linux Systems

Next what you will need to do is run the rclone config to create your config file. It will be used for authentication for future usage of rclone. To run the configuration setup run the following command.

You will see the following prompt:

The options are as follows:

  • n) – Create new remote connection
  • s) – set password protection for your configuration
  • q) – exit the config

For the purpose of this tutorial lets press "n" and create new connection. You will be asked to give the new connection a name. After that you will be prompted to select the type of storage to be configured:

rclone - New Remote Connection

rclone – New Remote Connection

I have named by connection “Google” and selected “Google Drive”, which is under the number 12. The rest of the questions you can answer by simply leaving the default answer, which is an empty “”.

When asked to, you may select “autoconfig”, which will generate all the required info to connect to your Google Drive and give rclone permissions to use data from Google Drive.

The process looks something like this:

How to Use rclone in Linux Systems

Rclone has quite a long list of available options and commands to be used with. We will try to cover some of the more important ones:

List Remote Directory

rclone - List Remote Directory

rclone – List Remote Directory

Copy Data with rclone

Note that if rclone finds duplicates, those will be ignored:

rclone - Copy Data

rclone – Copy Data

Sync data with rclone

If you want to sync some data between directories, you should use rclone with sync command.

The command should look like this:

In this case the source is synced to destination, changing the destination only! This method skips unchanged files. Since the command can cause data loss, you can use it with “–dry-run” to see what exactly will be copied and deleted.

rclone Sync Data

rclone Sync Data

Move Data with rclone

To move data, you can use rclone with move command. The command should look like this:

The content from the source, will be moved (deleted) and placed on the selected destination.

Other useful rclone Commands

To create a directory on destination.

To remove a directory.

Check if files on source and destination match:

Delete files:

Each of rclone commands can be used with different flags and includes its own help menu. For example, you can do a selective delete using the delete option. Lets say you want to delete files larger than 100M, the command would look like this.

It is highly recommend to review the manual and help for each command to get the most of rclone. The full documentation of rclone is available at: https://rclone.org/

Conclusion

rclone is a powerful command line utility to help you manage data between different Cloud storage providers. While in this article we scratched just the surface of rclone capabilities, there is much more to be achieved with it especially when used in combination with cron service.

Source: https://www.tecmint.com/rclone-sync-files-from-cloud-storage/

Share: