During development and some other administration scenarios, I have a folder of code, media, or configuration files on my local computer which I’m actively updating and want to make sure a copy of that data gets put on a server where I only have SSH access using private key. So what to do?
Well, in this case, I’ve found the best tool for the job is rsync. Rsync is a tool that has been around forever and therefore is pretty rock-solid as far as reliability and performance. It’s an advanced sync tool which does a compare between your local set of files and the remote set of files and only copies the changes. When you have a large set of files and you are updating individual ones randomly, this allows you to save a lot of time and network utilization by only copying what has changed and not doing a full copy of all the files each time.
Here is a basic rsync command that you can use to connect from your local system to the remote system over SSH and copy files from your local system to that remote system.
rsync -aze "ssh -i $HOME/.ssh/mykey.pem" /home/myname/data myusername@servername:/some/folder/data
To break that down, we have:
- This is the name of the rsync command. Obviously you’ll need to have rsync installed on your system first.
- These are rsync command line options. You can look them up, but, in this example, a = archive mode, z = compress while copying (to speed up network transfer), e = remote shell. The a and z are optional and can be tweaked based on what you want, but the e is required to specify the SSH options. In our case we need it to specify the private key we are using.
- “ssh -i $HOME/.ssh/mykey.pem”
- This part is actually connected to the “-e” option we talked about in the last step. After the “e” option, you need to specify the ssh command with options. In this case we are passing the SSH command the -i option to specify a private key. This is then followed by $HOME/.ssh/mykey.pem. You should replace this part with the path to your private key. In this case we are using the $HOME environment variable, but you can just use a normal path too, like /home/joe/myprivatekey.pem
- This is the source path. It is a path to a directory on your local system where you want to copy the files from. It will process all files and folders under this path unless you use other options to exclude.
- “myusername” should be replaced with the username you want to use to login with to the remote server. “servername” should be replace with the name or IP address of the server. and “/some/folder/data” should be replaced with the path of the folder on the remote server where you want to copy the files.
You can see that it takes a little bit of effort to build the rsync command to do exactly what you want, but once you have it, you can now just run the same command over and over and each time it will check to see if any files have changed in your source directory and only copy files if they have changed.
One option you’ll want to look at is the “–del” option which tells rsync it is allowed to delete files off the remote server if they don’t exist on the source side. This is good if you want to be able to delete a file locally and have rsync do the same on the remote server. You obviously need to be very careful with this option as a mistake in your command could result in deleting files off the server that you didn’t mean to delete.