Using rsync for managing backups

Even a noob would like to keep his data safe. Be it some basic documents, few music files or even a downloaded video file. The strategy might simply be to copy everything over to some external drive or upload something to the cloud so that later, if need be, important files can be copied over and work can continue as usual. The bottom line is backup matters and is important to safeguard oneself from any unlikely event that a crashed computer has left you out in the open with no trace of any important data you ever owned.

For me the easiest approach was having a separate /home partition. It housed my documents, ebooks, tech notes, study material as well as music or video files. I also had my software development projects which I would work on from time to time. That way if I ever needed to upgrade my linux distro or some installations went rogue, I would install a latest distro and maintain all the data in the /home partition. This seemed to work but once a while I did feel the need to copy important stuff over to an external drive. There was no way for me to know which data was already backed up, so the only option was to keep copying the entire folder over again. I kept at this seriously flawed strategy for quite some time before I got hold of a better way to do things.

Using a backup software like Deja Dup did not suite me, since I was not looking to archive chunks of data as a single zip file and use the latest one to recover lost files. Plus keeping multiple copies of backed up data did not appeal much to me. I decide to stay away from any UI programs for my back up needs.

Finally I settled up for rsync which is a command line tool to back up data. It had simple options to copy files from source to destination, help manage the timestamps of the files, maintain directory structure of the sources as well. It had ability to copy only newer files by sending only the incremental data to the backup directory. It did have an option to sync the source and destination folders by using the “delete” flag which will remove any files in the destination which are not present in the source. Though useful, I did not feel the need to use this option since any backed up file would always be available even if I removed it from my laptop.

After playing around with different flags and options I settled for the following for backing up my data

rsync -avh --partial --inplace --append --progress ~/./<sourceFolder> <destFolder>

Briefly the options translate to following
-a = copy files recursively and preserve timestamps
-v = verbose, so that I some insight into whats rsync is up to when running the command
-h = human readable output
–partial = keeps the partially transferred files, so next time it can continue from where it left of, useful for large files
–inplace = update the file that is already backed up rather than create a new copy
–append = updates the file by appending data to the end of file
–progress = show percentage of the file that is being copied, useful when copying large files

There is a ‘.’ in the path of the source folder being copied. The presence of this ‘.’ allows rsync to maintain the directory structure in the destination folder. This way it is possible to copy multiple sources to a root back up folder and still maintain directory structure.

After settling for this command, I decide to have a simple script in place so that it were possible to invoke it from the destination directory and copy bunch of files across. The script below can be expanded to include more source folders and have interactive options to backup only one folder at a time or all at once. The choice is entirely up to me on how I want to get it done.

#! /usr/bin/sh

currdir=`pwd`
menuOptionsCount=0

showUsage() {
	echo "Usage:"
    echo "sh backup-data [ -i | -a ]"
    echo "-i - interactive mode"
    echo "-a - run for all directories"
    exit 1
}

showMenu() {
	echo "Select option:"
	echo "1. ~/Documents"
	echo "2. ~/Videos"
	echo "3. ~/Music"
	#add more options here and assign the count to menuOptionsCount
	menuOptionsCount=3
	echo "Enter your choice"
}

validateUserChoice() {

	userChoice=$1
	re='^[0-9]+$'

	# validate number input using regular expression
	if ! [[ $userChoice =~ $re ]] ; then
	   echo "Input is not a valid number. Try again."
	   exit 1
	fi
	# check if user input is valid choice
	if [[ $userChoice -lt 1 || $userChoice -gt $menuOptionsCount ]]; then
		echo "Invalid choice "$userChoice". Try again"
		exit 1
	fi

}

executeChosenOption() {
	echo 'Backing up to '$currdir

	case $1 in
		1) 
			rsync -avh --partial --inplace --append --progress ~/./Documents $currdir
			;;
		2) 
			rsync -avh --partial --inplace --append --progress ~/./Videos $currdir
			;;
		3) 
			rsync -avh --partial --inplace --append --progress ~/./Music $currdir
			;;
		# add more code here as and when you add more folders to back up	
		*)
			echo not coded yet	
	esac 
}

if [ $# -eq 0 ]; then
    showUsage
fi

if [[ "$1" == "-i" ]]; then

	clear
	showMenu
	read userChoice
	validateUserChoice $userChoice
	executeChosenOption $userChoice	

elif [[ "$1" == "-a" ]]; then
	
	echo 'Backing up to '$currdir
	rsync -avh --partial --inplace --append --progress ~/./Documents $currdir
	rsync -avh --partial --inplace --append --progress ~/./Videos $currdir
	rsync -avh --partial --inplace --append --progress ~/./Music $currdir
	# add more code here as and when you add more folders to back up	

else
	showUsage
fi

This script is housed in the root directory of my back up folder on the external drive. When I run it, it simply copies all incremental files from the Documents, Videos, Music folders onto the external drive. I can easily add more folders to expand its utility. The first run would obviously take time, but later on just copying incremental files takes no time at all.

This would definitely work for me and would help manage backups without much hassle or additional software.