5 Aug 2015

Robocopy - backup and file synchronisation PART 2

In my first article about Robocopy (click here) I gave an example of how to use the Robocopy command and how to make a simple command file (batch file) to backup (synchronise) a folder.

In this article we will look at how we can automate the use of Robocopy further. This time we'll use AutoIt to control the parameters used and how our backup runs. What's the advantage of using AutoIt over a command file? You can do lots of clever things with AutoIt that would be painful to do with a mere command file.

AutoIt is a free scripting language for Windows. It is easy to learn and creates standalone EXE files. For more on AutoIt click here.


Example Scenario
I want to do the following:
  • Backup files to a USB external hard drive (storage that can be detached from the computer is best in case computer dies your backup won't die with it).
  • A simple solution where I can double click a file and it will backup, no prompts.
  • I don't want to overwrite my backup each time I run it. Imagine if I make a mistake, run the backup, my mistake would be backed up. I need a solution for this.
  • A log or report of what's been backed up.
  • It should be efficient and fast, only changed files backed up.
Of course your needs maybe different but I hope that at least this article will give you a few ideas as to how you can use Robocopy and AutoIt together to do some amazing things! 


My AutoIt and Robocopy Example Solution
I wrote a script call RoboBackup.au3. I wrote it using AutoIt in the custom SciTE editor available from AutoIt's website.

In the above example scenario I state that I'd like to be able to run the backup multiple times and not overwrite each time. To do this I decided to make my script manage three backups (by default). What this means is that I'll have three folders; backup1, backup2, backup3. Each time the backup script is run it will copy files to each folder consecutively (one by one). Because it's using Robocopy not all the files will be copied each time, only those changed since. However, it won't be since the last backup, but since three backups ago. That's a compromise on my part otherwise the script might've become more complex, I want to keep it as simple as possible. Anyway, the general concept is that you can have a maximum of three backups at any one time. This should cover most situations. 


In the above screen shot you'll see the code as I see it when I writing it. On the left hand side you can see the line numbers. I'll explain line by line what the script does but first a few basics, here's how the colour coding works:

Green - comments, text to explain what the script is doing. Therefore lines 1 to 11 is just for information. 
Black - variables, they also start with a dollar sign $
Blue - functions like IF.
Orange - operators like <>=
Red - text
Yellow - internal 'macros', these start with @ and are like system variables.

--- Line 15
So that the script 'knows' which backup folder (1, 2 or 3) to use there has to be an INI configuration file to hold these settings. Line 15 creates a variable $ini with the name of the file: RoboBackup.ini. The @ScriptDir means the current folder.

--- Line 16
$number is the number of backups folders you want to use. By default there are three. This can be changed though, if you put a different number in the ini file, that number will be read and used instead of 3. 

--- Line 17
$count is used to keep count of which folder was backed up last.

--- Line 19 
Folder to backup - this script will backup one folder but all sub-folders are included. You can specify the folder name in the ini file or it will use the My Documents folder.

--- Line 20
In case the folder to backup has spaces in it, double quotes are added. This is important because the $source variable will be used on the Robocopy command line where any spaces without double quotes " " would cause trouble.

--- Line 21 
The destination is Backup1, Backup2, etc. The number on the end comes from the ini, the $count variable. This means you'll maintain multiple backups.

--- Line 22
The log file is named as RoboBackupLOGx.txt where "x" is the number ($count) of the backup. If you are using the default settings there will only be three log files created. Each time the backup runs the old log file with the same name will be overwritten. I've done it this way so there are not lots of log files created. However, you might prefer to have more comprehensive logging than the simple three log file approach I've used. For example, you could change this line to create a log that has the date in it, meaning that a log file will be created every time the script is run and it will remain until you delete it (it will not be overwritten unless you run the script twice on the same day):

$log = @ScriptDir & "\RoboBackupLOG" & @YEAR & @MON & @MDAY & "_" & $count & ".txt"

The above would create a file name like this: RoboBackupLOG20150803_1.txt
I've put the date backwards, year-month-day, to ensure that all the logs will line up when you look at them in explorer. The last digit is the count, it'll tell you which backup folder to look in.

--- Lines 25 and 26
These lines increment the $count. When the $count is greater than the $number, it resets. This is for naming the backup folders 1, 2, 3 and then going back to 1 and starting again.

--- Lines 29 and 30
This is actually one line. At the end of line 29 you'll see an underscore "_" in AutoIt this means to start a new line. The only purpose here is for readability as it's a long command line so I wanted to split it to two lines. Just imagine it is one long line though. You can see that this is the Robocopy command itself! It's run using the AutoIt function RunWait(). It opens a command window and runs a command. In my script I've done this but minimised the command window. It's nice like that because you can check it if you want. However, if you prefer you can hide it altogether. 

--- Lines 32 to 36
Comments, an explanation of the command line parameters for Robocopy that I used. For line 36, regarding the RoboCopy /XF command, this is to stop RoboCopy coping the desktop.ini file. The Desktop.ini sometimes has a command inside it to show a different folder name. This is the case for My Documents and it can cause confusion. Not including the Desktop.ini means that the folders appear with their real names.

--- Line 39
Open Notepad and display the log file. This will happen at the end and shows you what has happened during this backup. It's the usual output from Robocopy. 

--- Line 40
Exit just closes the script. You don't really need this but usually put it just to show where it ends.



My RoboBackup Script Source Code for Copy/Paste
Here's the same script as shown above in the screen shot. But this time it's text so you can copy/paste it and use it. Feel free you change it :-)

#cs ----------------------------------------------------------------------------

RoboBackup

 AutoIt Version: 3.3.10.2
 Author:         Michael Gerrard

 Script Function:
 Create three backups of the same data using the RoboCopy mirror function

#ce ----------------------------------------------------------------------------


; Set variables
$ini   = @ScriptDir & "\RoboBackup.ini"
$number   = IniRead($ini, "Settings", "Number", "3")  ; number of backups
$count   = IniRead($ini, "Settings", "Count", "1")   ; count of backups (where we are now)

$FolderToBackup = IniRead($ini, "Settings", "FolderToBackup", @MyDocumentsDir)
$source   = '"' & $FolderToBackup & '"'
$dest   = '"' & @ScriptDir & '\backup' & $count & '"'
$log   = @ScriptDir & "\RoboBackupLOG" & $count & ".txt"

; Increment the count
If $count >= $number Then $count = 0
IniWrite($ini, "Settings", "Count", $count + 1)

; Run RoboCopy
RunWait(@ComSpec & " /c ROBOCOPY " & $source & " " & $dest & _
" /MIR /R:0 /LOG:" & $log & " /TEE" & " /XF desktop.ini ", @ScriptDir, @SW_MINIMIZE)

; /MIR = mirror
; /R:0 = retry zero times
; /LOG: = record to a log file
; /TEE = output to the screen too
; /XF desktop.ini = don't copy the desktop.ini file (avoids getting the My Documents folder name)

; Display the log file
Run("notepad " & $log)
Exit

Here's what to do:
  • Select the above text, press Ctrl-C to copy it. Switch to your text editor (SciTE) and press Ctrl-V to paste. 
  • Make sure you have AutoIt installed. 
  • Compile and run the script. 

RoboBackup.ini
The RoboBackup.ini is just a text file. You can create this yourself in Notepad or in any case you'll see it is created by the script the first time you run it. I'll look like this:

[Settings]
count=2
number=3
FolderToBackup=C:\YourData

The count is the variable the script reads and it will change every time you run the script. In my example above it says "2" so the latest backup is number 2. Next it'll be 3 and then 1, etc. This line is generated automatically by the script. 

The number line is optional. I have it set to 3, the default. But you can change this to 4 or 8, whatever. This is the number of backup copies you'd like.

The FolderToBackup line is also optional. By default it'll backup your My Documents folder. If you want to backup something else you can specify the folder here. All sub-folders below it will be included in the backup.


Where to put the files
To use the script put the compiled RoboBackup.exe and the RoboBackup.ini files in the destination drive. For example, if you have a USB flash memory drive or a USB hard disk drive, make a folder called backup on it, copy the files RoboCopy.exe (and RoboBackup.ini) into that folder. Run the script from that folder, it'll create the Backup1, Backup2, etc, folders below that point. This is a nice way of working because it means you can unplug your removable backup drive and put it somewhere safe. The next time you want to backup, plug it in and run RoboBackup.exe. 


IMPORTANT: Disk Space 
The main purpose of this script is to make a number of backup copies. This is so that you don't overwrite your backup each time, that you have a chance to recover files from a previous backup. However, to make this all work you do need to be aware of the disk space required for the backup. I've not included any disk space checks in the script (maybe in version 2!!! ;-)). Therefore look at what you want to backup and make sure your backup drive has enough space to accommodate multiple copies of that data. Also factor in that the data will grow in size as you use it. 


Conclusion
RoboCopy is a powerful command line tool for file/folder synchronisation on its own. Combined with AutoIt you can make a script that provides much more control. Here I've provided just one example and I hope it's given you ideas for how it can be adapted to solve your backup requirements. Often the simplest form of backup is best. Imagine with this, restoring is easy, just copy the files. Also for the backup itself it's easy with few requirements, just one standalone exe file (your script, RoboCopy is included as part of Windows by default) and enough disk space. This is a free solution too!


Disclaimer
The script here is just an example. I do not provide it with any guarantee. I'm not advocating this script be used for anything specific, this article is just to demonstrate what can be done. Use at your own risk! 


Reference

AutoIt 

My first RoboCopy article explaining the Robocopy command itself:


No comments: