GDR Forum Index
Podcast Podcast
Dev Dev Logs
Search Search
RSS RSS
Register Register
Log in Log in
Reply to topic GDR Forum Index -> Game Developer's Refuge -> Backup Dat Data!
View previous topic :: View next topic  
Author Message
Bean
Admin

Joined: 20 Aug 2005
Posts: 3779

PostPosted: Mon Apr 06, 2015 8:45 pm    Post subject: Backup Dat Data! Reply with quote

I'm in need of a backup solution. While I'm at it I figured we ought to have a thread for backup discussion so here we go! :)

I have a drive that is ever changing. Files come, get edited and are eventually moved to another drive for archival or deleted. I want something that can be scheduled to maintain a daily/weekly backup of this drive.

Windows 7 Backup basically does everything I need. The only problem is that it never removes files from the backup. It will add new or modified files but if I delete something from the target drive then run a backup, it doesn't see this change and also remove it from the backup. I suppose this is typically the desired behavior but it's not what I want. I'm moving massive amounts of data to and from this drive and if the backup never removes anything it's going to balloon very fast!

A dirty solution to this would be to delete the backup image before each backup. But I'd like to hear if there are any better options out there.


-Bean
_________________
Kevin Reems | Nuclear Playground | Solid Driver
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1696
Location: Your consciousness.
PostPosted: Tue Apr 07, 2015 3:25 am    Post subject: Reply with quote

I'm using "robocopy" in Windows 7 and "rsync" in ArchLinux, on each system in a script file.

run_backup.bat - Windows 7 using robocopy
Code:
set RCOPT=/MIR /COPY:DAT /XA:S
REM /MIR (mirror) /COPY:DAT (data attributes timestamp) /XA:S (exclude files with system flag)
robocopy F:\ X:\BACKUP\1_WORK\ %RCOPT%
robocopy I:\_RepositorySVN\ X:\BACKUP\2_DATA_B\_RepositorySVN\ %RCOPT%
robocopy I:\_RepositoryGIT\ X:\BACKUP\2_DATA_B\_RepositoryGIT\ %RCOPT%
robocopy H:\Install\ X:\BACKUP\2_DATA_A\Install\ %RCOPT%
robocopy H:\Archiv\ X:\BACKUP\2_DATA_A\Archiv\ %RCOPT%
robocopy H:\Docs\ X:\BACKUP\2_DATA_A\Docs\ %RCOPT%


run_backup.sh - Linux equivalent to the robocopy batch script, using rsync
Code:
#!/bin/bash
RSOPT='-vv -a -x -h -H -X --progress --delete -s'

# -vv verbosity (-c skip based on checksum(slow)) -a archive mode (-rlptgoD) -x don't cross filesystem boundaries
# -h human readable numbers -H preserve hardlinks -X preserve extended attributes
# --progress show progress --delete delete extraneous files from destination -s protect args

if [[ $EUID -ne 0 ]]; then
  echo 'backup not started: this script needs to run with sudo or as root'
  exit 1
fi;

#echo $RSOPT
rsync $RSOPT /mnt/win/F_1_WORK/ /mnt/win/X_BACKUP/BACKUP/1_WORK/
rsync $RSOPT /mnt/win/I_2_DATA_B/_RepositorySVN/ /mnt/win/X_BACKUP/BACKUP/2_DATA_B/_RepositorySVN/
rsync $RSOPT /mnt/win/I_2_DATA_B/_RepositoryGIT/ /mnt/win/X_BACKUP/BACKUP/2_DATA_B/_RepositoryGIT/
rsync $RSOPT /mnt/win/H_2_DATA_A/Install/ /mnt/win/X_BACKUP/BACKUP/2_DATA_A/Install/
rsync $RSOPT /mnt/win/H_2_DATA_A/Archiv/ /mnt/win/X_BACKUP/BACKUP/2_DATA_A/Archiv/
rsync $RSOPT /mnt/win/H_2_DATA_A/Docs/ /mnt/win/X_BACKUP/BACKUP/2_DATA_A/Docs/


I keep the scripts on an external SATA drive which I connect about once a week or whenever I just made some huge changes I want to backup immediately and then run it from there. Served me well for several years now.
_________________
0xDB
View user's profile Send private message Visit poster's website
PoV
Moderator

Joined: 21 Aug 2005
Posts: 10977
Location: Canadia
PostPosted: Tue Apr 07, 2015 5:19 am    Post subject: Reply with quote

For my Art, and old Source Code, I had an SVN server running locally on a NAS (Network attached storage, I.e. a tiny computer that is basically just hard drives).

When I need it, I have an external 3TB external HDD that I copy my things to (a few actually). Whenever I fill one up, I seem to buy a new one. I have probably 5 now.

For my newer Source Code, I now use BitBucket. Private repositories are free for 5 or less users. For Ludum Dare we will be using Github. Everything is set up, I just have to use it.


Video definitely requires a different setup. If the videos were worth something, then $50-$100 an external HDD to store backups would makes sense. I know everything I record is worthless after, so aside from an HQ export, my source files tend to go away when I need the space.
_________________
Mike Kasprzak
'eh whatever. I used to make AAA and Indie games | Ludum Dare | Blog | Tweetar
View user's profile Send private message
Sirocco
Moderator

Joined: 19 Aug 2005
Posts: 9471
Location: Not Finland
PostPosted: Wed Apr 08, 2015 5:19 pm    Post subject: Reply with quote

I'm afraid I don't do anything special. I have a few hard drives and USB thingies I occasionally back up to, and a big-ass file on my site so I've got my bases covered in the event of a natural disaster.
_________________
NoOP / Reyn Time -- The $ is screwing everyone these days. (0xDB)
View user's profile Send private message Visit poster's website
Bean
Admin

Joined: 20 Aug 2005
Posts: 3779

PostPosted: Wed Apr 08, 2015 8:44 pm    Post subject: Reply with quote

I ended up going with a very simple solution. I made a batch file (see below). Then I use Windows Task Scheduler to handle the scheduling, which it does probably better than many 3rd party apps would since it has many useful options like not running the backup unless the system is idle (which is likely if it's encoding video).

Code:

::Backup Batch File ŠKevin Reems

@echo off

date/t
time/t
echo Backup in Progress...

(
echo Backup Started,
date/t
time/t
echo ___________________________________________________________________________________

echo Deleting Previous Backup from Drive Z
z:
cd\
rd /S /Q z:\Business
rd /S /Q z:\Media


echo --------------------------------------------------

echo Copying all files from Drive F to Z:\Business
F:
cd\
xcopy *.* z:\Business\ /E /K /Y /J


echo --------------------------------------------------


echo Copying all files from Drive G to Z:\Media
G:
cd\
xcopy *.* z:\Media\ /E /K /Y /J


echo ___________________________________________________________________________________
echo Backup Ended,
date/t
time/t

) > Z:\backup.log 2>&1

echo Backup Complete.
date/t
time/t

pause

exit


Basically that deletes the old backup then copies what I want into a couple of directories on a dedicated HDD. All output is streamed to a file (which is viewable even while the backup is happening). If for some reason I happen to be awake while this is running, "Backup in Progress..." is displayed with the date/time that it started.

Currently there's a Pause at the end just so I can see that things are working (otherwise the window would close once it completed). In a day or two when I'm convinced there's no issues I'll remove that. Task Scheduler (Win7) even has an option to run is completely silently so if I wanted to there wouldn't be a window at all. I kind of like knowing it's running though so I don't wonder why my HDD is doing something.


-Bean
_________________
Kevin Reems | Nuclear Playground | Solid Driver
View user's profile Send private message Visit poster's website
Bean
Admin

Joined: 20 Aug 2005
Posts: 3779

PostPosted: Thu Apr 09, 2015 11:50 pm    Post subject: Reply with quote

Thanks Dennis for pointing me towards the Robocopy command. The mirror function looks exactly like what I'm after. I shall implement it this weekend :D

-Bean
_________________
Kevin Reems | Nuclear Playground | Solid Driver
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1696
Location: Your consciousness.
PostPosted: Fri Apr 10, 2015 12:06 am    Post subject: Reply with quote

One is glad to be of service. :)

It might also be interesting to note that both robocopy and rsync can be configured to put the backup onto a different machine across a network connection. Might be useful to keep that in mind for potential future upgrades where you might have a few machines each dedicated to specific tasks and would still like to have a central backup of each machine on a separate server.
_________________
0xDB
View user's profile Send private message Visit poster's website
Bean
Admin

Joined: 20 Aug 2005
Posts: 3779

PostPosted: Tue Apr 14, 2015 9:14 pm    Post subject: Reply with quote

Well RoboCopy is being a pain in the ass. I'm trying to get it to Mirror and even with an empty destination it skips all files and directories.
even more odd is if I have it output to the console it says it skipped everything (doesn't give a reason). If I output to a log file it says it copied everything (even though it didn't). also the log file appears garbled after about half way down.
Apparently this thing has a history of being buggy. I love the concept but I just don't think I can trust it with my data :(


-Bean
_________________
Kevin Reems | Nuclear Playground | Solid Driver
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1696
Location: Your consciousness.
PostPosted: Wed Apr 15, 2015 2:02 am    Post subject: Reply with quote

Strange, never ran into any issues with it.
I could take a look at your script, maybe it is missing something.
_________________
0xDB
View user's profile Send private message Visit poster's website
Bean
Admin

Joined: 20 Aug 2005
Posts: 3779

PostPosted: Wed Apr 15, 2015 7:59 pm    Post subject: Reply with quote

Here's the last command I had tried:

ROBOCOPY F:\ Z:\Business\ /LOG+:Z:\backup.log /E /PURGE /FFT /W:1 /xd $RECYCLE.BIN /xd "System Volume Information"

/MIR is the same as using /E /PURGE together. I've tried both with the same results. I've also tried without the other extras. Maybe I'm missing something obvious I donno.

XCOPY is working great but it does take an hour to transfer everything. Which isn't horrible since I'm asleep at the time anyway but getting this to work would be far more efficient.

-Bean
_________________
Kevin Reems | Nuclear Playground | Solid Driver
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1696
Location: Your consciousness.
PostPosted: Thu Apr 16, 2015 9:30 am    Post subject: Reply with quote

The command looks fine. Without an error message or exit code (echo %ERRORLEVEL%) all we can do is stab in the dark...random things to check:

It might be necessary for the destination folder to exist (create Z:\Business\ if it's not there before running robocopy).

It might also be necessary to specifiy the /COPY:DAT switch despite the docs saying it is set to that by default (without it, it might think it is not supposed to actually copy anything at all).
_________________
0xDB
View user's profile Send private message Visit poster's website
Bean
Admin

Joined: 20 Aug 2005
Posts: 3779

PostPosted: Thu Apr 16, 2015 8:39 pm    Post subject: Reply with quote

Mystery solved! When using a root directory robocopy will create the destination directory with Hidden and System attributes set. So it was actually working, the results were just hidden. I added an attrib command after the copy and all is well.

I discovered this when trying to manually create the directories and being told they already existed hehe

More info here: http://serverfault.com/questions/55668/why-does-robocopy-create-a-hidden-system-folder

I'll have this sucka run daily and see how it does. I've kept a copy of my old batch file that used xcopy just in case something explodes.
It's doing the initial run now which I suspect will take an hour like xcopy did but subsequent runs should be hella quick :)
Thanks for the infos!

-Bean
_________________
Kevin Reems | Nuclear Playground | Solid Driver
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1696
Location: Your consciousness.
PostPosted: Sat Apr 18, 2015 5:10 am    Post subject: Reply with quote

Wow, of all the things that could have gone wrong, never would have thought of that. :D
_________________
0xDB
View user's profile Send private message Visit poster's website
Reply to topic GDR Forum Index -> Game Developer's Refuge -> Backup Dat Data!

Use this link to get a Sign-On Bonus when you get started!

All trademarks and copyrights on this page are owned by their respective owners. All comments owned by their respective posters.
phpBB code © 2001, 2005 phpBB Group. Other message board code © Kevin Reems.