[Box Backup] Production use

[Ben Summers]

On 27 Jan 2004, at 14:05, Eduardo Alvarenga wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Tue, 27 Jan 2004, Ben Summers wrote:
>
>>>  Our system is snapshot-based, where every machine connects to the
>>>  main backup repository at each time, syncing the files using
>>>  incremental checks, checksums, preserving symlinks and permissions
>>>  and backing up on a weeekly-directory basis (rsync's 'backup-dir'
>>>  option) + regex exclusion of sockets and non-backupeable data.
>>>
>>>  Will boxbackup suit my needs? Any Pros or Cons?
>>
>> It will be a bit of a change in how your backups work. Instead of
>> saying "take a snapshot now" the backup daemon uploads files as it
>> notice changes, as long as those changes were made a configurable time
>> ago. Future versions will allow a snapshot based approach in addition
>> to this behaviour, but for now, you have to be aware that what you
>> restore is not going to be what was on the disc at a particular moment
>> in time (if you see what I mean). But you will have all your data
>> backed up in a timely manner.
>
>  Is there any workaround to make weekly-{directory,files} backup? I
>  mean, a directory for each day-of-{week,month} where the files
>  (encrypted or not) are stored?

Not really, without loosing all the efficiency gains. As it's so  
completely against the basic design, it's not really worth thinking  
about.

>
>  This is confortable (at least for me) to retrieve any file revision
>  from the last week very easily.

This is what you can do now to view and retrieve an old version:

======================================================================== 
==

/usr/local/bin/bbackupquery

query > cd home/profiles/USERNAME

query > list -ot
00000078 f--o- 2004-01-21T20:17:48 NTUSER.DAT
00000079 f--o- 2004-01-21T20:17:48 ntuser.dat.LOG
0000007a f--o- 2004-01-21T17:55:12 ntuser.ini
0000007b f---- 2004-01-12T15:32:00 ntuser.pol
0000007c -d--- 1970-01-01T00:00:00 Templates
00000089 -d--- 1970-01-01T00:00:00 Start Menu
000000a0 -d--- 1970-01-01T00:00:00 SendTo
000000a6 -d--- 1970-01-01T00:00:00 Recent
00000151 -d--- 1970-01-01T00:00:00 PrintHood
00000152 -d--- 1970-01-01T00:00:00 NetHood
00000156 -d--- 1970-01-01T00:00:00 My Documents
0000018d -d--- 1970-01-01T00:00:00 Favorites
00000215 -d--- 1970-01-01T00:00:00 Desktop
00000219 -d--- 1970-01-01T00:00:00 Cookies
0000048b -d--- 1970-01-01T00:00:00 Application Data
000005da -d--- 1970-01-01T00:00:00 UserData
0000437e f--o- 2004-01-24T02:45:43 NTUSER.DAT
0000437f f--o- 2004-01-24T02:45:43 ntuser.dat.LOG
00004380 f--o- 2004-01-23T17:01:29 ntuser.ini
00004446 f--o- 2004-01-24T02:45:43 NTUSER.DAT
00004447 f--o- 2004-01-24T02:45:43 ntuser.dat.LOG
000045f4 f---- 2004-01-26T15:54:16 NTUSER.DAT
000045f5 f---- 2004-01-26T15:54:16 ntuser.dat.LOG
000045f6 f---- 2004-01-26T16:54:31 ntuser.ini

query > get -i 0000437e NTUSER.DAT
Object ID 0000437e fetched sucessfully.

======================================================================== 
==

Basically, list the directory with o and t options, which displays the  
time and doesn't filter out old version, find the ID number of the file  
you want, and fetch that. (using the name only fetches the latest  
version)

Fetching an entire directory isn't easy at the moment, as restore to  
snapshots is required for that.

What I'm planning to do is to be able to do a "show snapshots" command,  
which lists all the snapshots on the server, then a "select snapshot",  
and then all the queries and restores you do relate to that.

The design is there, and it's all built to do this, just a matter of  
implementing that last bit of code which actually does the magic!


>
>> Regex exclusion isn't done yet, but is not a big job. If you wanted to
>> use this seriously, then I'll put it in as a priority item.
>
>  Yes, this is serious. Many large temporary files may not be backuped.
>  So exclusions based on regex are very welcome. If you mind... please
>  keed it priority #1.

Not a problem. It's now next on the list!

>
>> Hard links are not handled (multiple copies of the same data would be
>> made). Symlinks and UNIX attributes are. Checksums are intrinsic to  
>> the
>> system.
>
>  I hardly use hard links, so it'll not be a problem for me at all.
>
>> You get RAID on your backup server very cheaply and easily if you put
>> in multiples of three discs.
>>
>> You probably need a less powerful machine to run the backup server.
>
>  Well, the machine is a development one, running mysql, apache, php,  
> cvs
>  and another tools. Any statistics about CPU/Memory usage of boxbackup?

Nothing terribly meaningful. Here's one I made earlier:

USER       PID %CPU %MEM   VSZ   RSS TT   STAT STARTED       TIME  
COMMAND
root     11081  0.0  0.9  2172  3052 ??  I     11:29AM    0:05.05  
bbackupd: idle (bbackupd)

Memory usage is in the order of a few Mb, rather than hundred of Mb.  
Depends on how many directories (not files) you have. And it will get  
less as I go forward, as there are a load of optimisations I know I can  
make.

CPU usage is fairly minimal, it's basically all I/O bound (usually  
network). There's no reason you couldn't nice it a bit if it were a  
problem.

>
>>>  For the community: What are your impressions about boxbackup? How
>>>  stable is it until now?
>>>
>> I am taking the very conservative and cautious view that it is not
>> ready for production machines, as it simply hasn't had enough testing.
>> However, I would be very glad if people would test it, but I do not
>> feel I can currently recommend using it as your only means of backup.
>>
>> However, I am working hard to get to a stage where I can recommend  
>> it's
>> use. It would help me enormously if you would consider running it in
>> parallel with your existing backup systems, and let me know how it
>> works out.
>
>  Our backup system as noticed in the last message is poor. The current
>  backup machine is getting it's resources exausted and I'm going to
>  exchange it by this new one (more disk, more speed, more reliable). So
>  I'm looking for a better tool then ours. I'm seriously thinking about
>  boxbackup, for real.
>
>  All I need to make it blow here: Regex exclusion and {daily,weekly}
>  based backup structure; as explained above.

Splendid. How long have I got? :-)

Ben