[Box Backup] Two backup-servers

Markus Liljergren boxbackup@boxbackup.org
Sun, 27 Jan 2008 17:57:01 +0100


This is a multi-part message in MIME format.
--------------050803030004060804010409
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Chris,

Chris Wilson wrote:
> Hi Markus,
>
> On Sun, 27 Jan 2008, Markus Liljergren wrote:
>
>   
>> I'm trying to figure out the best way to deploy BoxBackup as a service 
>> for my servers. All services provided by my servers have redundancy in
>> the way that there is always another server ready to take over one or 
>> more services in the event of one server failing to provide its assigned 
>> services. When implementing BoxBackup in this setup I would like to have 
>> all data stored on two different servers (with limited bandwidth between 
>> them, for example NFS-based solutions would not work well).
>>     
>
> It should be possible to synchronise the two servers with rsync. However 
> rsync is not _that_ bandwidth-efficient or disk-efficient. You might 
> consider running filesystems that support snapshots at both ends, and then 
> locally computing the difference between two snapshots on the master and 
> transferring just that difference to the slave. ZFS has built-in support 
> for this (using the "zfs send" command).
>
>   
The idea of using a filesystem which support snapshots seems very 
interesting, though I don't know if that is possible without making your 
own port of for example ZFS since I'm running OpenBSD 4.2. Do you know 
if this functionality could be achieved using something which comes with 
OpenBSD core or any packages?
>> It would be preferable if the clients didn't have to backup to the two 
>> servers separately.
>>     
>
> I believe that someone here (I think it was Pete Jalajas) was running a 
> similar setup and discovered that the bandwidth use between the two server 
> sites (for synchronisation) was too high, so he found it better to run two 
> completely independent backup configurations on each client.
>
>   
If I can't find a solution using snapshots I think the option of using 
two independent backup configurations on each client is better.
>> The question I have, is it possible to do it in this or a similar way 
>> without risk of corruption due to incomplete syncs on server-failure? 
>> Should I just abandon this and have the clients send backups to two 
>> different servers?
>>     
>
> I think that it is possible and that people are currently doing it.
> I don't believe you have anything to worry about from a failed sync.
>
> If you rsync from server A to server B while clients are backing up to 
> server A, you will probably get some incomplete updates, lock files and 
> inconsistent store statistics on B. You might want to run "bbstoreaccounts 
> check fix" on each account on B after the transfer is complete.
>
> If you do have any problems, please let us know. Good luck!
>
>   
Thank you for your thoughts and information. I'm sure I will find a good 
solution to this problem.

Regards,
Markus

--------------050803030004060804010409
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Chris,<br>
<br>
Chris Wilson wrote:
<blockquote cite="mid:Pine.LNX.4.64.0801271556230.26095@top.qwarx.com"
 type="cite">
  <pre wrap="">Hi Markus,

On Sun, 27 Jan 2008, Markus Liljergren wrote:

  </pre>
  <blockquote type="cite">
    <pre wrap="">I'm trying to figure out the best way to deploy BoxBackup as a service 
for my servers. All services provided by my servers have redundancy in
the way that there is always another server ready to take over one or 
more services in the event of one server failing to provide its assigned 
services. When implementing BoxBackup in this setup I would like to have 
all data stored on two different servers (with limited bandwidth between 
them, for example NFS-based solutions would not work well).
    </pre>
  </blockquote>
  <pre wrap=""><!---->
It should be possible to synchronise the two servers with rsync. However 
rsync is not _that_ bandwidth-efficient or disk-efficient. You might 
consider running filesystems that support snapshots at both ends, and then 
locally computing the difference between two snapshots on the master and 
transferring just that difference to the slave. ZFS has built-in support 
for this (using the "zfs send" command).

  </pre>
</blockquote>
The idea of using a filesystem which support snapshots seems very
interesting, though I don't know if that is possible without making
your own port of for example ZFS since I'm running OpenBSD 4.2. Do you
know if this functionality could be achieved using something which
comes with OpenBSD core or any packages?<br>
<blockquote cite="mid:Pine.LNX.4.64.0801271556230.26095@top.qwarx.com"
 type="cite">
  <pre wrap=""></pre>
  <blockquote type="cite">
    <pre wrap="">It would be preferable if the clients didn't have to backup to the two 
servers separately.
    </pre>
  </blockquote>
  <pre wrap=""><!---->
I believe that someone here (I think it was Pete Jalajas) was running a 
similar setup and discovered that the bandwidth use between the two server 
sites (for synchronisation) was too high, so he found it better to run two 
completely independent backup configurations on each client.

  </pre>
</blockquote>
If I can't find a solution using snapshots I think the option of using
two independent backup configurations on each client is better.<br>
<blockquote cite="mid:Pine.LNX.4.64.0801271556230.26095@top.qwarx.com"
 type="cite">
  <pre wrap=""></pre>
  <blockquote type="cite">
    <pre wrap="">The question I have, is it possible to do it in this or a similar way 
without risk of corruption due to incomplete syncs on server-failure? 
Should I just abandon this and have the clients send backups to two 
different servers?
    </pre>
  </blockquote>
  <pre wrap=""><!---->
I think that it is possible and that people are currently doing it.
I don't believe you have anything to worry about from a failed sync.

If you rsync from server A to server B while clients are backing up to 
server A, you will probably get some incomplete updates, lock files and 
inconsistent store statistics on B. You might want to run "bbstoreaccounts 
check fix" on each account on B after the transfer is complete.

If you do have any problems, please let us know. Good luck!

  </pre>
</blockquote>
Thank you for your thoughts and information. I'm sure I will find a
good solution to this problem.<br>
<br>
Regards,<br>
Markus<br>
</body>
</html>

--------------050803030004060804010409--