[Box Backup] BadBackupStoreFile

[Chris Wilson]

Hi Johann,

> We have two >2GB files in the backup store. See the bottom of a sorted
> listing:
> [...]
> 748651521 ./cc/02/o1f.rfw
> 748679825 ./72/03/o8e.rfw
> 748705025 ./3c/03/oa7.rfw
> 759350104 ./20/04/o10.rfw
> 759356296 ./23/06/o12.rfw
> 759552073 ./9e/06/o8f.rfw
> 759633897 ./1a/07/obc.rfw
> 1529736909 ./ba/o55.rfw
> 1539937214 ./69/01/o05.rfw
> 2744679666 ./16/01/o5f.rfw
> 5133609317 ./f2/o44.rfw

Do you have any errors restoring or comparing the other large file?

>>> Is it sure that the backup continues after the problem with all other
>>> files? Or does the backup stop after the first error?
...
> I think that for all users of a backup tool it is vital that the backup
> runs as reliable and complete as possible. Therefore it should continue
> even if some "small" errors occur and only notify the user of them.

Of course I agree, but an exception is an exception. It's not supposed to 
happen, usually no recovery strategy is planned, and it's not even sure 
that it's safe to continue execution or to continue to use the current 
connection to the server. In this case it indicates that an internal bug 
was found.

> It should also be stated clearly (either in documentation or better in 
> syslog) whether an error is "small" enough and the backup continues or 
> if the backup run was stopped. With "clearly" I suggest that it is 
> explicitly written "Backup continues" or "Backup stopped" instead of the 
> above exception message where one needs to know the meaning and 
> implications.

I agree partly, but I think that we shouldn't have to write "backup 
continues" after every error message. It should be safe to say that if you 
see a message saying that it stopped because of an exception, then it did, 
otherwise it didn't. Perhaps we should document that better.

> In our backup store we have files with hex numbers in their names, e.g.
> "./3c/05/o95.rfw". How can I find out to which file on the backup client
> that belongs? Or (the other way round): how can I find out which backup
> store files belong to a particular file on the client?

The path name is converted to ID by taking the two hex digits from each 
component, reversing the order (most significant byte is the last one, 
before ".rfw") and padding with zeroes on the left. So, for example, 
./cc/02/o1f.rfw is 001f02cc. (I think that's right anyway).

You can compare those IDs to the ones given in the remote directory 
listings in bbackupquery, but unfortunately there isn't a global reverse 
mapping so you need to manually hunt through directories to find them, 
sorry.

> Another question: In our backup store we have lots of files with
> identical size but different md5sums, e.g.
> 532869699 ./3d/05/o6a.rfw
> 532869699 ./42/04/of3.rfw
> 532869699 ./46/06/o24.rfw
> 532869699 ./8d/04/o89.rfw
> 532869699 ./94/03/o45.rfw
> 532869699 ./96/o81.rfw
> 532869699 ./a6/07/o00.rfw
> 532869699 ./c2/06/o38.rfw
> 532869699 ./ca/05/oa9.rfw
> 532869699 ./d8/04/o19.rfw
> There are several such runs. I assume that these all belong to the same
> original file on the client. So, my question is, if a large file changes
> slightly (e.g. gets a bit longer, or a few (kilo)bytes are modified),
> will the backup store then hold only the difference (like SVN) or the
> whole new plus the whole old versions of the file?

Normally it will only hold the difference. It would be interesting to know 
if these are really multiple copies of old files, and if so why that 
happened.

In order to calculate the difference, bbackupd will perform a diff of the 
file against a downloading stream. Since this could take a long time, 
there is a built-in timeout (MaximumDiffingTime) and once that time is 
exceeded, bbackupd will upload an entire new copy. If these are copies 
of the same file, you might like to increase that timeout.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |