[Box Backup] BadBackupStoreFile

[Johann Glaser]

Hi Chris!

> > The Iomega StorCenter is a NAS, i.e. one large hard disk (to be precise=
:=20
> > RAID5) offered via NFS. The limiting factors are both, CPU and IO. When=
=20
> > looking at the processes with "top" bbackupd sometimes utilizes nearly=20
> > 100% CPU, and sometimes only very few. bbstored also sometimes utilizes=
=20
> > approx. 20% CPU.
>=20
> Looks like it's bbackupd's fault then. Certainly the symptoms you're=20
> seeing of repeatedly reading the same file might well be related to that,=
=20
> and the high CPU usage does imply that it's calculating checksums or doin=
g=20
> some other CPU-intensive work.
>=20
> Since the client appears to be the limiting factor, what is its CPU and=20
> memory size?

It has an Intel Celeron with 2.4GHz and 1GB RAM. I don't think its only
the client, because the server daemon also runs on the same machine. But
the long delays stem from backing up large files again and again due to
the short diffing time.

> I think that "total file size uploaded" should actually read "total file=20
> size covered by backup". The actual amount uploaded, before compression,=20
> is the difference between this and "bytes already on server". This seems=20
> to be 2-7 GB for you each day.

I see. I've got a suggestion to clairify this in the logs: Please change
the output to some statement which states the following items:
 - total file count (on the client)
 - total file size (on the client)
 - number of files already on server
 - number of new files
 - number of deleted files
 - number of files which have changed
    - split up into algorithms how the changes have been detected (date,
      size, checksum, ?)
 - total size of changes (only the differences) and new files
 - total file size uploaded (is the same?)
 - bytes already on server
 - encoded size

And please add human readable values in parenthesis, e.g. "4567890123
(4.6GiB)".

> > I suspect the low diff time, which I've now increased to 20.000 seconds=
.=20
> > :-)
>=20
> That may be a bit too long! I wonder how long the diffs take in practice?=
=20
> If we're constantly re-reading files then we're certainly wasting a lot o=
f=20
> time in diffing.

Why should it be too long?

The re-reading of files is indeed a strange thing. Does this make any sens?

> That would definitely hit Box Backup badly. It's designed for use on hard=
=20
> disks and conventional filesystem, where many small files are not a=20
> problem at all, and Box makes full use of this.
>=20
> If you're backing up to a device that doesn't meet those expectations,=20
> then I'm afraid you're going to see awful performance with Box. I really=20
> don't know if it's worth changing that for this use case alone, given tha=
t=20
> a change to the store format will invalidate all existing stored data for=
=20
> all our users.
>=20
> Have you tried raising the issue with Iomega? If the device has a hard=20
> disk inside it, this sounds like unreasonable behaviour, i.e. probably a=20
> bug.

The bad performance is for sure a problem with Iomega. We have a similar
device from Linksys here, which is also extremely slow (despite it has a
way better CPU and a lot more and RAM).

I could imagine that there are more people who consider to user
BoxBackup together with an external NAS. I suggest that you add a
warning to the BoxBackup homepage, that with certain models of "cheap"
NAS boxes (~ =E2=82=AC 1000,--) performance issues were observed. I would
propose to direct people to external USB harddisks if (and only if) they
want external storage devices.

Regarding modifying BoxBackup to tolerate slow storage: Changing its
storage format should be avoided. :-)

> > BTW: The Sesam backup needs 7 minutes for "daily" backups (differential=
)
> > but also has to check a similarely large amount of data.
>=20
> Because Sesam is designed to work with tapes, it probably doesn't touch=20
> the tape unless it has to, i.e. it stores all the information locally (on=
=20
> disk) to decide what it needs to back up.

I'm sure it has a local database for the files, but also writes this to
the tape, so only the tapes themselves are required to restore a
system.=20

Talking about this last week to a collegue, I found that BoxBackup
should also have such a feature. :-)

> It's difficult to tell from their website whether it does incremental=20
> diffs of changes within files, or whole files, and how much metadata it=20
> stores. But if it keeps all metadata on a local disk then it could=20
> certainly be faster than Box.

The documentation (in German) is quite elaborate but comes short when
specifying how changes are observed and how changed files are stored. As
far as I read, they always store the whole file when it has changed. So
no diffs within files. And considering the high speed checking a lot of
files, I think it finds changes only by file system attributes (i.e.
modification date, file size and the archive-bit on Windows machines).

> We might wish to look at that as a possible performance optimisation in=20
> future (including maybe keeping a copy of the file block checksums=20
> locally). But for the use case that Box was designed for, i.e. backing up=
=20
> over a slow internet connection (rather than a fast LAN) this should not=20
> be the limiting factor anyway.

Yea, the best solution would be to install BoxBackup locally on the NAS
device. I did this for a server in my dormitory where we built a "NAS"
ourselves and it finishes within a few seconds, even for several GB on
the remote clients.

Bye
  Hansi