[Box Backup] New encryption format & Space usage

[boxbackup@fluffy.co.uk]

On Thu, 29 Apr 2004, Ben Summers wrote:

> > One thing thats annoying me lately (and it's not your fault of
> > course), is
> > trying to find out what exactly I'm backing up when the daemon runs. I
> > tried the option in the config file that should tell you what's going
> > on,
> > but what I was really hoping for was a list of files (and their sizes)
> > that were being backed up.
>
> Extended logging will give you some hints.

How do I turn that on?

>
> >
> > The problem behind this, is that I see some pretty huge numbers for
> > amount
> > of data backed up sometimes (and even for uploaded), and I feel that my
> > machine should have been *quiescent* before the large amount of data
> > was
> > backed up.
>
> In the default configs, changes are uploaded 6 hours ahead. So it would
> have to be quiescent for over 6 hours for the upload to make no
> changes.

Yea, this was pretty frequent though, I just can't believe as much data
changed on the disk as it was seemingly uploading/patching. As I said, I
think I found the culprit in my wife's work email account which gets
spammed pretty heavily. I'm not doubting what your program is saying, but
I'm rather curious as to what is going on that's changing all that data.

>
> >
> > I also just hit the wall with the amount of data on your store, I think
> > yesterday, and I don't think that I should be backing up enough stuff
> > to
> > cause that. I'm not saying I think your program is wrong, I'm just
> > trying
> > to figure out where the fat file(s) is/are so I can decide if I really
> > want it/them backed up. (The script to notify me of the problem also
> > had
> > an error (Debian Testing), but I haven't had time to look at that yet.
>
> You are requested to test and possible change that by the config script
> -- but let me know if you have a patch for it.

I will send you a patch without question, it just might take me a day or
so.

>
> >
> > I started looking around last night, and I seemed to have some rather
> > large files in my wife's email related folders, 'caught spam' (which
> > only
> > I check) where she gets a couple of hundred spam a day, and the files
> > related to spamassassin. So I'm thinking they were the culprit, it may
> > be
> > that the file gets some big changes which spook the delta algorithm.
>
> Did you just upgrade the client to 0.05PLUS1B? It won't diff if the old
> version of the file was written before you upgraded.
>
> >
> > I'm thinking of using *find* to try and hunt down the large files in
> > the
> > directories,
>
> This is probably the best approach.

Yea, I understand that you don't want to clutter up the program, I wasn't
asking you to put it huge query tools, I just want a bit more data to help
me track stuff like this down.

I'm thinking the plugin idea might be good (as in a previous email).

>
> >  but, I guess even some stats in the syslog would be helpful,
> > IE number of files backed up, maybe even some size stats (smallest,
> > largest, average - for file size and delta size), so I could try and
> > decide if there were 1000 small files that were changed, or one huge
> > file
> > that somehow was changed.
>
> I'll put some more stats in.

Thanks :).

I watch the syslogs like a hawk, for me that's the most useful place for
stats right now.

>
> >
> > This sort of thing would be helpful for someone trying to manage their
> > data on a third party data store (as you are for me). I'd be
> > particularly
> > interested in doing say trending analysis to find out what was
> > happening,
> > and to look for ominous changes. Or even to predict when I'd hit the
> > wall,
> > and need to *purchase* more space from the data store provider.
>
> In bbackupquery, type "usage" for space usage on the server. (New in
> 0.05PLUS1B.) Or
>
>    /usr/local/bin/bbackupquery usage quit
>
> as a simple one liner.
>
> (I've just increased your allocation a bit, BTW)

Thanks, one of the first things I tried out was the new usage command in
the client.

I appreciate your upping my limit, just so you know though, my concern
wasn't that you hadn't given me enough space, but _why_ I _had_ run out of
space with a perfectly reasonable upper limit.

>
> >
> > Part of my interest is also related to bandwidth usage, I'm on a ADSL
> > connection, but some of my friends who would be backing up to me would
> > be
> > on dialup, being able to manage the bandwidth usage would be helpful,
> > knowing what sort of changes were happening would help that way (you
> > could
> > try and have multiple backup sets or know *why* a huge upload
> > happened).
> > Other possible options would be deferring large delta's until a set
> > time,
> > while letting small changes through. Although that could be risky and
> > complicated.
>
> I'm designing taking ADSL a minimum requirement for this system. That's
> not to say it couldn't be used with dial-up, it's just it's not
> designed for it.

So far it seems to work admirably with dialup just so you know...

Rick