[Box Backup] keeping files 28 days patch

[Chris Wilson]

Hi Torsten,

On Fri, 10 Jul 2009, Torsten wrote:
> Am Thursday 09 July 2009 21:08:33 schrieb Chris Wilson:
>> On Thu, 9 Jul 2009, Torsten wrote:
>>> i build a very little patch that only deletes files from store that are
>>> old or deleted and more than 28 days old. So files less that 28 days old
>>> are kept.
>>>
>>> I think such a feature is important for Box Backup but i do not
>>> unterstand it completely.
>>
>> Yes, I think it's a good idea, at least in the mean time until snapshots
>> are fully implemented and the client can control how much history is kept
>> on the store.
>
> Is there any documentation about this snapshot mode and how it will work? I
> did not find anything in the wiki about that.

I don't think there is much at the moment. Ben described his proposal 
here: http://www.boxbackup.org/trac/wiki/020Redesign#Directorystructure. 
This is basically what I'm planning to implement, in stages. It will store 
each backup as a separate directory under the root of the server, which 
will allow you to:

* see what files were in each backup;
* restore a directory to exactly the state that it was at a given point in
   time in the past (modulo changes during the backup);
* manage how much history you keep on the store by deleting old versions
   (e.g. grandfather-father-son);
* behave more like tape (which allows recycling old tapes).

>>> - So the new soft limit should be calculated from actual data and
>>>   secure old and deleted files. How could this be done?
>>
>> Why do we need a new soft limit? Either housekeeping can bring the 
>> store under the soft limit or it cannot (because all old files are 
>> newer than 28 days). It makes no difference to the client, which will 
>> keep uploading files until it hits the hard limit and then stop.
>
> I think there is a problem for the user. "bbackupquery usage" does not 
> show me if i should increase backup store limit. Because i see that hard 
> limit is reached but there are many old and deleted files. So i think 
> housekeeping will delete them. But the 28 day patch can produce the 
> following situation. Old and deleted files < 28 day are not deleted, but 
> new files cannot be uploaded. What do you think?

I think that if the account is over the soft limit after housekeeping has 
run, then you may need to increase the soft limit, or else wait for 
deleted files to become old enough that they will be deleted 
automatically. I think what you need to watch for is the free space (hard 
limit - used space) to determine whether the client can still back up 
normally.

Does that make sense? Am I missing something?

>> As this patch is implemented, it's a global option at compile time, and 
>> can't be set per account. Making it configurable at runtime would be 
>> useful. Making it configurable per account would probably require a 
>> change to the account info file format, and I'm reluctant to do that as 
>> it would break forwards compatibility at least, and I think snapshots 
>> are a better way to implement this in the long run. But I'm seriously 
>> considering applying your patch.
>
> Making it configurable at runtime should be possible. Per account 
> settings is not as important i think. If this patch is applied it could 
> be disabled in default configuration, so nobody should have a problem i 
> think.

I think it will break the current tests as they are designed to reduce the 
soft limit and watch the server deleting old revisions whenever it runs 
housekeeping. This would not happen with the patch applied unless the 
time to keep revisions for the test was set very low. That makes it 
non-trivial to apply the patch, because I need to keep all tests passing, 
so I need to make the minimum age of deleted files configurable - and test 
that the patch is actually working. However that does not mean that I 
won't implement this feature.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer |
\__/_/_/_//_/___/ | We are GNU : free your mind & your software |