[Box Backup] Purge based on age?

[Per Thomsen]

On 6/17/06 12:14 PM, W. Chris Shank wrote:
> Here is an instance of what I'm talking about. This account has a 41G soft limit and 46G hard limit.  I got an email last night that they were out of space.
>
> There is a single log entry on the client about this:
> #Jun 16 21:57:05 localhost bbackupd[19475]: Exceeded storage limits on server -- not uploading changes to files
>
> But then at the next upload interval - this is the log entry:
> #Jun 16 22:39:59 localhost bbackupd[19475]: Finished scan of local files
> #Jun 16 22:39:59 localhost bbackupd[19475]: File statistics: total file size uploaded 169350835, bytes already on server 1783611, encoded size 99172204
>
> So, did the server clean up some space before the next interval? Did the changes from the failed interval get uploaded on the following successful upload? Do I need to increase space or will the 25G of old files work their way out naturally?
>   
I'm pretty sure this is how it's supposed to work. Folks with a deeper
understanding, please chime in. Anyway: There are two pieces to the
bbstored daemon:

   1. The piece that handles the backups and other sundry client
      communications tasks (restores, usage info, etc.)
   2. The housekeeping piece, which purges data that should be purged,
      when it is over the limit.

The housekeeping piece runs at a set interval (controlled by the
'TimeBetweenHouseKeeping' parameter in bbstored.conf), and all the
cleanup/purging happens then.

I looked at my logs for my production server, but did not find any
instances of what you're seeing. I think a scenario like this is possible:

   1. An account is close to the soft limit, and is 'eating' the excess
      of old and deleted files, when it uploads new data.
   2. It has a large update set (in your case in excess of 5G), and the
      bbstored server puts a stop to the uploading, in the middle of the
      upload. A subset of the backup is not done at this time.
   3. Between 21:57:05 and 22:39:59, the housekeeping daemon runs, and
      purges old/deleted files down to just below the soft limit. That
      is enough to fit the remainder of the backup.


I'm still running 0.09, while I'm building my new 0.10 server, so it is
possible that that has something to do with it, but I don't think so.

I'm running some tests of this to confirm my statements, but they are
taking a while to complete. I will let you know how it goes.

Thanks,
Per
>
> Here is the server output:
> # bbstoreaccounts info ca114
>                   Account ID: 000ca114
>               Last object ID: 192469
>                  Blocks used: 41886028 (40904.32Mb)
>     Blocks used by old files: 25909364 (25302.11Mb)
> Blocks used by deleted files: 2154215 (2103.73Mb)
>   Blocks used by directories: 28906 (28.23Mb)
>             Block soft limit: 41943040 (40960.00Mb)
>             Block hard limit: 46137344 (45056.00Mb)
>          Client store marker: 1150215713000000
>
> Here is the client side view:
>           Used    40904.3Mb  90% ************************************
>      Old files    25302.1Mb  56% **********************
>  Deleted files     2103.7Mb   4% *
>    Directories       28.2Mb   0%
>     Soft limit    40960.0Mb  90% ************************************
>     Hard limit    45056.0Mb 100% ****************************************
>
>
>
>
>
> ----- Original Message ----- 
> From: Per Thomsen <pthomsen@reedtz.com> 
> To: boxbackup@fluffy.co.uk 
> Sent: Friday, June 16, 2006 4:50:07 PM GMT-0500 
> Subject: Re: [Box Backup] Purge based on age? 
>
> On 6/16/06 1:44 PM, W. Chris Shank wrote: 
>   
>> Is it possible to purge old archive data from a box archive that is older than a specified date? 
>>     
> Not presently. 
>   
>> I guess specifically, what I want to do is force out the oldest versions of files to make extra room in the archive. I've found that once the backup limit is reached, the oldest data is not being removed. Or at least I think so. Is there any way to ensure that old stuff is purged? 
>>
>>     
> What you are looking for is how Box is designed. It will remove the 
> oldest versions (or deleted files) first, when housekeeping. If you have 
> examples of the oldest files (versions or deleted ones) not being purged 
> first, please send the details to the list, and we'll take a look at it. 
>
> Thanks, 
> Per 
>
>   


-- 
Per Reedtz Thomsen | Reedtz Consulting, LLC | F: 209 883 4119
V: 209 883 4102    |   pthomsen@reedtz.com  | C: 209 996 9561
GPG ID: 1209784F   |  Yahoo! Chat: pthomsen | AIM: pthomsen