[Box Backup-dev] "setlimit 0 (...) back with vengeance (Was: Proposal (patch): bbstored HousekeepingStyle mod)

[Chris Wilson]

Hi Gary,

On Tue, 23 Jan 2007, G. wrote:

>> I think we should remove this check in bbackupd. I can't see the point 
>> of it, since we could easily start off below the threshold, and then 
>> have
>
> I think the problem is that the number of up-to-date blocks already on 
> the server, plus the number of new blocks to be uploaded from a client, 
> might actually be greater than the number of free blocks available on a 
> server, due to auxiliary blocks representing old file versions and 
> deleted files.
>
> This situation will happen when housekeeping has not run for a while for 
> whatever reason, or a soft-limit has been set too high in the first 
> place (and housekeeping cannot ever help, since it does not clean up 
> enough blocks for future deltas).

Yes, I think all that is true.

> There is currently no way for a client to know how many (new) blocks it 
> is going to upload at the beginning of a cycle (in order to do an exact 
> check), nor can housekeeping figure out that a soft-limit is too high 
> for future deltas, so I am guessing that this check tries to approximate 
> such a situation.

But why? What's the relationship between a point somewhere between the 
soft and hard limits, and the amount of data that you might upload during 
this backup run?

> My vote would be to let a server hit a hard-limit (unless it means some 
> kind of store corruption or bbstored panic) and do away with the 
> approximation. A soft-limit should have no bearing on new block upload 
> capability.

Yes, that's what Martin and I are proposing. There should be no panic, 
Box Backup was designed to handle this situation. The client will get an 
exception and sleep for about 100 seconds; housekeeping will happen sooner 
or later to delete some old files and then the client should be able to 
upload some more. I have a feeling that 100 seconds might not be enough 
here.

> a.) a client should calculate the number of new blocks to be uploaded, 
> and execute - beforehand - some kind of a synchronous request for a 
> server to housekeep an equal number of blocks immediately to make room 
> for the new stuff that's coming, or:

I think that's impossible; data might change between the time that the 
estimate is made, and the time that the client actually gets around to 
uploading files.

> b.) (better yet) a client should happily ignore the whole situation, 
> while a server keeps housekeeping dynamically as the client pushes up 
> more and more new blocks.

Yes, I think that would be an improvement, but we are planning to remove 
housekeeping entirely in Box Backup 0.20, and have the client manage the 
store itself. Then you will be able to set any data retention policy that 
you want, on the client.

Any changes that we make to housekeeping now are a band-aid to keep us 
going until 0.20, and should probably be kept as simple as possible to 
avoid introducing new bugs, especially since Ben doesn't have much time 
for reviewing patches these days.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |