[Box Backup] child bbstored process ramping up to 100% cpu during bbackupd sync?

[reticent]

Ben Summers wrote:

>
> On 31 Jan 2005, at 21:19, reticent wrote:
>
>> Ben Summers wrote:
>>
>>>
>>> On 28 Jan 2005, at 22:45, reticent wrote:
>>>
>>>
>>> What you are seeing is a patch being received by the server, and 
>>> applied. The 20-30kb of data is the patch, then there's a bit of 
>>> computation required to apply the patch to the existing file, which 
>>> overwrites the original file.
>>>
>>> What is the usage pattern of files on the client? Are they static? 
>>> Are they being constantly modified? Are files being added regularly?
>>>
>> The files are, for the most part, static. There are only a few 
>> hundred updates/additions per day.
>> They are also very small, ~1-20kB
>> No modifications are made to them
>
>
> What's the directory structure like?
>

there are 70 directories with 5000 subdirectories each

>>
>>
>> The quoted speed was during a backup, i'll have to take some time to 
>> do some debugging as somthing is definitly not right here.
>>
>> I'm getting the feeling that it might be related to the amount of 
>> directories that are being delt with, the backup completed (30+ hours 
>> later for 1.6 gigs..) and i'm seeing the hk process run and eat up 
>> 100% cpu when checking backup data for host mentioned in the original 
>> email.
>
>
> Yes, housekeeping can take a while for large directory structures. I 
> will be addressing this soon.
>
>>
>> Looking at the lsof output for the file, it seems to spend approx 6 
>> minutes opening/closing a single file (that is exactly 234k, for some 
>> reason all the files i've noticed it working with are exactly 234k)
>
>
> Interesting. It's probably the directory file you see being re-written 
> repeatedly.

The strange thing is the system eats up 100% CPU both during sync (seems 
like its writing the temporary file over and over again)

And also when running the housekeeping process, currently the 
housekeeping process has been running for 3 days and has consumed 4198 
CPU minutes.

I plan on doing the local testing today, hopefully i'll have some more 
information for you

>
>>
>> I'm going to do some testing with faster machines in a seperate 
>> environment.
>
>
> You are using the latest version, aren't you? This had a fix for large 
> directories which may help.
>
using version 0.09