[Box Backup] child bbstored process ramping up to 100% cpu during bbackupd sync?

[reticent]

Ben Summers wrote:

>
> On 28 Jan 2005, at 22:45, reticent wrote:
>
>> This is just based off of observation, i'm just curious if anyone has
>> experienced this in the past.
>>
>> The client bbackupd server is a slow USPARC IIi 333mhz with 1.6 gigs of
>> data that doesn't compress well (audio data).
>>
>> There are about 350,000 directories and half that amount of files.
>>
>> The (lazy) sync is painfully slow (somthing like 24 hours).
>>
>> However what i'm seeing is that during the sync on the backup server
>> (bbstored) the child process spawned on connection ramps up to 100% cpu
>> usage as the file that is currently being written increases in size.
>>
>> Looking as lsof output i can see that there are two files being written
>> to  by the child process.
>> (for example): o28.rfw and o28.rfwX
>> o28.rfw seems to grow according the amount of data that is coming in.
>> However it seems that the file (and this changes for every new rfw file
>> that is written) o28.rfwX is constantly being written and deleted.
>> The strange thing is the file seems to be written up the current size of
>> o28.rfw and then deleted. So as o28.rfw grows in size the cpu useage
>> increases evetually using 100% once o28.rfw gets to about 20-30KB.
>>
>> Backup server is a dual pIII xeon 500mhz with plenty of spare (ecc) 
>> memory.
>>
>>
>> I havn't noticed this when backing up data from other (much faster)
>> servers.
>>
>> Has anyone done any benchmarks?
>
>
> What you are seeing is a patch being received by the server, and 
> applied. The 20-30kb of data is the patch, then there's a bit of 
> computation required to apply the patch to the existing file, which 
> overwrites the original file.
>
> What is the usage pattern of files on the client? Are they static? Are 
> they being constantly modified? Are files being added regularly?
>
>
>> I've been getting really slow rates when backing up/restoring.. 
>> 200-500BPS in the exchange described above.. This is over a local FD 
>> 100M network
>
>
> I would expect restore to be quite fast.
>
> Ben
>
> _______________________________________________
> boxbackup mailing list
> boxbackup@fluffy.co.uk
> http://lists.warhead.org.uk/mailman/listinfo/boxbackup
>
>
The files are, for the most part, static. There are only a few hundred 
updates/additions per day.
They are also very small, ~1-20kB
No modifications are made to them


The quoted speed was during a backup, i'll have to take some time to do 
some debugging as somthing is definitly not right here.

I'm getting the feeling that it might be related to the amount of 
directories that are being delt with, the backup completed (30+ hours 
later for 1.6 gigs..) and i'm seeing the hk process run and eat up 100% 
cpu when checking backup data for host mentioned in the original email.

Looking at the lsof output for the file, it seems to spend approx 6 
minutes opening/closing a single file (that is exactly 234k, for some 
reason all the files i've noticed it working with are exactly 234k)

I'm going to do some testing with faster machines in a seperate environment.