[Box Backup] 100% CPU on big directories

[Jamie Neil]

Ben Summers wrote:
>> Yes, I know it's not clever to have a flat directory with 150,000 files
>> in it :), but I've noticed the effect to a smaller degree on directories
>> with only 1000 files. Does anyone know if it's fixable?
> 
> I can think of a couple of places in the code which wouldn't appreciate
> this. It might be possible to get a speed-up by using a hash_map instead
> of a map in critical places.

I removed a large amount of the files in the offending directories which
brought the number of files in each directory down to about 30,000. The
utilisation still hit 100%, but it did complete this time (took about 8
hours to back up 446MB).

> I've added a note to the wiki about this to make sure we think about it
> in the next round of changes.
> 
> Sorry not to be more helpful now.

That's ok - the problem is avoidable in most situations. It might
present a problem where you can't control what people are backing up though.

-- 
Jamie Neil | <jamie@versado.net> | 0870 7777 454
Versado I.T. Services Ltd. | http://versado.net/ | 0845 450 1254