[Box Backup] Unclogging data store
Tue, 22 Nov 2005 18:46:06 +0000
>>>> How do I purge them short of actually deleting them on my
>>>> computers and waiting for data store to sync up? Thanks.
>>> If you are hell-bent on getting rid of the files right away, I
>>> would clear out the store on the server, and restart the backups.
>>> That will stop the files from backing up the next time, and they
>>> won't be backed up on the server. That also means that you don't
>>> have to delete any files on the client machine.
>> What confuses me is the status indicating no deleted files...
> The client might not have deleted the files yet - it may be waiting
> for a timer to expire, having detected their "supposed to be
> absent" nature. I'm running a test now to see whether this is
> likely to be the case, but it will take quite a while to complete,
> as it involves overloading an existing backup installation. In the
> meantime, just leave the whole thing alone overnight and see what
OK, I think I've identified a bug which you may be running into. It
effectively means that your backup client won't get around to
deleting the files from the backup until space is created on the
server by some other means. Here are the symptoms:
- First, the backup client uploads sufficient data to run out of
allocated space on the server. In my test example, I attempted to
backup (among other things) the whole of /usr and /var on my firewall
machine into a 4GB allocation. This includes /usr/portage (which has
several hundred MB of source archives), and /var/cache/squid (which
could be over 10GB).
- Second, the user sets an Exclude directive on some unimportant yet
bulky set of data. I used specific ExcludeDir directives on the two
bulky (but non-critical) directories mentioned above.
- Third, the client is told to reload configuration. At this point
it is supposed to remove the excluded directories from the backup,
despite the server being full. (The space will not actually be freed
until the server subsequently performs housekeeping, but this is not
The problem is that in this third step in my example situation, the
client will remove /usr/portage but not /var/cache/squid. This is
because, when the server is full, the client only examines the top-
level directories that it is configured to backup, whereas the second
exclusion is not visible unless at least the second level of the tree
I imagine that this is also why the OP's media files (which are
presumably tucked away neatly) were also not removed by the client.
The fix is to force a full directory traversal after SIGHUP,
regardless of whether the server is full or not. The present
behaviour of not uploading new or changed files can remain, as long
as unnecessary files are reliably removed.
As a temporary workaround, I suggest the OP temporarily increases the
SOFT limit to be equal to his present HARD limit on the server - the
hard limit will have to be temporarily raised to compensate. (NB:
you don't need any extra physical disk space to do this.) Then
restart the client and allow it to run once - it should now delete
the appropriate media files if configured correctly. Then, change
the server limits back to the originals - it will now perform
housekeeping and free up the appropriate amount of space. The client
should then run normally.
from: Jonathan "Chromatix" Morton
tagline: The key to knowledge is not to rely on people to teach you it.