[Box Backup] Full server store not deleting old and deleted files (boxwin0.09)
E.W. Peter Jalajas
boxbackup@fluffy.co.uk
Wed, 1 Mar 2006 14:36:54 -0800 (PST)
Thanks, Ben.
Yes, no output at all; I couldn't see "housekeeping" or "hk" anywhere
in the log.
So, I changed the hard limit on problematic account 00000014
temporarily. I then ran a bbackupctl sync and it seemed to be doing
something, backedup a couple of little files, but hung for too many
minutes on an 8MB QuickBooks file. It's an old file, so I zipped it
and deleted the original. Restarting the Windows client service, I
could see in the Event Viewer that it was still trying to backup the
unzipped file. So I stopped the windows service again, and stopped the
server daemons with a sudo killall bbstored. I restarted
/etc/init.d/bbstored and it ran housekeeping at 95% CPU for over 20
minutes (350MHz PII with 512MB RAM). But it finally stopped.
petjal@ns1b:~$ grep -i hk /var/log/box Mar 1 16:43:37 localhost
bbstored/hk[8995]: Housekeeping process started
Mar 1 16:43:37 localhost bbstored/hk[8995]: Starting housekeeping
Mar 1 16:43:37 localhost bbstored/hk[8995]: Finished housekeeping
Mar 1 16:43:37 localhost bbstored/hk[8995]: Terminating daemon
Mar 1 16:44:17 localhost bbstored/hk[9036]: Housekeeping process
started
Mar 1 16:44:17 localhost bbstored/hk[9036]: Starting housekeeping
Mar 1 16:44:20 localhost bbstored/hk[9036]: On housekeeping, sizes in
store do not match calculated sizes, correcting
Mar 1 16:44:21 localhost bbstored/hk[9036]: different (store,calc):
acc 0x00000011, used (159014,159666), old (8227,8257), deleted
(8263,8263), dirs (1233,1233)
Mar 1 16:44:25 localhost bbstored/hk[9036]: On housekeeping, sizes in
store do not match calculated sizes, correcting
Mar 1 16:44:25 localhost bbstored/hk[9036]: different (store,calc):
acc 0x00000014, used (663024,694959), old (16068,16560), deleted
(96893,96893), dirs (1965,2030)
Mar 1 16:45:49 localhost bbstored/hk[9036]: Housekeeping giving way to
connection for account 0x00000014
Mar 1 16:45:50 localhost bbstored/hk[9036]: Account 0x00000014,
removed 12372 blocks (1669 files, 0 dirs) was interrupted
Mar 1 16:45:52 localhost bbstored/hk[9036]: Account 0x01abc23d,
removed 93 blocks (0 files, 93 dirs)
Mar 1 16:45:52 localhost bbstored/hk[9036]: Account 0x123abc4d,
removed 1 blocks (0 files, 1 dirs)
Mar 1 16:45:52 localhost bbstored/hk[9036]: Finished housekeeping
Mar 1 16:54:59 localhost bbstored/hk[9036]: Terminating daemon
Mar 1 16:55:19 localhost bbstored/hk[9614]: Housekeeping process
started
Mar 1 16:55:20 localhost bbstored/hk[9614]: Starting housekeeping
Mar 1 17:18:36 localhost bbstored/hk[9614]: Account 0x00000014,
removed 44398 blocks (3816 files, 19 dirs)
Mar 1 17:18:36 localhost bbstored/hk[9614]: Finished housekeeping
Mar 1 17:19:36 localhost bbstored/hk[9614]: Starting housekeeping
Mar 1 17:19:39 localhost bbstored/hk[9614]: Finished housekeeping
I restarted the service on the windows client and it is now stuck
trying to backup the old file that doesn't exist anymore; from the
windows Event Viewer:
Send ListDirectory(0x5647,0xffffffff,0xc,true).
Receive Success(0x5647).
Receiving stream, size 489.
Send GetBlockIndexByName(0x5647,"Hallmos 101503 2.QBB").
Receive Success(0x5648).
Receiving stream, size 89148.
Sending stream, size uncertain.
Send
StoreFile(0x5647,0x40dca63288f40,0x263be2696554d151,0x5648,"Hallmos
101503 2.QBB").
Sending stream, size uncertain.
>From /var/log/box on the server:
Mar 1 17:26:37 localhost bbstored[11200]: Send Success(0x5647)
Mar 1 17:26:37 localhost bbstored[11200]: Sending stream, size 489
Mar 1 17:26:37 localhost bbstored[11200]: Receive
GetBlockIndexByName(0x5647,OPAQUE)
Mar 1 17:26:37 localhost bbstored[11200]: Receive
GetBlockIndexByName(0x5647,OPAQUE)
Mar 1 17:26:37 localhost bbstored[11200]: Send Success(0x5648)
Mar 1 17:26:37 localhost bbstored[11200]: Send Success(0x5648)
Mar 1 17:26:37 localhost bbstored[11200]: Sending stream, size 89148
Mar 1 17:26:54 localhost bbstored[11200]: Receive
StoreFile(0x5647,0x40dca63288f40,0x263be2696554d151,0x5648,OPAQUE)
Mar 1 17:26:54 localhost bbstored[11200]: Receive
StoreFile(0x5647,0x40dca63288f40,0x263be2696554d151,0x5648,OPAQUE)
Mar 1 17:26:54 localhost bbstored[11200]: Receiving stream, size
uncertain
Mar 1 17:34:37 localhost bbstored/hk[9614]: Starting housekeeping
Mar 1 17:34:38 localhost bbstored/hk[9614]: Finished housekeeping
The problem again is the the .QBB file doesn't exist anymore; it has
been replaced by the .ZIP file of the same name. And the bb system is
hanging on the non-existent .QBB file. What should I do? (I haven't
checked exactly how much storage on the client is represented by the
BackupLocations, as compared to the soft limit.)
Thanks again,
Pete
--- Ben Summers <ben@fluffy.co.uk> wrote:
>
> On 1 Mar 2006, at 14:04, E.W. Peter Jalajas wrote:
>
> >>>
> >>> So, why haven't the old and deleted files been deleted? And what
> >>> should I do to fix it?
> >>
> >> Check your server logs to see if there's anything about
> interrupted
> >> housekeeping.
> >
> > These all showed nothing:
> > 530 grep -i house /var/log/box.0
> > 531 grep -i house /var/log/box
> > 532 grep -i hk /var/log/box
> > 533 grep -i hk /var/log/box.0
>
> No output at all? Or no messages about interruptions? If the former,
>
> this suggests that there was no housekeeping activity in that
> timeframe, so you need to search further.
>
> >
> >> If the housekeeping takes too long and the client
> >> connects too frequently, then housekeeping never deletes anything.
> >
> > Here's the timings from their bbackupd.conf:
> >
> > UpdateStoreInterval = 3600
> > MinimumFileAge = 21600
> > MaxUploadWait = 86400
> > FileTrackingSizeThreshold = 65535
> > DiffingUploadSizeThreshold = 8192
> > MaximumDiffingTime = 20
> >
> >>
> >> This is something to be aware of, and the reason I want to move to
> >> ref-counted stores in the 0.20 version.
> >>
> >
> > So, for now, should I just increase their quota? Temporarily?
> Should
> > I put the known large-file BackupLocations (those with the big
> > QuickBooks file) at the end of the bbackupd.conf listing?
>
> Increase the hard limit temporarily. Check the logs for the message
> where it says it deleted stuff. If you don't find that in the near
> future, then you need to check that the client does not have more
> data than your soft limit.
>
> But the first task is to find out exactly what the housekeeping is
> doing.
>
> Ben
>
>
>
> _______________________________________________
> boxbackup mailing list
> boxbackup@fluffy.co.uk
> http://lists.warhead.org.uk/mailman/listinfo/boxbackup
>