[Box Backup] housekeeping issues

[Alex Howansky]

I discoverd today that my main database server had not been backed up for quite
a few days. (I have the quotas set such that I can fit about 5 days worth of
dumps.) The client showed only deleted files in the dump directory -- the last
10 days or so worth of dumps had not been backed up. An info request on that
account showed that the entire allocated space was being used up by deleted
files:

                  Account ID: 00000008
              Last object ID: 9158
                 Blocks used: 2386891 (4661.90Mb)
    Blocks used by old files: 0 (0.00Mb)
Blocks used by deleted files: 2383864 (4655.98Mb)
  Blocks used by directories: 264 (0.52Mb)
            Block soft limit: 2097152 (4096.00Mb)
            Block hard limit: 2621440 (5120.00Mb)
         Client store marker: 1135020945000000

The size of the dumps is about 700 meg, which is less than the difference
between the soft and hard limits, so there should be no problem making room for
new files. It seems that housekeeping is failing.

I logged into the backup server, ran a ps command, and found about 50 or so
bbstored processes running. The log files (as far back a few days as logrotate
keeps them) were completely empty. I stopped bbstored, but one process would
not terminate, even with a -9. I had to reboot the machine to get rid of it,
and while shutting down, it complained about not being able to unmount the
filesystem that bbstored was configured to use.

After the machine came back up, I shutdown all client bbackupd daemons, and
restarted bbstored to watch the logs for housekeeping events. Since it seemed
to be only logging the "Starting housekeeping" message, I wondered how long it
was taking per account, and so I modified the source code to print "Starting
housekeeping on account N" and "Finished housekeeping on account N" for each
account.

Next, I watched the logs for my new messages. It whipped through the first two
accounts (which have only a few meg of files) and took about fifteen minutes on
the third. 35 minutes later now, it's still running on the fourth. I'm not sure
how long I'm going to wait for it.

So, two questions:

1) What happens if housekeeping takes longer than the TimeBetweenHousekeeping
setting? It seems there might be a bug here.

2) The account I really care about is #8. I need it to be backed up ASAP, but
I'm stuck on #4. Can I temporarily manually edit the accounts.txt file so that
it contains only the account #8, then restart bbstored so that it housekeeps
just that one account? Will doing so screw up any of the other accounts
(assuming I restore the file to its previous contents once the backup is
complete.)

Thanks,

-- 
Alex Howansky
Wankwood Associates
http://wankwood.com/