[Box Backup] bbackupd crashing
Ben Summers
boxbackup@fluffy.co.uk
Sat, 7 Jan 2006 08:32:35 +0000
It looks like your Berkeley inode tracking databases have got
corrupted. Stop bbackupd, delete the contents of /var/bbackupd (or
whatever specified by DataDirectory in bbackupd.conf), and restart
bbackupd.
This is harmless, although if you've been renaming large files it may
result in some unnecessary use of bandwidth and disc space on the
server.
Berkeley DB v1 does seem to do this occasionally. In the forthcoming
release, we also support a later version which I hope is more reliable.
Ben
On 7 Jan 2006, at 05:31, Soren Dossing wrote:
> Thanks for boxbackup. It's a great tool that has saved me several
> times.
>
> I run bbackupd 0.09 on freebsd 5.4. It has been running somehow
> stable for several months. It sometimes crashed, but I can just
> restart it as if nothing happened. No system changes have happened
> for long time.
>
> Recently bbackupd began crashing soon after starts up. It always
> crashes on the same file, even though there have been no changes to
> the file. I have tried to remove the file it chokes on; bbackupd
> then runs a little longer but chokes on a different file. Both the
> debug and no-debug version of bbackupd stops at the same point.
>
> To run 'bbstoreaccounts check 1234 fix' on the backup server did
> not help.
>
> To reboot the backup client did not help.
>
> syslog says:
>
> kernel: pid 7330 (bbackupd), uid 0: exited on signal 5 (core dumped)
>
> Last 10 lines from truss output:
>
> # truss -f -o bbackup.truss /usr/ports/sysutils/boxbackup/work/
> boxbackup-0.09/debug/bin/bbackupd/bbackupd
> # sleep 60
> # tail -10 bbackup.truss
> 7027: lstat("/usbdisk/home/sauber/projects/dev/linker/htdocs/.p/
> scene2/uploadcontribution.jpg",0xbfbfc660) = 0 (0x0)
> 7027: lstat("/usbdisk/home/sauber/projects/dev/linker/htdocs/.p/
> scene2/uploadcontribution2.gif",0xbfbfc660) = 0 (0x0)
> 7027: lstat("/usbdisk/home/sauber/projects/dev/linker/htdocs/.p/
> scene2/validatevotekey.gif",0xbfbfc660) = 0 (0x0)
> 7027: lstat("/usbdisk/home/sauber/projects/dev/linker/htdocs/.p/
> scene2/validatevotekey2.gif",0xbfbfc660) = 0 (0x0)
> 7027: gettimeofday({1136609196 233645},0x0) = 0 (0x0)
> 7027: stat("/usbdisk/home/sauber/projects/dev/linker/htdocs/.p/
> scene2/CVS",0xbfbfc3e0) = 0 (0x0)
> SIGNAL 10 (SIGBUS)
> SIGNAL 10 (SIGBUS)
> Process stopped because of: 16
> process exit, rval = 138
> #
>
> Examinig the core dump file:
>
> # gdb
> (gdb) core bbackupd.core
> [...]
> (gdb) bt
> [...]
> #3962 0x1840f750 in ?? ()
> #3963 0x00020000 in ?? ()
> #3964 0x688e0375 in ?? ()
> #3965 0x0067b844 in ?? ()
> #3966 0xcd500000 in ?? ()
> #3967 0x90feeb80 in ?? ()
> #3968 0x28365578 in ?? ()
> #3969 0x00000001 in ?? ()
> #3970 0xbfbfed58 in ?? ()
> #3971 0x0000000d in ?? ()
> Cannot access memory at address 0xbfc00000
> (gdb)
>
> Output from gdb when attached to a process:
>
> # gdb
> (gdb) attach 7277
> Attaching to process 7277
> [...]
> (gdb) c
> Continuing.
>
> Program received signal SIGBUS, Bus error.
> 0x28353597 in memmove () from /lib/libc.so.5
> (gdb) bt
> #0 0x28353597 in memmove () from /lib/libc.so.5
> #1 0x28345a41 in __delpair () from /lib/libc.so.5
> #2 0x28348783 in __call_hash () from /lib/libc.so.5
> #3 0x0805a19d in BackupClientInodeToIDMap::AddToMap (this=0x80f1250,
> InodeRef=3509281, ObjectID=963, InDirectory=914)
> at BackupClientInodeToIDMap.cpp:228
> #4 0x08050b52 in BackupClientDirectoryRecord::SyncDirectory
> (this=0x81293c0,
> rParams=@0xbfbfe8c0, ContainingDirectoryID=914,
> rLocalPath=@0xbfbfc650,
> ThisDirHasJustBeenCreated=false) at
> BackupClientDirectoryRecord.cpp:185
> #5 0x08053891 in BackupClientDirectoryRecord::UpdateItems
> (this=0x8129300,
> rParams=@0xbfbfe8c0, rLocalPath=@0xbfbfcac0,
> pDirOnStore=0x8129780,
> rEntriesLeftOver=@0xbfbfc7a0, rFiles=@0xbfbfc830,
> rDirs=@0xbfbfc820)
> at BackupClientDirectoryRecord.cpp:996
> #6 0x08051650 in BackupClientDirectoryRecord::SyncDirectory
> (this=0x8129300,
> rParams=@0xbfbfe8c0, ContainingDirectoryID=909,
> rLocalPath=@0xbfbfcac0,
> ThisDirHasJustBeenCreated=false) at
> BackupClientDirectoryRecord.cpp:376
> #7 0x08053891 in BackupClientDirectoryRecord::UpdateItems
> (this=0x8128c80,
> rParams=@0xbfbfe8c0, rLocalPath=@0xbfbfcf30,
> pDirOnStore=0x8129180,
> rEntriesLeftOver=@0xbfbfcc10, rFiles=@0xbfbfcca0,
> rDirs=@0xbfbfcc90)
> at BackupClientDirectoryRecord.cpp:996
> #8 0x08051650 in BackupClientDirectoryRecord::SyncDirectory
> (this=0x8128c80,
> rParams=@0xbfbfe8c0, ContainingDirectoryID=906,
> rLocalPath=@0xbfbfcf30,
> ThisDirHasJustBeenCreated=false) at
> BackupClientDirectoryRecord.cpp:376
> #9 0x08053891 in BackupClientDirectoryRecord::UpdateItems
> (this=0x8128b00,
> rParams=@0xbfbfe8c0, rLocalPath=@0xbfbfd3a0,
> pDirOnStore=0x8128cc0,
> rEntriesLeftOver=@0xbfbfd080, rFiles=@0xbfbfd110,
> rDirs=@0xbfbfd100)
> at BackupClientDirectoryRecord.cpp:996
> #10 0x08051650 in BackupClientDirectoryRecord::SyncDirectory
> (this=0x8128b00,
> rParams=@0xbfbfe8c0, ContainingDirectoryID=890,
> rLocalPath=@0xbfbfd3a0,
> ThisDirHasJustBeenCreated=false) at
> BackupClientDirectoryRecord.cpp:376
> #11 0x08053891 in BackupClientDirectoryRecord::UpdateItems
> (this=0x81158c0,
> rParams=@0xbfbfe8c0, rLocalPath=@0xbfbfd810,
> pDirOnStore=0x8115e40,
> rEntriesLeftOver=@0xbfbfd4f0, rFiles=@0xbfbfd580,
> rDirs=@0xbfbfd570)
> at BackupClientDirectoryRecord.cpp:996
> #12 0x08051650 in BackupClientDirectoryRecord::SyncDirectory
> (this=0x81158c0,
> rParams=@0xbfbfe8c0, ContainingDirectoryID=684,
> rLocalPath=@0xbfbfd810,
> ThisDirHasJustBeenCreated=false) at
> BackupClientDirectoryRecord.cpp:376
> #13 0x08053891 in BackupClientDirectoryRecord::UpdateItems
> (this=0x8112e80,
> rParams=@0xbfbfe8c0, rLocalPath=@0xbfbfdc80,
> pDirOnStore=0x8113000,
> rEntriesLeftOver=@0xbfbfd960, rFiles=@0xbfbfd9f0,
> rDirs=@0xbfbfd9e0)
> at BackupClientDirectoryRecord.cpp:996
> #14 0x08051650 in BackupClientDirectoryRecord::SyncDirectory
> (this=0x8112e80,
> rParams=@0xbfbfe8c0, ContainingDirectoryID=598,
> rLocalPath=@0xbfbfdc80,
> ThisDirHasJustBeenCreated=false) at
> BackupClientDirectoryRecord.cpp:376
> #15 0x08053891 in BackupClientDirectoryRecord::UpdateItems
> (this=0x80f3b80,
> rParams=@0xbfbfe8c0, rLocalPath=@0xbfbfe0f0,
> pDirOnStore=0x8112080,
> rEntriesLeftOver=@0xbfbfddd0, rFiles=@0xbfbfde60,
> rDirs=@0xbfbfde50)
> at BackupClientDirectoryRecord.cpp:996
> #16 0x08051650 in BackupClientDirectoryRecord::SyncDirectory
> (this=0x80f3b80,
> rParams=@0xbfbfe8c0, ContainingDirectoryID=493,
> rLocalPath=@0xbfbfe0f0,
> ThisDirHasJustBeenCreated=false) at
> BackupClientDirectoryRecord.cpp:376
> #17 0x08053891 in BackupClientDirectoryRecord::UpdateItems
> (this=0x80f3100,
> rParams=@0xbfbfe8c0, rLocalPath=@0xbfbfe560,
> pDirOnStore=0x80f3bc0,
> rEntriesLeftOver=@0xbfbfe240, rFiles=@0xbfbfe2d0,
> rDirs=@0xbfbfe2c0)
> at BackupClientDirectoryRecord.cpp:996
> #18 0x08051650 in BackupClientDirectoryRecord::SyncDirectory
> (this=0x80f3100,
> rParams=@0xbfbfe8c0, ContainingDirectoryID=3,
> rLocalPath=@0xbfbfe560,
> ThisDirHasJustBeenCreated=false) at
> BackupClientDirectoryRecord.cpp:376
> #19 0x08053891 in BackupClientDirectoryRecord::UpdateItems
> (this=0x80c7dc0,
> rParams=@0xbfbfe8c0, rLocalPath=@0x80f01c4,
> pDirOnStore=0x80e0780,
> rEntriesLeftOver=@0xbfbfe6b0, rFiles=@0xbfbfe740,
> rDirs=@0xbfbfe730)
> at BackupClientDirectoryRecord.cpp:996
> #20 0x08051650 in BackupClientDirectoryRecord::SyncDirectory
> (this=0x80c7dc0,
> rParams=@0xbfbfe8c0, ContainingDirectoryID=1,
> rLocalPath=@0x80f01c4,
> ThisDirHasJustBeenCreated=false) at
> BackupClientDirectoryRecord.cpp:376
> #21 0x0805bb89 in BackupDaemon::Run2 (this=0xbfbfecc0) at
> BackupDaemon.cpp:492
> #22 0x0805b05a in BackupDaemon::Run (this=0xbfbfecc0) at
> BackupDaemon.cpp:270
> #23 0x0807ed13 in Daemon::Main (this=0xbfbfecc0,
> DefaultConfigFile=0x809ae54 "/usr/local/etc/box/
> bbackupd.conf", argc=1,
> argv=0xbfbfed50) at Daemon.cpp:330
> #24 0x0806334a in main (argc=1, argv=0xbfbfed50) at bbackupd.cpp:61
> (gdb)
>
> In this case, bbackupd stopped on a directory. Checking the file
> system:
>
> # ls -ld /usbdisk/home/sauber/projects/dev/linker/htdocs/.p/scene2/
> CVS
> drwxr-xr-x 2 sauber sauber 512 Sep 4 2000 /usbdisk/home/
> sauber/projects/dev/linker/htdocs/.p/scene2/CVS
> # stat /usbdisk/home/sauber/projects/dev/linker/htdocs/.p/scene2/CVS
> 1049 3509281 drwxr-xr-x 2 sauber sauber 14038505 512 "Jan 7
> 14:28:44 2006" "Sep 4 06:06:57 2000" "Jan 4 03:10:51 2005" "Sep
> 4 06:06:57 2000" 4096 4 0 /usbdisk/home/sauber/projects/dev/linker/
> htdocs/.p/scene2/CVS
> #
>
> Obviously I would like to get bbackupd up and running again. Please
> let me know if I can provide any other information.
>
> Thanks,
>
> Soren
> _______________________________________________
> boxbackup mailing list
> boxbackup@fluffy.co.uk
> http://lists.warhead.org.uk/mailman/listinfo/boxbackup