[Box Backup] How to move store to new machine
Ben Summers
boxbackup@fluffy.co.uk
Tue, 16 May 2006 07:31:00 +0100
Few comments...
* Block size is not used if you're not using userland raid.
* What's up with the _bbstored user?
* bbstored doesn't need any regex libraries, only the clients do. =20
UNIX clients find your regex support (man re_format), Windows clients =20=
use boost.
* To remove the exception, comment out the regex directives in =20
bbackupd.conf. But that will backup more files, so Chris needs to =20
help you with the Win32 client.
Ben
On 16 May 2006, at 06:18, E.W. Peter Jalajas wrote:
> Thanks, Ben. Replies below.
>
> Most urgently, at the very bottom:
> Exception caught (Common RegexNotSupportedOnThisPlatform (Your =20
> platform
> does not have built in regular expression libraries.) 1/30), reset
> state and waiting to retry...
>
> On the new store I've got:
> user@new:~$ dpkg -l | grep -e boost -e pcre
> ii libboost-dev 1.33.1-2
> Boost C++ Libraries development files
> ii libboost-regex-dev 1.33.1-2
> regular expression library for C++
> ii libboost-regex1.33.1 1.33.1-2
> regular expression library for C++
> ii libpcre3 6.4-1.1ubuntu2
> Perl 5 Compatible Regular Expression Lib
> ii pcregrep 6.4-1.1ubuntu4
> grep utility that uses perl 5 compatible
>
> What should I do? Do I have to reboot after installing boost? Do I
> have too much boost?
>
> Thanks,
> Pete
>
> --- Ben Summers <ben@fluffy.co.uk> wrote:
>>
>> On 14 May 2006, at 16:30, Chris Wilson wrote:
>>>
>>>> Hmm, I just noticed that my old store raidfile.conf has:
>>>> BlockSize =3D 4096
>>>> and my new store raidfile.conf has:
>>>> BlockSize =3D 1024
>>>
>>> I don't think it makes any difference without RAID, which you
>>> aren't using?
>>
>> No, block size is ignored for non-raid files. But do keep the configs
>> the same.
>>
>> Block size is used by the raidfile lib to make sure that
>> it doesn't waste filing system blocks. So you don't "choose" a block
>> size, you set it to what your filing system uses as the underlying
>> unit of storage.
>
> We should probably be clearer when we talk about raid. I _am_ using
> raid (software md raid tools, RAID1 (mirror)), but not boxbackup
> userland raid.
>
> So, for the search engines, after some googling, I figured out "howto
> determine block size":
>
> user@old:~$ sudo tune2fs -l /dev/hda1 | grep "Block size"
> Block size: 4096
> user@new:~$ sudo tune2fs -l /dev/md0 | grep "Block size"
> Block size: 4096
>
> dumpe2fs also works.
>
> (I'll add a thing about block size to the wiki.)
>
> But, I figured that if I'm gonna start from scratch, I might as well
> tune this baby to meet my needs (with a reasonable sample of 8 =20
> clients,
> 85,000 files, and 9GB), so I did:
>
> user@old:~$ for x in 1 2 3 4 8 16 512 1024 10000 100000 1000000
> 10000000 100000000 ; do echo -n "${x}: " ; sudo find /root/.bu/backup
> -size -$x -type f | wc -l ; done
> 1: 8
> 2: 21886
> 3: 39772
> 4: 47932
> 8: 60383
> 16: 69076
> 512: 82582
> 1024: 83371
> 10000: 85227
> 100000: 85473
> 1000000: 85492
> 10000000: 85493
> 100000000: 85493
>
> So, while my average filesize is 9GB/80K =3D ~100K, it looks like more
> than half my files are under 4k, so I'll keep that as my block size.
>
> (The clients are a small tax firm, property management firm,
> environmental consultant, a database guy, a hardware broker, me (x2),
> and Mom -- maybe that's a reasonable cross-section of small
> businesses.)
>
>>>> I'm too exhausted at the moment to freak out about the fact that I
>>>> have to upgrade my clients.
>
>> This decision was made to avoid releasing code which would be used in
>> an untested manner.
>
> Sorry, Ben, the freaking was a comment about my own current lifeload,
> not at all about the product.
>
>> Really, I would suggest starting again.
>
>> * Stop bbstored. (and make sure all the child processes terminate too
>> -- they may delay doing so until the user has logged out)
>>
>> * As _bbstored, tar up the accounts.
>
> user@old:~$ su _bbstored
> su: User not known to the underlying authentication module
> user@old:~$ grep bb /etc/passwd
> _bbstored:x:1003:100::/home/_bbstored:
> user@old:~$ sudo passwd _bbstored
> Password:
> passwd: User not known to the underlying authentication module
> But then, persistence paid off:
> user@old:~$ sudo su _bbstored
> sh-3.00$ id
> uid=3D1003(_bbstored) gid=3D100(users) groups=3D100(users)
> I did that on both the old and new store server.
>
>> * Transfer the tar file. (if you don't have enough disc space, use
>> netcat to transfer it as it's created)
>> * As _bbstored, untar the accounts.
>
> On the new (target) store server, I first made a new store directory:
> sudo mkdir -p /media/bu2/backup
> sudo chown _bbstored /media/bu2
> cd /media/bu2
>
> I ran on the new store server, to get it waiting for the transfer,
> based on http://www.securitydocs.com/library/3376 :
> sh-3.00$ nc -l -p 4321 | tar xvpz =E2=80=93
>
> Then on the old (source) store server, I ran:
> sh-3.00$ tar zcp - ./backup/ | nc -w 3 new 4321
> putting the files into a new store area. (I took out the "f" option
> because I got an error.) I modified my /etc/box/bbstored.conf to
> reflect that new store directory.
>
> That took a few hours, but not too long.
>
> user@new:~$ cat /etc/box/raidfile.conf
> disc0
> {
> SetNumber =3D 0
> BlockSize =3D 4096
> Dir0 =3D /media/bu2
> Dir1 =3D /media/bu2
> Dir2 =3D /media/bu2
>
> I checked my
> /etc/box/bbstored.conf
> and
> /etc/box/bbstored/accounts.txt
> to make sure they were ok.
>
>> * Restart bbstored on the second machine.
>
> user@new:~$ sudo /usr/local/bin/bbstored
> user@new:~$ ps -ef | grep bb
> 1005 22047 1 0 23:27 ? 00:00:00 /usr/local/bin/=20
> bbstored
> 1005 22048 22047 16 23:27 ? 00:00:01 /usr/local/bin/=20
> bbstored
>
> I then did:
> user@new:~$ sudo crontab -e
> and inserted:
> #restart bbstored if it isn't running
> */15 * * * * if [[ `ps -ef | grep bbstored | grep -v "grep bbstored" |
> wc -l` -lt 2 ]] ; then killall bbstored ; /usr/local/bin/bbstored ; fi
>
> I then sent a mail to all my clients saying that they are not backing
> up at the moment, and that I need to upgrade their client backup
> software ASAP. (Maybe I shoulda woulda coulda done them one at a
> time...)
>
> So, now I'm setting up via Remote Desktop Connection one of my =20
> clients,
> following the Upgrading instructions at
> http://boxbackup.hostworks.ca/index.php/Compile_on_Windows . The
> install went fine (first make a backup copy of your C:\Program
> Files\Box Backup\bbackupd.conf, and make sure that the zip Extract
> function actually worked (you should see "Replace...?" messages); I
> changed the CommandSocket to "pipe" because that was in the
> bbackupd.conf that came with 0.10, and I extended =20
> MaximumDiffingTime to
> 40, but I'm not sure why.)
>
> And so, it's trying to connect to my new server, but now on the client
> I get:
> Exception caught (Common RegexNotSupportedOnThisPlatform (Your =20
> platform
> does not have built in regular expression libraries.) 1/30), reset
> state and waiting to retry...
> On the new store I've got:
> user@new:~$ dpkg -l | grep -e boost -e pcre
> ii libboost-dev 1.33.1-2
> Boost C++ Libraries development files
> ii libboost-regex-dev 1.33.1-2
> regular expression library for C++
> ii libboost-regex1.33.1 1.33.1-2
> regular expression library for C++
> ii libpcre3 6.4-1.1ubuntu2
> Perl 5 Compatible Regular Expression Lib
> ii pcregrep 6.4-1.1ubuntu4
> grep utility that uses perl 5 compatible
>
> The server /var/log/box shows this client (and all the others) logging
> in, and for this client, show listing directories, setting store
> marker, session finished, but no mention of any problem.
>
> What am I missing?
>
> Thanks,
> Pete
>
>> It seems like "something funny" has happened. This should all just
>> work.
>>
>> Ben
>>
>>
>>
>>
>> _______________________________________________
>> boxbackup mailing list
>> boxbackup@fluffy.co.uk
>> http://lists.warhead.org.uk/mailman/listinfo/boxbackup
> _______________________________________________
> boxbackup mailing list
> boxbackup@fluffy.co.uk
> http://lists.warhead.org.uk/mailman/listinfo/boxbackup