From boxbackup-dev@boxbackup.org  Wed Sep  3 22:57:52 2008
From: boxbackup-dev@boxbackup.org (James O'Gorman)
Date: Wed, 03 Sep 2008 22:57:52 +0100
Subject: [Box Backup-dev] Trunk: bbackupd test broken?
Message-ID: <48BF0860.8030601@netinertia.co.uk>

I've just tried building the latest trunk on OS X 10.5.4 and FreeBSD 
7.0/amd64 and both fail on the bbackupd test. The failure is straight 
away in release mode, but takes a little longer in debug mode on FreeBSD 
only.


Here is the failure from OS X:

$ ./runtest.pl bbackupd debug
2 processors detected, will set make to perform concurrent jobs
   [CXX]     testbbackupd
testbbackupd.cpp: In function ‘void do_interrupted_restore(const 
TLSContext&, int64_t)’:
testbbackupd.cpp:576: error: ‘::kill’ has not been declared
*** Error code 1

Stop in /Users/jamesog/Projects/box/0.11rc2_trunk/test/bbackupd.
--------
bbackupd: make failed


and from FreeBSD:

% ./runtest.pl bbackupd release
`../../release/lib/win32/win32.a' is up to date.
`../../release/lib/common/common.a' is up to date.
`../../release/lib/compress/compress.a' is up to date.
`../../release/lib/crypto/crypto.a' is up to date.
`../../release/lib/server/server.a' is up to date.
`../../release/lib/backupclient/backupclient.a' is up to date.
`../../release/lib/raidfile/raidfile.a' is up to date.
`../../release/lib/backupstore/backupstore.a' is up to date.
`../../release/lib/intercept/intercept.a' is up to date.
`../../release/bin/bbackupctl/bbackupctl' is up to date.
`../../release/bin/bbackupquery/bbackupquery' is up to date.
`../../release/bin/bbstoreaccounts/bbstoreaccounts' is up to date.
`../../release/bin/bbstored/bbstored' is up to date.
`../../release/bin/bbackupd/bbackupd' is up to date.
   [CXX]     testbbackupd
testbbackupd.cpp: In function 'void do_interrupted_restore(const 
TLSContext&, int64_t)':
testbbackupd.cpp:576: error: '::kill' has not been declared
testbbackupd.cpp:576: error: 'SIGTERM' was not declared in this scope
testbbackupd.cpp: In function 'int test_bbackupd()':
testbbackupd.cpp:1668: warning: deprecated conversion from string 
constant to 'char*'
testbbackupd.cpp:1677: warning: deprecated conversion from string 
constant to 'char*'
testbbackupd.cpp:1742: error: '::signal' has not been declared
testbbackupd.cpp:1742: error: 'SIGPIPE' was not declared in this scope
testbbackupd.cpp:1742: error: 'SIG_IGN' was not declared in this scope
testbbackupd.cpp:1760: error: '::signal' has not been declared
testbbackupd.cpp:1760: error: 'SIG_DFL' was not declared in this scope
testbbackupd.cpp:2397: warning: deprecated conversion from string 
constant to 'char*'
testbbackupd.cpp:2406: warning: deprecated conversion from string 
constant to 'char*'
testbbackupd.cpp:2417: warning: deprecated conversion from string 
constant to 'char*'
*** Error code 1

Stop in /usr/home/jamesog/src/box/trunk/test/bbackupd.
--------
bbackupd: make failed

There goes my plan to actually get my iMac backing up tonight :-)

James

From boxbackup-dev@boxbackup.org  Thu Sep  4 00:07:24 2008
From: boxbackup-dev@boxbackup.org (Chris Wilson)
Date: Thu, 4 Sep 2008 00:07:24 +0100 (BST)
Subject: [Box Backup-dev] Trunk: bbackupd test broken?
In-Reply-To: <48BF0860.8030601@netinertia.co.uk>
References: <48BF0860.8030601@netinertia.co.uk>
Message-ID: <Pine.LNX.4.64.0809040006370.8696@top.qwarx.com>

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

---128931150-1516056998-1220483244=:8696
Content-Type: TEXT/PLAIN; charset=windows-1252
Content-Transfer-Encoding: QUOTED-PRINTABLE

Hi James,

On Wed, 3 Sep 2008, James O'Gorman wrote:

> Here is the failure from OS X:
>=20
> $ ./runtest.pl bbackupd debug
> 2 processors detected, will set make to perform concurrent jobs
>   [CXX]     testbbackupd
> testbbackupd.cpp: In function =91void do_interrupted_restore(const TLSCon=
text&,
> int64_t)=92:
> testbbackupd.cpp:576: error: =91::kill=92 has not been declared
> *** Error code 1

Thanks for pointing this out, I think I've fixed it, please could you=20
update and retest?

> There goes my plan to actually get my iMac backing up tonight :-)

sorry :(

Cheers, Chris.
--=20
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer |
\ _/_/_/_//_/___/ | We are GNU : free your mind & your software |
---128931150-1516056998-1220483244=:8696--

From boxbackup-dev@boxbackup.org  Thu Sep  4 07:31:27 2008
From: boxbackup-dev@boxbackup.org (James O'Gorman)
Date: Thu, 04 Sep 2008 07:31:27 +0100
Subject: [Box Backup-dev] Trunk: bbackupd test broken?
In-Reply-To: <Pine.LNX.4.64.0809040006370.8696@top.qwarx.com>
References: <48BF0860.8030601@netinertia.co.uk> <Pine.LNX.4.64.0809040006370.8696@top.qwarx.com>
Message-ID: <48BF80BF.2020603@netinertia.co.uk>

Hi Chris,

Chris Wilson wrote:
> Hi James,
> 
> On Wed, 3 Sep 2008, James O'Gorman wrote:
> 
>> Here is the failure from OS X:
>>
>> $ ./runtest.pl bbackupd debug
>> 2 processors detected, will set make to perform concurrent jobs
>>   [CXX]     testbbackupd
>> testbbackupd.cpp: In function ‘void do_interrupted_restore(const TLSContext&,
>> int64_t)’:
>> testbbackupd.cpp:576: error: ‘::kill’ has not been declared
>> *** Error code 1
> 
> Thanks for pointing this out, I think I've fixed it, please could you 
> update and retest?

Thanks, that's done the trick.

>> There goes my plan to actually get my iMac backing up tonight :-)
> 
> sorry :(

No worries :-)

James

From boxbackup-dev@boxbackup.org  Sat Sep  6 19:11:34 2008
From: boxbackup-dev@boxbackup.org (Peter Jalajas, GigaLock Backup Services)
Date: Sat, 6 Sep 2008 14:11:34 -0400
Subject: [Box Backup-dev] Please create me an svn account
Message-ID: <74d01c7a0809061111w243c6448i4c489fb13e505c15@mail.gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi James,

Chris has suggested that I need a svn account so that I can work on
the new Windows InstallJammer installer.

Could you please set that  up for me?

Thanks,
Pete

Digital Fingerprint: 5D13 7C5C D003 D806 8BD1 438A B396 EC3D 1C23 B5F0

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)


RZ8BfHsVh9UFqU52kelpKJVAG2QZMg9FllUexnt4lhcHnyFJYEO1ug==
=cijY
-----END PGP SIGNATURE-----

From boxbackup-dev@boxbackup.org  Fri Sep 12 23:35:33 2008
From: boxbackup-dev@boxbackup.org (Chris Wilson)
Date: Fri, 12 Sep 2008 23:35:33 +0100 (BST)
Subject: [Box Backup-dev] Re: [Box Backup] How to move boxbackup store to a larger
 =?UTF-8?Q?disk=3F?=
In-Reply-To: <Pine.LNX.4.64.0809122124470.19283@top.qwarx.com>
References: <e1ae30bb338cf19e43d60bbaec69d313@kaufmanfamily.net>
 <Pine.LNX.4.64.0809122124470.19283@top.qwarx.com>
Message-ID: <Pine.LNX.4.64.0809122324210.19283@top.qwarx.com>

Hi Ben,

> On Fri, 12 Sep 2008, David H Kaufman wrote:
> 
> > At the end of which I had a happy, larger EXT3 filesystem on new-disk. 
> > But boxbackup doesn't like something I did - I ran:
> > 
> > bbstoreaccounts check <id>
> > 
> > and got a HEAP of these:
> > 
> > File ID 61321 has different container ID, probably moved
> > File ID 61a01 has different container ID, probably moved

I could use a little help here if you have time. I'm not really sure what 
the code in BackupStoreCheck is doing.

BackupStoreCheck::CheckDirectories() loops over the stuff in mInfo (where 
exactly does this come from?) which looks like a list of blocks, each of 
which contains a list of block IDs and attributes. I think these are the 
IDs of the directory entries in the directory blocks.

Around lib/backupstore/BackupStoreCheck.cpp:634, it calls LookupID, which 
looks like it's reading the same kind of data, but returning a 
pointer to an IDBlock called piBlock (not pblock) this time. We then 
check, on line 671, whether piBlock->mContainer[iIndex] != pblock->mID[e], 
and if so, and it's a file, we report the warning above.

I don't yet follow the origins of these two data sources so I don't 
understand how they can be different. Obviously in this user's case they 
are, and the warnings emitted by the code indicate that this is due to a 
file being moved (presumably to a different container ID), and that this 
is a "normal" thing.

The code also "fixes" the entry, on line 687:

	piBlock->mContainer[iIndex] = pblock->mID[e];

but it doesn't set isModified = true, so the "fixed" directory isn't written 
back to the store.

What's the reason for not setting isModified = true at this point and 
correcting the entry's container ID? Why isn't the container ID changed 
when the file is moved, to keep the store consistent?

Thanks in advance for your help.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer |
\ _/_/_/_//_/___/ | We are GNU : free your mind & your software |

From boxbackup-dev@boxbackup.org  Sun Sep 14 13:41:04 2008
From: boxbackup-dev@boxbackup.org (Ben Summers)
Date: Sun, 14 Sep 2008 13:41:04 +0100
Subject: [Box Backup-dev] Re: [Box Backup] How to move boxbackup store to a larger disk
In-Reply-To: <20080913110002.27046.14765.Mailman@fear.love.warhead.org.uk>
References: <20080913110002.27046.14765.Mailman@fear.love.warhead.org.uk>
Message-ID: <661D48B4-18D5-4AA6-9C60-8B3D3D8DED46@fluffy.co.uk>

Chris Wilson write:
>
>> On Fri, 12 Sep 2008, David H Kaufman wrote:
>>
>>> At the end of which I had a happy, larger EXT3 filesystem on new- 
>>> disk.
>>> But boxbackup doesn't like something I did - I ran:
>>>
>>> bbstoreaccounts check <id>
>>>
>>> and got a HEAP of these:
>>>
>>> File ID 61321 has different container ID, probably moved
>>> File ID 61a01 has different container ID, probably moved
>
> I could use a little help here if you have time. I'm not really sure  
> what
> the code in BackupStoreCheck is doing.
>
> BackupStoreCheck::CheckDirectories() loops over the stuff in mInfo  
> (where
> exactly does this come from?) which looks like a list of blocks,  
> each of
> which contains a list of block IDs and attributes. I think these are  
> the
> IDs of the directory entries in the directory blocks.
>
> Around lib/backupstore/BackupStoreCheck.cpp:634, it calls LookupID,  
> which
> looks like it's reading the same kind of data, but returning a
> pointer to an IDBlock called piBlock (not pblock) this time. We then
> check, on line 671, whether piBlock->mContainer[iIndex] != pblock- 
> >mID[e],
> and if so, and it's a file, we report the warning above.
>
> I don't yet follow the origins of these two data sources so I don't
> understand how they can be different. Obviously in this user's case  
> they
> are, and the warnings emitted by the code indicate that this is due  
> to a
> file being moved (presumably to a different container ID), and that  
> this
> is a "normal" thing.
>
> The code also "fixes" the entry, on line 687:
>
> 	piBlock->mContainer[iIndex] = pblock->mID[e];
>
> but it doesn't set isModified = true, so the "fixed" directory isn't  
> written
> back to the store.
>
> What's the reason for not setting isModified = true at this point and
> correcting the entry's container ID? Why isn't the container ID  
> changed
> when the file is moved, to keep the store consistent?


The container ID is set on the client side when the stored file is  
originally created. It's not used for anything except store  
reconstruction, so it seemed a bit of a waste of time to update it  
when files were moved. If it were to be changed, the server would have  
to do it, which given the restrictive RaidFile interface would require  
rewriting the file in it's entirety.

Maybe a reasonable solution is to simply remove that "warning"?

Ben






From boxbackup-dev@boxbackup.org  Sun Sep 14 14:05:07 2008
From: boxbackup-dev@boxbackup.org (Chris Wilson)
Date: Sun, 14 Sep 2008 14:05:07 +0100 (BST)
Subject: [Box Backup-dev] Re: [Box Backup] How to move boxbackup store
 to a larger disk
In-Reply-To: <661D48B4-18D5-4AA6-9C60-8B3D3D8DED46@fluffy.co.uk>
References: <20080913110002.27046.14765.Mailman@fear.love.warhead.org.uk>
 <661D48B4-18D5-4AA6-9C60-8B3D3D8DED46@fluffy.co.uk>
Message-ID: <Pine.LNX.4.64.0809141401340.19283@top.qwarx.com>

Hi Ben,

On Sun, 14 Sep 2008, Ben Summers wrote:

> The container ID is set on the client side when the stored file is originally
> created.

Is it part of the file's header on the server then?

> It's not used for anything except store reconstruction, so it seemed
> a bit of a waste of time to update it when files were moved.

I guess if we had to use it for reconstruction, and it was wrong, then 
we'd end up putting the file back into the wrong directory, or not knowing 
where it should be at all?

> If it were to be changed, the server would have to do it, which given 
> the restrictive RaidFile interface would require rewriting the file in 
> it's entirety.

Can we fix the RaidFile interface? Especially as actual 3-stripe RAID is 
going away, we could make this a no-op on 3-stripe RAID and trivial on 
non-RAID?

> Maybe a reasonable solution is to simply remove that "warning"?

If it's too hard to actually repair the problem then I fully support doing 
that to ease needless user fears.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer |
\ _/_/_/_//_/___/ | We are GNU : free your mind & your software |

From boxbackup-dev@boxbackup.org  Mon Sep 15 21:31:54 2008
From: boxbackup-dev@boxbackup.org (Ben Summers)
Date: Mon, 15 Sep 2008 21:31:54 +0100
Subject: [Box Backup-dev] Re: [Box Backup] How to move boxbackup store to a larger disk
In-Reply-To: <20080915110002.14464.87333.Mailman@fear.love.warhead.org.uk>
References: <20080915110002.14464.87333.Mailman@fear.love.warhead.org.uk>
Message-ID: <941E60AB-3F1F-4619-9270-F1F9CB102E23@fluffy.co.uk>

Chris Wilson wrote:
>
> Hi Ben,
>
> On Sun, 14 Sep 2008, Ben Summers wrote:
>
>> The container ID is set on the client side when the stored file is  
>> originally
>> created.
>
> Is it part of the file's header on the server then?

Yes.

See BackupStoreFileEncodeStream::Setup() and file_StreamFormat.

>
>
>> It's not used for anything except store reconstruction, so it seemed
>> a bit of a waste of time to update it when files were moved.
>
> I guess if we had to use it for reconstruction, and it was wrong, then
> we'd end up putting the file back into the wrong directory, or not  
> knowing
> where it should be at all?

Yes, that's correct. But after the first sync from the backup client,  
it would be moved to the correct position. Failure of the server and  
backed up server should be unlikely to occur at the same time.

>
>
>> If it were to be changed, the server would have to do it, which given
>> the restrictive RaidFile interface would require rewriting the file  
>> in
>> it's entirety.
>
> Can we fix the RaidFile interface? Especially as actual 3-stripe  
> RAID is
> going away, we could make this a no-op on 3-stripe RAID and trivial on
> non-RAID?

I wouldn't say "fix" is quite the word, you'd need to change the way  
it worked underneath. I suspect it would be better to remove RaidFile  
complete, and do some sort of automated change to FileStreams.

>
>
>> Maybe a reasonable solution is to simply remove that "warning"?
>
> If it's too hard to actually repair the problem then I fully support  
> doing
> that to ease needless user fears.

Yes. Or at a very verbose logging method and a rewritten message. It's  
more useful for developers than anyone else.

Ben





