[Box Backup] cygwin issues, changes during upload (was: help again)

[Ben Summers]

On 18 Aug 2004, at 15:01, Nick Knight wrote:

> Hello,
>
> I figure this one out late last night. Although cygwin was never
> installed on this machine a cygwin1.dll was in the system directory so
> old it couldn't handle files larger than 2GB - and it was so it
> couldn't! Because it was in the system directory it was picked up 
> before
> the one in the box backup directory. Deleted and now it works fine.
>
> Not sure were they come from - brand new machine running windows 2003
> server with hardly anything on it - surely MS can't be putting it 
> there?

DLL Hell. A favourite of any Windows user!

>
> You're known issue about a file changing is magnified with the problem
> that it takes longer to ship it out over the Internet than to dump it 
> to
> tape, I have opened this up to discussion with the other guys - see if
> they can think of something. The immediate obvious one is to make a 
> copy
> then transmit - this will add time - and require an overhead in local
> disk space - but it reduces the probability of an error - but not the
> cleanest.

With your 2Gb file, that's a lot to copy. Where would you put it? And 
you've still got the problem of changes during the code. A difficult 
one.

It's not such a big problem as you initially might think:

* If you're using snapshot mode, you're scheduling it for a time when 
files aren't going to be active, aren't you?

* If you're using lazy mode, then the upload algorithm won't upload 
files which are recently changed (except if this would give an 
unacceptable delay in backing it up). So it won't do anything with a 
file until it's safely closed and ready to go.


A better solution would require changes to be detected during upload, 
and the upload aborted if this happens. Two ways.

1) Calculate a digest of the file as you upload it. After upload, 
calculate the digest again, and if it doesn't match, discard the file 
uploaded.

2) On platforms which support it, monitor changes using kqueue, and if 
it changes, abort the upload.

Neither of these are entirely satisfactory, or easy to implement well. 
Thoughts welcome.

Ben