[Box Backup] Files are uploaded/patched daily even though they have not changed in weeks.

[Matt Brown]

Hi Chris,

>> So I created a new bbstored on the remote server (a 1 & 1 root  
>> server)
>> and copied the same raw keys etc to it and the backup ran fine,  
>> however
>> due to 1 & 1 randomly reseting connections to their servers  
>> (presumably
>> to avoid the root servers being used in such a way)
>
> Damn, that sucks. I had no idea 1&1 would do that. I haven't seen any
> evidence of it happening with a root server that I use there. What  
> does it
> look like? Is it long-running or idle connections that get  
> terminated, or
> any connection? Could it be a firewall state table timeout?

Seems to be idle timeout, if I leave an ssh session open it gets  
closed. I do use their firewall option, so it could be a state thing.

>> I was forced to point the data back to the original server so at  
>> least
>> the data was being backed up - (the pipe would fail and box would  
>> fail
>> to continue once the connection failed without restarting the sync,
>> leaving me with incomplete backups).
>
> I would definitely like to know more about this. Can you reproduce  
> this
> issue? What version of Box were you using? What do you get in the logs
> after the aborted connection?

Well currently the version I am using is  v0.11rc2 but I noticed I am  
using the state db

i.e StoreObjectInfoFile = /var/bbackupd/bbackupd.state

It does say ...

#This saves time and increases efficiency when
# bbackupd is frequently stopped and started, since it removes the need
# to rescan all directories on the remote server. However, it is new and
# not yet heavily tested, so use with caution.

So this could be the culprit .. tho it has behaved until now ....

>> This is where the strange issue is, every day I get an email with the
>> contents of the backup via the boxreporter.py script, and the  
>> contents
>> are the same + changed files for that day. To make sure it was not a
>> reporting issue, I ran the sync by hand and monitored the logs. Sure
>> enough the same files are upload or patched - but the files in  
>> question
>> have not changed for over a week now.
>>
>> It appears somehow, the files in question are always seen as  
>> changed and
>> forces a sync or is cached (well thats the behavior) ?
>>
>> Any ideas how I can debug this to work out what is going on ?
>
> Could you try running bbackupd with LogAllFileAccess = yes and the -V
> option and see what it says about these files?
>
Yep got all these enabled, as used for the reporting function ....

> Could it be that their modification date is in the future, or is being
> updated?
>
Timestamps look fine.

>> Could it be down to the maxdiffing time ?
>
> It could be, although I think that if Box thinks the file has  
> changed then
> it will calculate and upload a patch, even if it's an empty patch  
> (i.e.
> there were no actual changes), so it probably happened before that,  
> when
> Box decided that it thought the file had been changed.

I could always blat the store for this account - which would clear the  
issue, however I am curious as to why it thinks the files are not sync'd

I shall do some more digging to see what is occuring ..

Matt