[Box Backup] Win32 port (was: BoxBackup Server Side Management Specs (Draft0.01))

[Ben Summers]

On 4 Oct 2004, at 22:27, Chris Wilson wrote:

> Hi Ben,
>
>>> However, my longer-term plan is to communicate with bbackupd, both by
>>> modifying its config file, and using its existing socket, so that it
>>> becomes possible to see what the daemon is doing in real time (e.g. 
>>> why
>>> hasn't it backed up File X)
>>
>> I was thinking of adding a mechanism to do this would would open an
>> external executable, and pipe in detailed descriptions of what it was
>> up to.
>
> Why an external executable? I can see no reason to prefer this to a
> simple unix socket. It adds extra complexity and inefficiency in the 
> case
> of a simple client like the one I'm writing. Why would I want to have 
> to
> write a program that just stores everything it gets from Box, and 
> spits it
> back out over a unix socket? Why not just extend the existing socket
> protocol, or create a new one?

Yes, you're probably right. I was just thinking about a previous plan 
for this to allow users to write scripts to provide whatever logging 
they wanted. Different application here.

[snip]
>> I welcome contributions to the code, obviously subject to quality and
>> complexity considerations. I would suggest a quick email with proposed
>> changes to me beforehand to check the design, though!
>
> Good idea. Here goes:
>
> My goal is for a client, connected (by pipe or socket) to bbackupd, to 
> be
> able to recreate all the important parts of bbackupd's state, 
> especially:
>
> * current state in state machine (initialising, idle, syncing, error,
>   limit exceeded), time of next sync
> * state of server connection (connected/not connected, reason for last
>   disconnection)

These are easy.

> * state of each file when last scanned (not changed, too new, synced,
>   error, excluded)

Is this to be queried at any time? Such state is not kept.

> * progress during sync (current file ID,

It would be easy to send the current filename and other details as the 
scan progressed.

>  number scanned,

Easy.

>  estimated number
>   remaining)

Not so easy. You don't really know where you are in the backup process 
until it's finished. It all depends on how many files there are, how 
much data has changed, etc. Giving a reliable progress indication isn't 
an easy task.

> * store status (bytes used, etc.)

You can also query the store for this.

> * statistics on last sync

Easy.

>
> Ideally, I would like to know when any of these things changes, in 
> order
> to provide live notifications to the user when something is wrong, and
> display live status as they are investigating the reason for a 
> problem, or
> just checking the status of their backup.
>
> I would like to suggest a protocol where, on connection from a client,
> bbackupd reports the current value of each of these state items 
> (including
> its entire file list) to the client, and subsequently reports the new
> value if any of them changes, or the client disconnects and reconnects.

Yes, it could do that.

>
> How about XML for the protocol? Here is my first suggestion for the 
> event
> types:
>
> <box:client-status-protocol version="1.0">
> 	<box:daemon state="init|idle|sync|error|overlimit"
> 		nextsync="..." />
> 	<box:conn state="connected|disconnected" reason="..." />
> 	<box:file id="[box-id]" parent="[parent-id]" name="..."
> 		age="..." state="unchanged|toonew|synced|error|excluded"
> 		errorcode="X/X" />
> 	<box:progress id="..." count="..." remaining="..." />
> 	<box:store used="..." old="..." deleted="..." dirs="..."
> 		softlimit="..." hardlimit="..." />
> 	<box:stats uploaded="..." unchanged="..." encoded="..."
> 		sent="..." rcvd="..." time="..." />
> </box:client-status-protocol reason="daemon-shutdown|admin-killed">

Let's not use XML. I think a simple text based protocol will be 
sufficient. XML is not the answer to every problem.

>
>> Yes, that would be a perfectly acceptable way to do that. It would be
>> better to adjust things so that the in-memory state is retained over
>> restarts to accommodate that without having to query the entire store
>> against the server after every device removal.
>>
>> An alternative would be to allow bbackupctl to selectively enable and
>> disable locations.
>
> I think the latter option would be better, especially if adding a new
> removable device to the configuration does not require Box to requery 
> the
> entire store.

I've added this to the list. (And I also plan to have an option to dump 
the state to disc so that requerying isn't necessary if you don't mind 
a bit of state stored on the disc.)

>
>> This use is, but multiple streams in a single file is something NTFS
>> has done for years.
>
> Sorry, yes, I do know about that. But in past versions, the security
> information is intrinsic to an object, not stored in a (user-visible)
> stream?

It's not quite security information. As I understand it, the source URL 
and a "zone identifier" is stored in a separate stream, so it knows 
where the exe came from and whether to run it without warning the user 
or not.

>
>> There is other work which is higher up on the list, mainly to make it
>> easier to use.
>
> Perhaps I can help you there :-) I had a look at TODO/txt and couldn't 
> see
> any items that appear to relate to ease of use. What needs doing on 
> that
> front?

I must update that list sometime. I have a separate file of notes on 
things I want to do.

Ben