[Box Backup] traffic statistics

[Ben Summers]

On 21 Oct 2004, at 16:16, Dennis Speekenbrink wrote:

>
>>
>> If you're doing billing, you'd probably want actual bytes sent and 
>> received. I was thinking of modifying the socket wrappers to count 
>> the bytes sent and received, then when they were closed send this 
>> information to some other daemon which would process the accounting 
>> data.
>>
>> Do you have a pressing need for this information?
>>
>> Ben
>
>
> One of our clients requested a remote backup service.  Box Backup 
> would seem a very likely candidate for this (I still need to verify 
> some of it's merits but so far so good!)

  ... and there might be a native Win32 client soon... :-)


>  and I'm contemplating offering the service to others.  Data 
> throughput statistics would keep my bandwith costs in check (i.e. 
> charging "heavy" customers), as well as diagnose any connectivity 
> issues due to burst traffic by some account.
> The initial load is not *expected* to be high, but for future plans 
> (or an unexpected surge) some primitive means of managing data traffic 
> is necessary.
> If as you say, some idea of accounting is in the works, (and you sound 
> like it would not be a big hurdle), can I assume that it is planned 
> for some time in the near future (within, say 6 months)?

It hasn't been implemented so far because I haven't needed it, and I 
still don't. But I'm always open to persuasion.


> As I will be needing such a feature in the future, deciding on which 
> backup solution to go for depends on this.  The initial deployment 
> will be soon I expect, but accounting need not be available at the 
> start.
>
> Of course I don't want to demand anything, and if the feature is not 
> too diffucult to realise, I will try to implement it myself (no 
> promises here though :) ).
>
> As for the wishlist of this feature:
> Basic:
> - total data traffic per account (preferably including any command and 
> control communications)
> - ability to poll  (reset?)
> - statics integrity between subsequent runs of the daemon (i.e. no 
> (large) loss of statistical data on reboot or crash of the server)
>
> Ultimate:
> (to be able to generate informational graphs and such, I'm thinking of 
> WebAlizer and the like here)
> - number of files
> - seperate upload / download statistics
> - number of syncs
> - seperate verify/command/transfer statistics?
>
> Note: I'm not fully familiar with the protocol, so where I say: 
> "command and  control" I mean any traffic that is not client data, 
> such as checksums.

Yes, there's a big difference between transferred data uploaded and the 
protocol overhead of both Box Backup and SSL. And you'd also want to 
catch usage for restores. What I was thinking of was whenever a 
connection was closed, a message was sent to a UNIX domain 'UDP' 
socket, looking like

   4a 3f466a 83be32

where the numbers are

   <32 bit account number> <64 bit bytes sent> <64 bit bytes received>

in hex. Then you just write a script or something which listens on the 
relevant socket, and logs all data transfers. You should be able to 
easily do just about everything there with a spot of perl.

The only complication is that to do things properly on SSL connections 
you have to write your own BIO object, which should be OK, but not 
completely trivial -- you have fun and games when you try and implement 
C++ style objects using only C.


There are some others on the list working on some management tools, I'm 
sure they'll get in touch.

Ben