[Box Backup-dev] Soft-RAID support

[Chris Wilson]

Hi David,

On Sat, 6 Jun 2009, David Sommerseth wrote:

> Chris Wilson wrote:
>>> "The server currently supports a kind of RAID 5 in userland for extra 
>>> reliability... This is deprecated and will be removed in a future 
>>> version."
>>>
>>> Is there any reasons this will be changed?
>>
>> Support for it was never finished (no recovery procedure), it is pretty
>> limited (only supports RAID 5 and three devices) and it was written at a
>> time when OS/software and hardware RAID were not as ubiquitous or well
>> supported as they are now.
>
> I would be willing, with some guidance to look into such a tool, if that
> is the main criteria for dropping this support.

That would definitely be very helpful, thanks in advance. You can read the 
encrypted objects (which are reconstructed successfully) and then rewrite 
them, which will reestablish the redundant copies.

> The soft-raid solution itself seems to work flawlessly and seems to only 
> need this recovery tool.  Or are there any other issues which is not to 
> well known with the soft-raid which should make me worried?  Are there 
> any critical bugs related to the current implementation?

No, I don't think so. All of our tests actually run in RAID mode, hence 
the "more tested" aspect. However it does impose significant performance 
limitations which may prevent me from making some optimisations to reduce 
disk I/O in future, and the new refcount database will not be mirrored, 
but it can be reconstructed by housekeeping in any case, so it's more of a 
cache than a database.

>> I can see your point about the usefulness of this for distributed
>> encrypted backup. However I'm not convinced about the overall merits of
>> storing the data in three separate locations.
>
> Regarding encryption, yes, that is one key element.  But if the
> organisation looses one remote storage with the complete backup directory,
> it got all the needed information needed to begin to crack the encryption.
> If you need minimum 2 sets to be able to crack the encryption, you have
> another layer of security.  And it was this combination which caught my
> attention.  When you add locally encrypted disks, you have the third layer
> of security.

That's a good point. I don't believe this idea has been proposed before, 
and I guess Ben didn't have it in mind when he implemented BB RAID or when 
he proposed to remove it.

>>> I evaluated BoxBackup and set it up before this part of the 
>>> documentation changed.  Anyhow, there's also a contradictory sentence 
>>> later on in the same URL:
>>>
>>> "NOTE Running the server in non-RAID mode has not been tested as 
>>> extensively as in RAID file mode."
>>
>> Strictly speaking, in my mind, this is not contradictory as it doesn't 
>> say that userland RAID is better or recommended, just more tested.
>
> Yes, exactly.  And that was also why I choose to setup the soft-raid 
> solution.  Increased possibilities for security, and better tested.

I don't think the "better tested" part is particularly true any more. 
While all the unit tests do use RAID, I don't think that any users use it 
in production.

>> However I think it may no longer be true. I suspect that few people are 
>> using the userland RAID feature in production. If anyone except David 
>> is, please speak up!
>
> I would also be interested in hearing others experiences as well!  If 
> I'm the only one, I agree, it's not much point in continuing this 
> support in BoxBackup.  Then I would need figure out another way how to 
> solve this.  I will not continue on this path if soft-raid disappears 
> for sure in BoxBackup.

As you have a good use case for it, I am not planning to remove it in the 
near future. However I would be interested in thinking about better ways 
to implement this, such as at the OS level. I do think it would be more 
efficient, not less, to implement this at the block level in the OS rather 
than in Box.

I'm also planning to implement S3 client support in Box Backup fairly 
soon, and I expect that most users will move to that as it frees them from 
the need to ever buy more disks or take their systems offline for disk 
upgrades. Unless we can find a good way to support userland RAID on top of 
S3, I expect that these code paths will diverge significantly and you may 
find that fewer users use libraidfile at all.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer |
\__/_/_/_//_/___/ | We are GNU : free your mind & your software |