[Box Backup-dev] Re: Soft-RAID support

[Ben Summers]

David Sommerseth wrote:
> Chris Wilson wrote:
>> Hi David,
>>
>> On Sat, 6 Jun 2009, David Sommerseth wrote:
>>
>>> Chris Wilson wrote:
>>>> Support for it was never finished (no recovery procedure), it is  
>>>> pretty
>>>> limited (only supports RAID 5 and three devices) and it was  
>>>> written at a
>>>> time when OS/software and hardware RAID were not as ubiquitous or  
>>>> well
>>>> supported as they are now.
>>>
>>> I would be willing, with some guidance to look into such a tool,  
>>> if that
>>> is the main criteria for dropping this support.
>>
>> That would definitely be very helpful, thanks in advance. You can  
>> read
>> the encrypted objects (which are reconstructed successfully) and then
>> rewrite them, which will reestablish the redundant copies.
>
> I'll grab the code soon after the holiday season is over, and poke  
> into
> this.  I'd consider the program as a stand alone program somehow,  
> which
> will do the recreation in a way which you suggest.  As a brief quick  
> idea
> of how I could imagine it:
>
>     root@host # bbackrecover --source-dir1 /path/to/origdata_part1 \
> 			     --source-dir2 /path/to/origdata_part2 \
>  	  		     --recover-dir /path/to/recovered_part3
>
> Only the missing part would then be recovered to the given directory  
> in
> --recover-dir.  Not sure though, if it would be needed to write data  
> to
> part1 and part2 directories in addition to the already mentioned  
> part3.
> Does this approach seem sensible?


Box Backup is written as a set of interdependent libraries. There's  
lots of generic code for building UNIX client/server applications,  
which make it easy to deal with sockets, SSL, communication protocols  
and so on. And it avoids depending on anything other than OpenSSL.

So I had imagined that the raidfile library would be used by other  
things. (That's why there's a separate raidfile.conf file.) In this  
scheme, there was going to be another daemon, raidfiled. Stuff which  
used the raidfile library would just write the file, then offload the  
conversion to the three stripes to raidfiled. You'll note the sequence  
of events in writing a raidfile lends itself nicely to this scheme.

I had then imagined that errors would be notified to raidfiled, which  
would do the necessary fixing and alerting. Plus, of course, automatic  
rebuild if a stripe is lost.

Rebuilding a raidfile should be as easy as opening it in read mode,  
then copying it to a raidfile of the same name in write mode. The  
difficult bit will be determining whether or not it needs recovery. A  
trivial approach will be to read everything and see if errors are  
reported, and this may indeed be the best way.

Personally, I don't use the raidfile stuff to stripe any more, but  
store everything on a ZFS volume with mirroring or RAIDZ. Feels much  
safer to me. As Chris says, this kind of thing wasn't available when I  
started the project.

Ben