User:Scott Parlane/DesignStudy

Project (Out of Band Synchronisation)

Description

Provide a solution for performing offsite back ups, where directly transferring the data is impossible or incredibly expensive, however regular transport between locations is avaliable and at least 1 operation computer is transferred between sites.

Goals/Requirements

• Synchronise data using out of band data transfer.
• Support extremely slow, high capacity links. (80+GiB with 1 week latency)
• Support multiple transfer agents.

Constraints

• WORM data assumed (allowing for rename/move/etc)

Initial Design

• Make use of [1] infrastructure (espically networking code).
• Use the code generator git://git.scottnz.com/generator.git/ to generate OO code from a design spec.
• Split system into 3 parts: metadata store, data servers, and transfer agents.

Warning: Notation used below is similar, but not the same as, the notation used by generator, this is intentional, until more graphical representation can be created.

MetaData Store

Tracks the entire tree of each data storage server and

• Location (Pointer(Char):Name, ArrayObjectPaired(Server,Directory):Servers,ArrayObject(Transporter):Agents)
• Directory (ArrayObject(Directory):SubDirs, ArraryObject(File):Contents, Pointer(Char):Name)
• File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)
• Server (Pointer(Char):Name, ArrayObject(Location):Locations)
• Transporter(Pointer(Char):UUID, ArrayObject(File):Carrying)

Data Store

Provides long term storage for all of the tracked files

• Location (Pointer(Char):Name, Array(Directory):Root, Pointer(Char):Path)
• Directory (ArrayObject(Directory):SubDirs, ArraryObject(File):Contents, Pointer(Char):Name)
• File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)

Transporter

Provides a temporary storage area for transporting data between data stores

• Path (Pointer(char):Name, ArrayObject(File):Contents)
• File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)