User:Scott Parlane/DesignStudy
From CSSEMediaWiki
(Difference between revisions)
| Line 7: | Line 7: | ||
* Support extremely slow, high capacity links. (80+GiB with 1 week latency) | * Support extremely slow, high capacity links. (80+GiB with 1 week latency) | ||
* Support multiple transfer agents. | * Support multiple transfer agents. | ||
| − | + | === Constraints === | |
| + | * WORM data assumed (allowing for rename/move/etc) | ||
== Initial Design == | == Initial Design == | ||
* Make use of [http://www.4underscore.net/] infrastructure (espically networking code). | * Make use of [http://www.4underscore.net/] infrastructure (espically networking code). | ||
* Use the code generator git://git.scottnz.com/generator.git/ to generate OO code from a design spec. | * Use the code generator git://git.scottnz.com/generator.git/ to generate OO code from a design spec. | ||
* Split system into 3 parts: metadata store, data servers, and transfer agents. | * Split system into 3 parts: metadata store, data servers, and transfer agents. | ||
| + | Warning: Notation used below is similar, but not the same as, the notation used by generator, this is intentional, until more graphical representation can be created. | ||
| + | === MetaData Store === | ||
| + | Tracks the entire tree of each data storage server and | ||
| + | * Location (Pointer(Char):Name, ArrayObjectPaired(Server,Directory):Servers,ArrayObject(Transporter):Agents) | ||
| + | * Directory (ArrayObject(Directory):SubDirs, ArraryObject(File):Contents, Pointer(Char):Name) | ||
| + | * File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1) | ||
| + | * Server (Pointer(Char):Name, ArrayObject(Location):Locations) | ||
| + | * Transporter(Pointer(Char):UUID, ArrayObject(File):Carrying) | ||
| + | === Data Store === | ||
| + | Provides long term storage for all of the tracked files | ||
| + | * Location (Pointer(Char):Name, Array(Directory):Root, Pointer(Char):Path) | ||
| + | * Directory (ArrayObject(Directory):SubDirs, ArraryObject(File):Contents, Pointer(Char):Name) | ||
| + | * File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1) | ||
| + | === Transporter === | ||
| + | Provides a temporary storage area for transporting data between data stores | ||
| + | * Path (Pointer(char):Name, ArrayObject(File):Contents) | ||
| + | * File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1) | ||
Revision as of 04:31, 2 August 2010
Contents |
Project (Out of Band Synchronisation)
Description
Provide a solution for performing offsite back ups, where directly transferring the data is impossible or incredibly expensive, however regular transport between locations is avaliable and at least 1 operation computer is transferred between sites.
Goals/Requirements
- Synchronise data using out of band data transfer.
- Support extremely slow, high capacity links. (80+GiB with 1 week latency)
- Support multiple transfer agents.
Constraints
- WORM data assumed (allowing for rename/move/etc)
Initial Design
- Make use of [1] infrastructure (espically networking code).
- Use the code generator git://git.scottnz.com/generator.git/ to generate OO code from a design spec.
- Split system into 3 parts: metadata store, data servers, and transfer agents.
Warning: Notation used below is similar, but not the same as, the notation used by generator, this is intentional, until more graphical representation can be created.
MetaData Store
Tracks the entire tree of each data storage server and
- Location (Pointer(Char):Name, ArrayObjectPaired(Server,Directory):Servers,ArrayObject(Transporter):Agents)
- Directory (ArrayObject(Directory):SubDirs, ArraryObject(File):Contents, Pointer(Char):Name)
- File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)
- Server (Pointer(Char):Name, ArrayObject(Location):Locations)
- Transporter(Pointer(Char):UUID, ArrayObject(File):Carrying)
Data Store
Provides long term storage for all of the tracked files
- Location (Pointer(Char):Name, Array(Directory):Root, Pointer(Char):Path)
- Directory (ArrayObject(Directory):SubDirs, ArraryObject(File):Contents, Pointer(Char):Name)
- File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)
Transporter
Provides a temporary storage area for transporting data between data stores
- Path (Pointer(char):Name, ArrayObject(File):Contents)
- File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)
