FileDrive Datasets

Screenshots

Problem Statement

FileDrive Datasets is a platform to effectively connect the huge storage market that Filecoin has built with publishers of public datasets, which will allow anyone to access data they are interested in or publish, manage and maintain their datasets. It has integrated a group of tools developed by FileDrive Team to provide storage service with the compatibility of Cloud Storage and Object-base Storage and better user experience to attract more users. After datasets have been uploaded onto the platform, these data will be aggregated or split into data pieces in specific sizes by Go-Graphsplit, and then stored onto the Filecoin network under the cooperation with Filecoin storage providers.

Solution

Go-Datastore-ClusterGo-Datastore-Cluster is one of the most significant parts of FileDrive's product system, and the storage layer of FileDrive Datasets.It aims to gather distributed key-value datastores to become a cluster to increase the data split function and improve data management. Even only one IPFS peer could take advantage of decentralized storage without the limitation of I/O by a single PC.Go-GraphsplitGo-Graphsplit is a tool for splitting a large dataset into graph slices to make deals in the Filecoin Network, which could generally be 32GiB or 64GiB.It takes advantage of IPLD protocol, follows the Unixfs format data structures. This tool regards the large dataset or its sub-directory as a big graph and cuts it into small graphs that could still keep its file system structure as possible as it used to be.It can streamline data retrieval from the Filecoin network and perfectly match IPFS to support other storage functions.