compose

Problem Statement

Compose is a modern data pipeline that automates a simple but powerful workflow for creating a public, verifiable dataset that maps out the projects and ecosystems that makes up the blockchain industry.The pipeline uses Walrus and Graph Hypergraphs to build an immutable, on-chain record that points to a specific version of the dataset stored on decentralized infrastructure.

Solution

Fetch Data: A containerized Dagster data pipeline is triggered, fetching the latest developer report data from the Electric Capital public repository.Store on Walrus: The JSONL dataset is compressed and then uploaded to the Walrus decentralized storage network, which returns a unique and a blob ID.Pipeline Indexing on HyperGraph: The pipeline calls a TypeScript script that uses the GRC-20 library to write the returned blob ID and a timestamp to a HyperGraph index running on the testnet.geobrowser.io network.