ETL Network

Screenshots

Problem Statement

This project is a new decentralised network of nodes that work together to facilitate an ETL (extract, transform, load) network.The network is capable of:extracting event data from different sources - such as Polygon, Ethereum or Streamr Networktransforming said data using publicly accessible Javascript functions developed and deployed to IPFS by anyone configuring their own "ETL pipeline"and finally, loading processed data as transactions submitted to different integrated destination blockchains.The goal here is to offer a more transparent alternative to traditional oracle networks for producing on-chain data & transactions from off-chain data sources.The problem with traditional oracle networks is that they require a centralised API to expose some pre-processed data for a Smart Contract to retrieve data from. This means that for a Smart Contract to move assets based on off-chain data, there is still a centralised authority governing & controlling the movement of said assets through the data that is exposed to the integrated Smart Contract.The ETL network proposes a solution to this. The ETL network accepts event data directly from integrated Blockchains and the Streamr Network, and processes said event data through publicly readable and immutable Javascript functions to deduce instructions for destination blockchains, as well as storing the outcome of processed data for future query-ability. The network achieves consensus about the outcome of processed data and the result transactions to be submitted before any transactions are submitted.The benefits of using such a platform include:Web3 Automation, whereby events from different smart contracts as well as events from Web & Mobile Apps can yield transactions submitted to other smart contractsCross-chain data syndication - such that events can be extracted from Polygon, Ethereum, etc. and result in transactions submitted to Solana, Ethereum, etc.Transparency around how event data is processed through publicly accessible and immutable Javascript functions (aka "pipeline contracts")Turning Web and Mobile events transported through the Streamr Network into transactions submitted to Polygon, Ethereum, etc.Storing all processed data onto the Arweave Blockchain for accessibility and eventual query-ability

Solution

The ETL network uses the Kyve Blockchain as a mechanism for achieving consensus by data validation. The Kyve Blockchain is a new (unsponsored) blockchain that enables the operation and coordination of a customised node network, referred to as a "Kyve Pool", responsible for archiving blockchain data onto the Arweave Blockchain in a decentralised manner.The ETL Network is a "hacked" version of this node network operated by the Kyve blockchain.Rather than archiving blockchain data from a single blockchain source, as is the intent with a "Kyve Pool", the ETL Network is "hacked" tointegrate with many sourcesread "Pipeline Configurations" from a Polygon Smart Contractload Javascript functions that are registered inside of the "Pipeline Configuration" as an IPFS (sponsored) Addressprocess event data extracted from the different sources through these Javascript functionsUse the response of the Javascript functions to construct transactions submitted to destination blockchain(s)During this hackathon, we integrated Polygon (sponsored), Ethereum, and Streamr Network (sponsored) as our sources of event data. Polygon is a widely used, fairly costed Layer 2 blockchain that we believe is a perfect candidate for integration due to its adoption. Streamr Network allows for the transport of event data from Web, Mobile and Server applications to a decentralised network of nodes, such that these events can be processed by the ETL Network.Polygon was also integrated as the only destination blockchain. To produce a single valid transaction submitted to Polygon by a network of nodes, we deployed a multi-sig wallet (Gnosis Safe) Smart Contract to facilitate the collection of "votes" (aka signatures) by each Node operating on the network. This way a single transaction can be submitted to any Polygon Smart Contract as instructed by the response of the Javascript Functions that process the event data, executed on the ETL Network.I'm impressed by how well the team understood the technology architecture and instructions for the development of the nodes.