kamu-lilypad

Screenshots

Problem Statement

This project puts kamu.dev into a bacalhau container and runs it by calling a lilypad v2 from the cli. It demonstrates that sql streaming can be performed through a lilypad module to help mange the provenance and veracity of a living dataset pulling from theoretically multiple sources. It takes a manifest from an external api, pulls it into kamu and than pushes it as a dag-pb to ipfs.

Solution

A docker image that pulls ipfs and kamu is created, one provides a kamu manifest that instructions from where to pull data and how to store it in sql, once it pulls the data one uses the kamu system ipfs add<dataset>method to add the sql database into a ipfs dab-pb. One than sends the root cid into the lilypad outputs folder, pinning the dag