Ocean Market Data Locker: Automated Filecoin Archival + IPFS Data Services

Ocean Market Data Locker: Automated Filecoin Archival + IPFS Data Services

Part 1 - Proposal Submission

Name of Project

Coral Market

Proposal in one sentence

Grant funds are requested to integrate automated decentralized file storage workflows for IPFS and Filecoin with Ocean Market data publishing features!

Description of the project and what problem is it solving

Ocean Market does not currently support data storage and retrieval on decentralized file storage networks such as IPFS and Filecoin. We propose adding automated Filecoin archival/retrieval and IPFS pinning services using an Estuary node as an integrated component in the Ocean Market Architecture. This includes adding a drag and drop “Data Locker” feature to interact with the Estuary node API.

Motivation

Thousands of petabytes of data on human health, economic activity, social dynamics, and scientific observations of the universe and our impact on it are siloed in legacy institutional web infrastructure.

Key challenges for unlocking scientific data include workflow gaps, infrastructural capacities, and cultural inertia, such as:

  • Expensive ingress/egress fees with traditional cloud storage
  • Insufficient tooling for dataset management, preprocessing, and archival
  • Lack of easy-to-use interoperable workflows & protocols for sharing data
  • Needs for dataset provenance that ensure requested content is the content that is received
  • Institutional compliance and regulatory protocols that gate-keep sensitive data based on academic credentials
  • Cultural inertia for laboratory procedures and protocols
  • No rewards for sharing data, enhanced risk of being “scooped”

The emergence of peer-to-peer data storage and standards for decentralized identifiers (DIDs) makes it possible to permanently establish a public records archive in a common web infrastructure accessible to all, regardless of professional/academic status, nationality, language, or age, while respecting the intrinsic self-sovereignty of data providers.

Our team is collaborating with Protocol Labs and Textile to build open source dataset processing and archival tools for archiving up to 250TB of Open Science Data on the decentralized file storage network, Filecoin, free of charge as part of the Filecoin+ service .

Opscientia plans to establish the largest collection of high quality open source data that can be found on Web3 - a decentralized data commons that preserves critical knowledge for future generations via sustainable Web 3 incentive loops.

Our mission is to make fundamental scientific observations and insights open to global citizens that are united by a vision for collective scientific discovery. A key missing piece of this vision is a data marketplace that allows curious netizens to search, find, and execute computation on datasets defined with standard specifications that support interoperable workflows.

Grant Deliverables

January (R13)

  • [ ] Finalize project integration brief for a Drag and drop “Web3 Data Locker” that pre-publishes data to decentralized file storage
  • [ ] Data Locker + Profile UI Mock-Up
  • [ ] Onboard project-dedicated React dev

Which category best describes your project?

  • Build / improve core Ocean software

Our primary goal is to empower the scientific community to share, analyze, and review data with web3 tools. To reflect this goal, we propose the number of scientific research objects that are published to the decentralized web as a KPI.

The addition of the Filecoin/IPFS archival feature will provide new revenue avenues for others that utilize the tech stack and choose to run their own data provider node. Examples include charging for archival, pinning, or retrieval.

Our team has a track record of 12+ successfully completed proposals in OceanDAO and serves as a mentor to many projects in the community. Our track record shows that we are the right team to implement and execute.

What is the final product?

In this round, we will begin work on first class integration of the Ocean Market with Filecoin and IPFS. Our plan is to introduce a “data locker” feature that allows users to pre-publish their data on decentralized file storage networks. Users can then select from assets in their data locker when creating a new data token on ocean. Data is downloaded directly from the IPFS gateway, or retrieved from Filecoin archival if needed, when the corresponding data token is purchased.

How does this project drive value to the overall Ocean ecosystem?

We seek to implement a core integration with Ocean Market and Ocean Provider to provide decentralized file storage as a first class service. This may potentially unlock new revenue streams for market operators as well as enhance the level of decentralization for deployed apps.

Funding Requested

USD$20,000

Proposal Wallet Address

0x057c9a25f1302484Bb34C9CEB6d3BC69Bd319e01
(opscientia.eth)

Have you previously received an OceanDAO Grant?

Yes

Team Website

https://opsci.io

Twitter Handle

@opscientia

Discord

Opscientia

Email address

contact@opscientia.com

Current Country of Residence

Opscientia Labs Inc. is a California registered company.

Part 2: Team

Project Lead

Shady El Damaty , M.Sc., Ph.D.

Team:

Kinshuk Kashyap, Fellow

Caleb Tuttle, Fellow

Part 3: Proposal Details

Project Deliverables - Category

A Pull Request will be made on the Ocean Market front end GitHub - oceanprotocol/market: 🧜‍♀️ THE Data Market and Ocean Data Provider back-end GitHub - oceanprotocol/provider: REST API for provider of data services

We commit to working with Ocean core developers to merging the PR, following software quality best practices.

Yes

Software Overview:

Users interact with the Data Locker UI embedded within the Ocean Market. A simple drag and drop results in an HTTP request to an external server running Estuary, a hybrid IPFS gateway + Filecoin Auction node.

Estuary has built in API access for creating collections, adding to collections, managing PINs, and preparing data for a Filecoin storage auction. React code for an example upload appears below.

class Example extends React.Component {
  upload(e) {
    e.persist();
    console.log(e.target.files);
    
    const formData  = new FormData();
    formData.append("data", e.target.files[0]);
    
    // NOTE
    // This example uses XMLHttpRequest() instead of fetch
    // because we want to show progress. But you can use
    // fetch in this example if you like.
    const xhr = new XMLHttpRequest();
    
    xhr.upload.onprogress = (event) => {
      this.setState({ 
        loaded: event.loaded, 
        total: event.total 
      });
    }
    
    xhr.open(
      "POST", 
      "https://api.estuary.tech/content/add"
    );
    xhr.setRequestHeader(
      "Authorization", 
      "Bearer REPLACE_ME_WITH_API_KEY"
    );
    xhr.send(formData);
  }

  render() {
    return (
      <React.Fragment>
        <input type="file" onChange={this.upload.bind(this)} />
        <br />
        <pre>{JSON.stringify(this.state, null, 1)}</pre>
      </React.Fragment>
    );
  }
}

This code will upload data to the Estuary node, and pin it locally to IPFS. An internal database is updated at the same time and the file is prepared (converted into a serialized DAG, or “CAR” file) for auction and a deal is made to be stored across 6 miners. Architecture details appear below.

Note that the current API requires a key to use the public service offered by ARG, the maintainers of Estuary. Users interested in this service can apply for an API key through Protocol Labs, or run their own estuary node. This use case may be appropriate for independent data providers that are running their own software and seek to automatically archive or serve their own data via IPFS or charge users access to archival.

The Ocean Market Web UI will be updated to support the new functionality. We plan to update the profile page with a Data Locker stats section dedicated specifically to storage deals and data uploads. This first variation will be a simple implementation with just key overview information.

We will also add a new tab to the Ocean Market to allow users drag and drop functionality. Users may also choose to archive a file to Filecoin directly using a CID of an existing IPFS pinned file.

Project Deliverables - Roadmap

Any prior work completed thus far?

  • Exhaustive technical review of decentralized file storage services can be found on our blog.
  • We have tested and deployed datasets stored with the Estuary API using our own Ocean Market fork running on GAIA-X testnet. This work has revealed critical bottlenecks and compatibility issues to be worked out such as file size, bandwidth, time to resolve IPFS links, and slow retrieval from Filecoin networks.
  • We have written low-level pipelines for submitting auctions to Filecoin, providing us with key insight into the data archival pipeline.

What is the project roadmap?

  • Finalize project integration brief (circa 17/1/22)
  • Data Locker + Profile UI Mock-Up (circa 21/1/22)
  • Onboard new React dev for this project (circa 31/1/22)
  • Market Front-End API Integration with Estuary (circa 28/2/22)
  • Perpetuate Storage Metadata + Stats to Ocean Provider (28/2/22)
  • Stress Test data uploading, work with Ocean Core Team to complete PR (ongoing)

Team’s future plans and intentions

We plan to request funding from OceanDAO to complete the development work for the month of January. We expect to return for funding in February. We may continue development on this project pending successful results and positive community feedback.

2 Likes

Hi @hebbianloop ,

Thank you for submitting your proposal for R13!

I have reviewed your proposal and would like to thank you for your participation inside of the Ocean Ecosystem!

Usage of Ocean - Data Locker is build one of the closest proposals to a core-tech earmark by building IPFS Gateway integration on Ocean Market via a service (Estuary) built to service the OP tech stack.

Viability - Shady has been leading core engineering activities for a while inside of OceanDAO. I believe his outcomes are being successfully delivered, and if this project is like previous ones, I look forward to OP gaining native Filecoin support.

Community active-ness — Shady is an extremely active DAO member, and although I do not know his team, I believe has been oboarding a broad range of contributors via his continuous outreach activities.

Adding value to the community — This project build directly on top of OP core tech. They are basically trying to deliver native IPFS support.

Based on the reasons above, I am in support of your project.

I look forward to continuing providing support and feedback to your project, and you can expect to receive a positive vote from me during the upcoming voting period.

All the best!
-Idiom

1 Like

Hello @hebbianloop,

Please notice that we are updating the funding process for R13 and there are important things for you to know. We will be deploying an update to the Ocean Website, where you will be able to claim your grant on your own time.
Funding the smart contract may take ~2 days after Voting Ends. There will still be a funding deadline of 2 weeks to claim your grant.
You will need a web3 wallet + Airtable to complete the process. Grant requests via request.finance will be deprecated.

You can find all instructions here: https://github.com/oceanprotocol/oceandao/wiki/Request-Grant-Guidelines
You will have until January 10th to update your proposal with all required fields.

Please let us know if you experience any issues or bugs. You can also submit tickets to Github and ping us inside of ocean-dao-engineering for any help. Thank you!

1 Like

Thank you all for checking in!

We’ve gone through and made sure all of the fields are present in the proposal:

  • Project lead full name: Shady El Damaty
  • Email address: contact@opscientia.com
  • Your country: USA
  • Description of grant reason: Grant funds are requested to integrate automated decentralized file storage workflows for IPFS and Filecoin with Ocean Market data publishing features!
  • Funding requested (USD): USD$20,000
  • ETH Wallet address: 0x057c9a25f1302484Bb34C9CEB6d3BC69Bd319e01
1 Like

I think this is a very important piece of the puzzle to make it possible for retail data owners - like myself - to participate in the new data economy.

You have my support!

1 Like

Congratulations to Ocean Market for winning a grant in R13!

Really looking forward to your core-tech contributions and integrating IPFS into the Ocean Protocol stack!

All the best,
-Idiom

1 Like

Hello @hebbianloop,

As part of the Project Guiding - Working Group [PG-WG], We would like to congratulate on your sustained interest in the OceanDAO and for winning the R13 grant.

We also encourage you to be part of our PG-WG Guiding group as an Attendee as well as being part of guiding the newcomers of PG-WG and OceanDAO. Say hello to us at https://discord.gg/kujC2QK9 #Project-Guiding and refer to the pinned messages to find more information regarding the WG.

You could also be part of async discussions by interacting on the Project-Guiding Discord channel

Regards,

Prakash

1 Like

The project integration brief draft and associated assets can be found here. We are currently onboarding a React dev and searching for a Dev Ops engineer to test server side implementations of a data provider node running a full Estuary node side-by-side with Ocean Provider. Major challenges involve identifying correct logic and required execution overhead for executing API calls to IPFS gateway and Filecoin Auction nodes (Brokers). Work is on-going and can be monitored at github.com/opscientia/desci-storage and github.com/opscientia/coral

Thank you for submitting your Deliverable Checklist @hebbianloop ! Airtable is now updated and your Proposal is set to Complete.

All the best!

1 Like