Mantis Data Solutions | Arweave Integration - Milestone 1: Publish using Arweave Transaction ID - Backend | Round 20

OceanDAO Round 20
#1

Project Name

Mantis Data Solutions

Project Category

Core Software

Proposal Earmark

General

Proposal Description

Retroactive Funding Request

This is a retroactive funding request for 44 hours of work on the Ocean Protocol backend towards Arweave Integration Milestone 1: Publish using Arweave Transaction ID.

This work resulted in backend PRs that enable publishing assets stored in Arweave using only the Arweave transaction ID. This has a couple benefits when compared with publishing assets stored in Arweave using a URL (arweave gateway + tx id):

  1. The published asset is not dependent on a single Arweave Gateway.
  2. The download endpoint won’t leak the transaction ID as the filename.

Note: This proposal does not include: (1) The frontend 1 PRs that will make this feature available in the Ocean Market. (2) the backend components for uploading a file to Arweave. 3rd party tools are still needed to perform this step.

Given that the PRs are not yet reviewed and merged, it is possible that additional backend work will be required.

Request Scope

819×212 16.6 KB

I have worked 69 hours on this Arweave Integration project thus far. This includes design, programming, and Github communication. This does not include time spent preparing proposals, attending core tech working group meetings, or dev support on Discord.

This proposal is requesting funding to cover 44 of these hours that were spent working on the Ocean Protocol backend. This proposal is not requesting funding for the full 69 hours because I am capped at requesting $7500. I intend to submit another proposal in R21.

Of the total 69 hours, 53 were spent on the backend, and 16 were spent on the frontend. Of the 53 that were spent on the backend, I will discount 9 of them because I over-eagerly implemented a suggestion from @kremalicious to generalize the FileType structure, only to later revert the changes because they were incompatible with existing DDOs on-chain. In the future, I will wait for confirmation from @alexcos20 before implementing design changes like this.

Hourly Rate

The requested $7500 is based on an hourly rate is $170/hr. This rate is in line with my R19 proposal that was fully funded.

Effort Scaling

Given that this is a retroactive funding request, there is no way for me to scale my effort down based on the actual amount of funding received. Nonetheless, I will respect the decision of the DAO, and I will not re-request funding for these 44 hours in future rounds. Nor will I seek funding from the Shipyard program for this same work.

Shipyard Proposal

Previously, I submitted a proposal to the Ocean Protocol Shipyard program for the full Arweave Integration, including all three milestones. However, I have been informed that the program is not set up to provide retroactive funding, hence why I am seeking funding from OceanDAO for the work already completed. I believe that doing so invalidates my existing Shipyard proposal. It’s not clear to me yet how the remainder of the project will be funded, but I will not include these 44 hours in any future funding requests/proposals.

Frontend Status Update

I submitted a Figma mockup and a brief video of the proposed publish screen changes to the ocean core team.

I still need to:

  • Remove the URL validation from the Transaction ID text box
  • Wire up the Transaction ID “Validate” button so that it makes the appropriate call to the provider fileinfo endpoint.
  • Add unit & integration tests to ocean.js that publish, download, and run c2d jobs using files stored in Arweave.
  • Update the docs to include instructions for using the new Arweave file publish flow and for how to use 3rd party tools to upload a file to Arweave.

You can see the in-progress PRs here:

Challenges

Staying in sync with the core team has been challenging.

I submitted the initial set of backend PRs for review on 2022/06/20. Sometime after that, the FileType structure in the provider was updated to use inheritance. I agree with these changes, as they will reduce code duplication in future integrations with Filecoin, Chainlink, The Graph, etc. Unfortunately, these changes invalidated parts of the work I had submitted and the work had to be redone, building on the new architecture.

Similarly, the docs repo migrated to use Gitbook meaning that my docs changes had to be redone. Getting the formatting right took longer than I would have liked.

Grant Deliverables

The following PRs have been submitted for review but have not yet been approved and merged so additional work may still be required.

Project Description

Mantis Data Solutions

Mantis Data Solutions is the legal entity wrapper around me, David Hunt-Mateo, also known as “MantisClone” online. I’m a full-stack web3 developer and I love building open-source software that runs on decentralized, permissionless networks.

Past work for OceanDAO:

Current Project: First-class Arweave Integration

Problem

Arweave’s unique value proposition of “permanent storage for a one-time fee” makes it an attractive storage medium for data publishers. By storing in Arweave, they can be certain their data will always be available for consumption without worrying about recurring storage costs that exist in other decentralized options like Filecoin or cloud options like AWS, Azure, or GCP.

Today, it is possible to publish an asset on Ocean Protocol that is stored in Arweave using an Arweave gateway URL. But there are problems with this approach:

  • The download endpoint leaks the Arweave transaction ID as the filename, which uniquely identifies the document and allows the downloader to bypass Ocean Protocol altogether.
  • By specifying the Arweave gateway URL, the asset is now dependent on a single Arweave gateway when in truth, any Arweave gateway would work.
  • The Ocean Market doesn’t make it clear that storing data in Arweave is an option.
  • The data publisher cannot upload their data to Arweave via the Ocean Market and must use a 3rd party interface instead.

Opportunity

Today, Arweave stores 80 TB of data. Arweave has over 66k followers on Twitter and over 15k members across its Discord servers.

Final Product

Solution

I propose a “first-class” integration between the Ocean Protocol and Arweave to be implemented in three phases:

Milestone 1: Publish using Arweave transaction ID

Once complete, data publishers can use a 3rd party interface to upload their data to Arweave, and then publish it in the Ocean Market by providing the Arweave transaction ID instead of a URL.

Milestone 2: Upload to Arweave - Backend

Once complete, data publishers can use ocean.py and ocean.js to upload their data to Arweave and publish it as an asset on the Ocean market in a few programmatic steps.

Milestone 3: Upload to Arweave from the Ocean Market

Once complete, data publishers can copy/paste or drag/drop their data into the Ocean market. The market will upload the file to Arweave and use the resulting transaction ID to publish the asset.

Value Add Criteria

Usage of Ocean

First-class integration with Arweave will drive data publish volume. Data publishers want to use Arweave because it eliminates ongoing storage costs. Instead, there's a single upfront cost to store the data, permanently.

Viability

Chances of success are high - I can leverage my existing knowledge of the Ocean Protocol core tech stack and relationships with the core team to push this project to completion.

Community Engagement

I am deeply engaged with the Ocean Protocol community:

  • I worked at the Ocean Protocol Foundation from April 2021 to June 2022
  • I have attended the core tech working group meetings since June 2022
  • I frequently provide dev support on Discord

Community Value

I believe my project helped galvanize the rising tide of ocean community builders to pursue core tech improvements.

Core Team

David Hunt-Mateo (aka. MantisClone)

Role: Full-stack Web3 Software Developer

Relevant Credentials (e. g):

  • B.S. double major in Computer Engineering and Computer Science from the University of Wisconsin - Madison,
  • CompTIA Security+ ce Certification, Credential ID BP4DZZ1NKK1EYNSP

Github: https://github.com/MantisClone

Linkedin: https://www.linkedin.com/in/david-hunt-mateo/

Background/Experience:

  • Developed Snapshot voting strategies for Arrakis, H2O, and Rari Fuse
  • 1 year at Ocean Protocol Foundation maintaining the ocean.py SDK, aquarius event cache, and provider access proxy. Developed features and tests for Ocean ONDA V4.
  • 3 years at Think-A-Move building a cross-platform, on-device, noise-robust speech recognition system along with desktop and native Android apps showcasing its capabilities.
  • 4 years at Northrop Grumman working on a variety of projects including GUI integration testing, experimental low-latency networking, hardware emulation, and embedded image processing.

Advisors

N/A

Funding Requested
7500

Minimum Funding Requested

Wallet Address
0x1EC1CcEF3e1735bdA3F4BA698e8a524AA7c93274

Mantis Data Solutions | Arweave Integration - Milestone 1: Publish using Arweave Transaction ID - Finish Backend, Progress on Frontend | Round 22
#2

Project submitted deliverables:

Grant Deliverables

The following PRs have been submitted for review but have not yet been approved and merged so additional work may still be required.

#3

Admin:

Hi David, thank you for submitting your completed deliverables. They have been accepted, knowing they include the majority of the work so you can continue submitting proposals and continue moving forward. Cheers.