research: polka storage fetching

[jmg-duarte]

Fetching files from the storage provider isn’t trivial because we need to unseal sectors. To make this efficient the original implementation leverages the IPLD graph — https://spec.filecoin.io/#section-systems.filecoin_nodes.repository.ipldstore

Questions that need to be answered:

• How does retrieval work in FC?
  • At a monetary level — e.g. should clients need to pay?
  • At the software level — e.g. how can we retrieve data from sealed sectors?
  • Do we need extra programs for the client?

The original research document discusses caching, but I don't think this is something we should be worrying about right now.

One reference is Lassie:

• https://github.com/filecoin-project/lassie

---

Implementation Suggestions

For the IPLD/Blockstore, we can review:

• https://github.com/eigerco/blockstore
• https://github.com/filecoin-project/dagstore

As for the fetching protocol, Filecoin uses Graphsync given it integrates with the IPLD graph. We may be able to leverage the following libraries

• https://github.com/retrieval-markets-lab/rs-graphsync
• https://github.com/eigerco/beetswap

---

Retrieval Markets

• While we can implement free retrievals and call it a day, there’s mention of markets in the original report so we need to have something ready. I believe the simpler way is to have the user pay the storage provider to retrieve the file and this payment happens as the file is sent over.
• The SP can create challenges that the user posts on chain and for the retrieval to proceed the SP must observe the proofs. There’s no incentive for the SP to retrieve their own files since the client is the one paying for it. This approach has a big downside which is — it introduces a BUNCH of overhead, so the blocks being transferred need to be worth it.

Videos on the topic:

• https://www.youtube.com/watch?v=tyXwki8cBlg
• https://www.youtube.com/watch?v=BglpOHe4zPo