research: polka storage indexer

[jmg-duarte]

polka-index — not to be confused with the crate in the repo

The polka-index is a network-indexer, similar to the IPNI — https://docs.filecoin.io/storage-providers/architecture/network-indexer

This task consists of:

• researching the necessary components for the indexer
• how it should look like — architecture & stack -wise
• creating a tracker with subtasks

---

Some notes (not rules, more like, ideas to possibly follow or stray away from)

I don't see a reason for the indexer not to be a separate component from the server — i.e. an indexer should be able to:

• Start up & connect to the chain
• Query all existing storage providers
• (connection magic)
• Ask them all what do they have on the network

This implies that the server will need to be extended with an API to share what they're storing.

---

Some questions:

• Having the SPs have PeerIDs allows them to change IP addresses with less effort, as the discovery requires bootstrap nodes, we should also have an answer/reasoning to the question — who should be a bootstrap node?

---

Some considerations:

• Storage providers may have an indexing service but they are not required to.
• Storage providers may refuse connection to an indexing service implying they’re “semi-private”.
• Storage providers may (i.e. ideally) gossip information between each other too.
• Indexing services gossip all information between them.

---

Library comparison

	libp2p	iroh	waku	(notes)
docs	🟢	🟡	🔴	its really hard to figure out what waku has or not
prod ready	🟢	🟡	🟢	iroh may be but there are no big projects using it
language support	🟢	🟢	🟡	rust & js (rust is unclear for waku, there are bindings but thats it)
protocols	🟢	🟡	❓	waku doesnt even advertise them
maturity	🟢	🔴	🟡	some people on the iroh team were part of the filecoin team, others have worked on libp2p directly & iroh's original product was an ipfs node — they know what they’re doing
protocols	🟢	🟡	❓	http & websockets
discovery	🟢	❓	❓	iroh definitely has this but not quite clear, much less for waku