From 88fe497643d9a4e58b42eb2a3f2fa87645b82fb6 Mon Sep 17 00:00:00 2001 From: aidan46 Date: Tue, 10 Dec 2024 16:14:42 +0800 Subject: [PATCH 1/2] docs: Add deal scraper RFC --- rfcs/004-deal-scraper.md | 42 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 rfcs/004-deal-scraper.md diff --git a/rfcs/004-deal-scraper.md b/rfcs/004-deal-scraper.md new file mode 100644 index 00000000..506b20f8 --- /dev/null +++ b/rfcs/004-deal-scraper.md @@ -0,0 +1,42 @@ +# RFC: On-chain deal scraper + +## Abstract + +This RFC proposes the design and functionality of an application to map `piece_cids` to their respective owners, identified by [Peer ID][1]s, in the Polka-Storage ecosystem. +Leveraging the market pallet and storage provider pallet within the Polka-Storage network, the application will extract, correlate, and maintain relationships between [AccountID][2]s, [Peer ID][1]s, deal IDs, and `piece_cids`. +The resulting mapping will facilitate efficient identification of deal owners and their associated data pieces in the network, providing a robust index for querying storage information. + +## Motivation + +In the Polka-Storage ecosystem, it is crucial to track the ownership and storage relationships of data pieces (`piece_cids`) efficiently. +Existing mechanisms in the storage and market pallets provide the underlying data but lack a comprehensive and centralized mapping of these relationships. +This application addresses the need for such mapping by integrating information across pallets to maintain a database that links [Peer ID][1]s with `piece_cids`. +This functionality enhances the usability and scalability of the Polka-Storage network. + +## Design + +The application will interface with the Polka-Storage network, specifically interacting with the storage provider pallet and the market pallet. +It will check storage values and subscribe to relevant events to construct and maintain a comprehensive mapping database. + +To initiate the mapping process, the application will utilize the `StorageProviders` storage value from the storage provider pallet. +This value links [AccountID][2]s to [Peer ID][1]s, which are integral to identifying storage providers in the network. + The application will subscribe to events emitted upon the registration of new storage providers, `StorageProviderRegistered`, ensuring that the mapping is continuously updated with the latest registration data. + +Once the association between [AccountID][2]s and [Peer ID][1]s is established, the application will utilize the `SectorDeals` storage value in the market pallet to match deal IDs with storage providers. +By subscribing to events emitted for new deal IDs, `DealsPublished`, the application will dynamically update its database with ongoing deal information. +This layer of mapping bridges the storage providers with their associated deals, forming the basis for the next stage. + +Finally, the application will leverage the `Proposals` storage value in the market pallet to associate deal IDs with their corresponding `piece_cids`. +This step completes the mapping by correlating the data pieces (`piece_cids`) with the owners ([Peer ID][1]s) through their deals. The application will also subscribe to events related to new proposals, ensuring the mapping remains accurate and up to date. + +The database maintained by the application will serve as a centralized source of truth for the mapping of [Peer ID][1]s to `piece_cids`. +By continuously monitoring and reacting to events in the Polka-Storage network, the application ensures real-time synchronization with the underlying storage and market data. + +## Conclusion + +This application enhances the Polka-Storage ecosystem by providing a vital indexing functionality that maps [Peer ID][1]s to `piece_cids`. +By integrating data across pallets and dynamically responding to network events, it offers a reliable and scalable solution for tracking storage ownership and deal relationships. +This tool will help clients with precise and up-to-date information, strengthening the overall infrastructure of Polka-Storage. + +[1]: https://docs.libp2p.io/concepts/fundamentals/peers/#peer-id +[2]: https://docs.rs/frame-system/latest/frame_system/pallet/trait.Config.html#associatedtype.AccountId From 61764981c5f0650c1e48b01c1cb1daa931cbad82 Mon Sep 17 00:00:00 2001 From: aidan46 Date: Fri, 13 Dec 2024 17:22:30 +0800 Subject: [PATCH 2/2] docs: Improve deal scraper RFC --- rfcs/004-deal-scraper.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/rfcs/004-deal-scraper.md b/rfcs/004-deal-scraper.md index 506b20f8..04f878d0 100644 --- a/rfcs/004-deal-scraper.md +++ b/rfcs/004-deal-scraper.md @@ -2,15 +2,15 @@ ## Abstract -This RFC proposes the design and functionality of an application to map `piece_cids` to their respective owners, identified by [Peer ID][1]s, in the Polka-Storage ecosystem. +This RFC proposes the design and functionality of an application to map content CIDs to their respective owners, identified by [Peer ID][1]s, in the Polka-Storage ecosystem. Leveraging the market pallet and storage provider pallet within the Polka-Storage network, the application will extract, correlate, and maintain relationships between [AccountID][2]s, [Peer ID][1]s, deal IDs, and `piece_cids`. The resulting mapping will facilitate efficient identification of deal owners and their associated data pieces in the network, providing a robust index for querying storage information. ## Motivation -In the Polka-Storage ecosystem, it is crucial to track the ownership and storage relationships of data pieces (`piece_cids`) efficiently. +In the Polka-Storage ecosystem, it is crucial to track the ownership and storage relationships of data pieces efficiently. Existing mechanisms in the storage and market pallets provide the underlying data but lack a comprehensive and centralized mapping of these relationships. -This application addresses the need for such mapping by integrating information across pallets to maintain a database that links [Peer ID][1]s with `piece_cids`. +This application addresses the need for such mapping by integrating information across pallets to maintain a database that links [Peer ID][1]s with content CIDs. This functionality enhances the usability and scalability of the Polka-Storage network. ## Design @@ -26,15 +26,15 @@ Once the association between [AccountID][2]s and [Peer ID][1]s is established, t By subscribing to events emitted for new deal IDs, `DealsPublished`, the application will dynamically update its database with ongoing deal information. This layer of mapping bridges the storage providers with their associated deals, forming the basis for the next stage. -Finally, the application will leverage the `Proposals` storage value in the market pallet to associate deal IDs with their corresponding `piece_cids`. -This step completes the mapping by correlating the data pieces (`piece_cids`) with the owners ([Peer ID][1]s) through their deals. The application will also subscribe to events related to new proposals, ensuring the mapping remains accurate and up to date. +Finally, the application will leverage the `Proposals` storage value in the market pallet to associate deal IDs with their corresponding `piece_cids` and map them to the correct storage provider account. +This step completes the mapping by correlating the data pieces (`piece_cids`) with the owners ([Peer ID][1]s) through their deals. The database maintained by the application will serve as a centralized source of truth for the mapping of [Peer ID][1]s to `piece_cids`. By continuously monitoring and reacting to events in the Polka-Storage network, the application ensures real-time synchronization with the underlying storage and market data. ## Conclusion -This application enhances the Polka-Storage ecosystem by providing a vital indexing functionality that maps [Peer ID][1]s to `piece_cids`. +This application enhances the Polka-Storage ecosystem by providing a vital indexing functionality that maps [Peer ID][1]s to content CIDs. By integrating data across pallets and dynamically responding to network events, it offers a reliable and scalable solution for tracking storage ownership and deal relationships. This tool will help clients with precise and up-to-date information, strengthening the overall infrastructure of Polka-Storage.