From 2fb8368ef58575ec5ebaacaa6f5bd0f800ce3eab Mon Sep 17 00:00:00 2001 From: NoahMaizels Date: Thu, 21 Sep 2023 00:31:15 +0700 Subject: [PATCH 1/2] Added additional descriptions of the implications of batch utilisation with examples --- .../technology/contracts/postage-stamp.md | 45 ++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/learn/technology/contracts/postage-stamp.md b/docs/learn/technology/contracts/postage-stamp.md index de3ff99bd..ddc17f730 100644 --- a/docs/learn/technology/contracts/postage-stamp.md +++ b/docs/learn/technology/contracts/postage-stamp.md @@ -40,6 +40,17 @@ $$ 2^{(20 - 16)} = 16 \text{ chunks/bucket} $$ +Since we know that one chunk can store 4 kb of data, we can easily calculate the maximum amount of data (due to [the way batches are utilised](/docs/learn/technology/contracts/postage-stamp#batch-utilisation), postage batches may be fully utilised before storing the maximum amount of data) which can be stored by a batch from the `batch depth`: + +$$ +\text{Maximum data in a batch} = 2^{batchdepth} \times \text{4 kb} +$$ + +For a batch of depth 16, for example, the maximum amount of data in kilobytes can be calculated like so: + +$$ +2^{16} \times \text{4 kb} = 65536 \text{ chunks} \times \text{4 kb} = \text{262144 kb} +$$ #### Depth to kb calculator: @@ -87,12 +98,44 @@ As more chunks get uploaded and stamped, the bucket slots will begin to fill. As ### Mutable Batches -Mutable batches use the same hash map structure as immutable batches, however its utilisation works very differently. In contrast with immutable batches, mutable batches are never considered fully utilised. Rather, at the point where an immutable batch would be considered fully utilised, a mutable batch can continue to stamp chunks. However, if any chunk's address lands in a bucket whose slots are already filled, that bucket's counter gets reset, and the new chunk will replace the oldest chunk in that bucket. +Mutable batches use the same hash map structure as immutable batches, however its utilisation works very differently. In contrast with immutable batches, mutable batches are never considered fully utilised. Rather, at the point where an immutable batch would be considered fully utilised, a mutable batch can continue to stamp chunks. However, if any chunk's address lands in a bucket whose slots are already filled, rather than the batch becoming fully utilised, that bucket's counter gets reset, and the new chunk will replace the oldest chunk in that bucket. ![](/img/batches_04.png) Therefore rather than speaking of the number of slots as determining the utilisation of a batch as with immutable batches, we can think of the slots as defining a limit to the amount of data which can be uploaded before old data starts to get overwritten. +### Which Type of Batch to Use + +Immutable batches are suitable for long term storage of data or for data which otherwise does not need to be changed and should never be overwritten, such as records archival, legal documents, family photos, etc. + +Mutable batches are great for data which needs to be frequently updated and does not require a guarantee of immutability. For example, a blog, personal or company websites, ephemeral messaging app, etc. + +The default batch type when unspecified is immutable. This can be modified through the Bee api by setting the `immutable` header with the [`\stamps POST` endpoint](https://docs.ethswarm.org/api/#tag/Transaction/paths/~1transactions~1%7BtxHash%7D/post) to `false`. + + +### Implications for Swarm Users + +Due to the nature of batch utilisation described above, batches are often fully utilised before reaching their theoretical maximum storage amount. However as the batch depth increases, the chance of a postage batch becoming fully utilised early decreases. At at batch depth 23, it is statistically highly unlikely that the batch will be fully utilised/start replacing old chunks before reaching 50% of its theoretical maximum. Therefore, when choosing a batch depth, it is recommended to use a batch depth of 23 or greater, and consider 50% of its theoretical maximum storage amount as the de facto maximum storage amount. + +Let's look at an example to make it clearer. Using the method of calculating the theoretical maximum storage amount [outlined above](/docs/learn/technology/contracts/postage-stamp#batch-depth), we can see that for a batch depth of 23, the maximum amount which can be stored is 33,554,432 kb: + +$$ +2^{23} \times \text{4kb} = \text{33,554,432 kb} +$$ + +Therefore we should use 50% of that value as our de facto storage limit: + +$$ +\text{33,554,432 kb} \times{0.5} = \text{16,777,216 kb} +$$ + +We find that at depth 23, we have a statistical guarantee of being able to store up to 16,777,216 kb or 16.77 gb worth of data. + + +:::info +More information about the expected utilisation levels for different batch depths will be added in the near future. +::: + From 2c7177d4212169041763f06d868d51a0ffea4b42 Mon Sep 17 00:00:00 2001 From: NoahMaizels Date: Thu, 21 Sep 2023 00:59:01 +0700 Subject: [PATCH 2/2] Added more info on stamp batches, changed 'keep your data alive' page title and added redirect, changed redoc search option --- .../{keep-your-data-alive.md => buy-a-stamp-batch.md} | 6 +++--- docs/develop/access-the-swarm/upload-and-download.md | 2 +- docusaurus.config.js | 7 ++++++- sidebars.js | 2 +- 4 files changed, 11 insertions(+), 6 deletions(-) rename docs/develop/access-the-swarm/{keep-your-data-alive.md => buy-a-stamp-batch.md} (98%) diff --git a/docs/develop/access-the-swarm/keep-your-data-alive.md b/docs/develop/access-the-swarm/buy-a-stamp-batch.md similarity index 98% rename from docs/develop/access-the-swarm/keep-your-data-alive.md rename to docs/develop/access-the-swarm/buy-a-stamp-batch.md index 6e2de680e..a0f985e60 100644 --- a/docs/develop/access-the-swarm/keep-your-data-alive.md +++ b/docs/develop/access-the-swarm/buy-a-stamp-batch.md @@ -1,9 +1,9 @@ --- -title: Keep Your Data Alive -id: keep-your-data-alive +title: Buy a Batch of Stamps +id: buy-a-stamp-batch --- -Swarm comprises the sum total of all storage space provided by all of our nodes, called the DISC (Distributed Immutable Store of Chunks). The _right to write_ data into this distributed store is determined by the postage stamps that have been attached. +Swarm comprises the sum total of all storage space provided by all of our nodes, called the DISC (Distributed Immutable Store of Chunks). The _right to write_ data into this distributed store is determined by the [postage stamps](/docs/learn/technology/contracts/postage-stamp) that have been attached. ### Fund your node's wallet. diff --git a/docs/develop/access-the-swarm/upload-and-download.md b/docs/develop/access-the-swarm/upload-and-download.md index a80cf596d..bb5f5b6af 100644 --- a/docs/develop/access-the-swarm/upload-and-download.md +++ b/docs/develop/access-the-swarm/upload-and-download.md @@ -28,7 +28,7 @@ In order to upload your data to swarm, you must agree to burn (spend) some of your xBZZ to signify to storer and fowarder nodes that this content is valued. Before you proceed to the next step, you must buy stamps! See this guide on how to [purchase an appropriate batch of -stamps](/docs/develop/access-the-swarm/keep-your-data-alive). +stamps](/docs/develop/access-the-swarm/buy-a-stamp-batch). ### Upload diff --git a/docusaurus.config.js b/docusaurus.config.js index d730d8e9a..e06e238fe 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -41,6 +41,10 @@ module.exports = { to: '/docs/develop/dapps-on-swarm/pss', from: '/docs/dapps-on-swarm/pss', }, + { + to: '/docs/develop/access-the-swarm/buy-a-stamp-batch', + from: '/docs/develop/access-the-swarm/keep-your-data-alive', + }, ], }, ], @@ -308,7 +312,8 @@ module.exports = { requiredPropsFirst: true, noAutoAuth: true, expandDefaultServerVariables: true, - scrollYOffset: 60 + scrollYOffset: 60, + searchMaxDepth: 10 }, }, }, diff --git a/sidebars.js b/sidebars.js index dd7b09f19..4290509f3 100644 --- a/sidebars.js +++ b/sidebars.js @@ -111,13 +111,13 @@ module.exports = { label: 'Access the Swarm', items: [ 'develop/access-the-swarm/introduction', + 'develop/access-the-swarm/buy-a-stamp-batch', 'develop/access-the-swarm/upload-and-download', 'develop/access-the-swarm/upload-a-directory', 'develop/access-the-swarm/direct-upload', 'develop/access-the-swarm/store-with-encryption', 'develop/access-the-swarm/host-your-website', 'develop/access-the-swarm/syncing', - 'develop/access-the-swarm/keep-your-data-alive', 'develop/access-the-swarm/pinning', ], collapsed: false