Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PC1/PC2: being able to disable TreeR building #27

Open
vmx opened this issue Aug 21, 2023 · 3 comments
Open

PC1/PC2: being able to disable TreeR building #27

vmx opened this issue Aug 21, 2023 · 3 comments

Comments

@vmx
Copy link
Contributor

vmx commented Aug 21, 2023

When looking at the PC2 API, it looks like you cannot run PC2 without building TreeR. It would be great to have an option that only generates the TreeC and the sector key. The sector key is what currently is called the sealed-file.

Background on why this would be nice. When doing data sectors (as opposed to CC sectors), it could be that the data files are not accessible from the machine that runs the PC2. Therefore, you would run the PC1 process as you currently do and disable the TreeC building. You would then use the sealed-file as sector key and move that to where the data file is. You'd the run the encoding step and TreeC building there, the existing tools/tree_r.cpp to do that.

I know that building TreeC is just a small fraction of the work done during PC1/PC2, but not doing things that are not needed are always a win.

This request is not urgent, but it will eventually be needed. I though I open the issue early on, so that you can schedule it into your work stream easily.

@simonatsn
Copy link
Contributor

Currently tree-c and tree-r are constructed concurrently since for CC sectors and when the data is present on the sealing machine they utilize the same data. This is important in those cases since the data movement is a significant part of the cost.

In PC2 tree-c is much more expensive than tree-r so in the scenario where the data is remote the potential savings from skipping tree-r is minimal.

We'll add this one to the list for future updates.

@vmx
Copy link
Contributor Author

vmx commented Aug 21, 2023

Currently tree-c and tree-r are constructed concurrently

Oh that's interesting. I though they would be built sequentially. But thinking about it, it makes sense (given the on disk data layout). Then indeed the TreeR costs are pretty small. I guess you would have a little more memory on the GPU available to do the TreeC, but that probably doesn't really matter.

@vmx
Copy link
Contributor Author

vmx commented Sep 1, 2023

On a side-channel we agreed on adding a option to not write the TreeR tree files to disk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants