[RFC] Separated Manifests | Addition of Tree Object

[rydrman]

Background and Problem Space

In some cases, we are seeing manifests reach into the hundreds of megabytes in size. Additionally, there is duplicated data between manifests of spk packages when minor changes happen in one or two sub directories in a large package with thousands of files.

Proposal

SPFS already has an object kind that holds trees. These represent a single directory and are combined together in a manifest to create a full filesystem hierarchy.

Instead of always storing the reference to this complete manifest in layers, we would allow the reference of the root tree object. This means that trees could/would be stored individually as objects in the repo, and that the manifest would/could be loaded incrementally as all trees are accessed.

Other Considerations

• What about a single file with deep structure? This case would create many trees and slow down the loading process because of back-and-forth with the repository. Maybe it's atypical and we don't care, or we have smarter ways to request the entire manifest via gRPC to reduce round-trips when networked.