Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make segment blocks uniquely addressable #164

Open
bruth opened this issue Jul 1, 2015 · 0 comments
Open

Make segment blocks uniquely addressable #164

bruth opened this issue Jul 1, 2015 · 0 comments

Comments

@bruth
Copy link
Contributor

bruth commented Jul 1, 2015

This follows how segments are implemented where each segment represents a current snapshot or delta of a log (composed of a set of blocks containing the facts). Segments point to the segment they are based on (also referred to as the parent) as well as the next segment to follow (which is usually the same as base).

To enable reuse across segments, blocks should be addressable independent of the segment they were originally created for. This enables new segments to reference existing blocks that have not changed between revisions. For the standard log-based index (ordered by transaction), blocks will never need to be replaced since this is the natural ordering. For sorted indexes, any block that comes before the new facts will need to be reindexed. This enables reuse of existing blocks that have not changed and requires only creating a few new blocks for subsequent segments.

More formally, this is known as a persistent data structure where copies or revisions of a value internally point to existing data rather than physically copying the data.

The high-level requirements for this are:

  • Add a Block struct type which defines a Next field for pointing to the next block in the sequence.
  • Add a Head field to the Segment structure that points to the first block in the sequence.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant