Skip to content

Latest commit

 

History

History
40 lines (28 loc) · 1.33 KB

NOTES.md

File metadata and controls

40 lines (28 loc) · 1.33 KB
Raw

AWS Services

S3

  • High durability.
  • Low cost (orders-of-magnitude less than any other storage option on AWS).
  • Keyspace partitioning by prefix (e.g. pseudo-directories).
  • Sorted keyspace retrieval.

DynamoDB

  • Conditional puts and updates -- allows for compare-and-set.
  • Ordered traversal via indexes.

Design Ideas

Naive Approaches

  • Implement crux's KV store on S3, and use a Moberg TxLog atop that. Use SQS with S3 notifications to pause a polling event consumer. Based on the "standalone" topology in crux.

  • Implement an atomic counter on DynamoDB, and use this to push events into S3, indexed by event ID generated by this counter. Again use SQS to pause a polling event consumer.

Trees in S3 + DynamoDB tx-log

  • Store transacted docs in DynamoDB, with sortable, increasing RANGE keys.
  • Once the tx-log grows large enough, flush the tx-log into a large B-tree stored as objects in S3. This can be a persistent data structure, so old segments just stay as they are.

Challenges

  • Still need an atomic counter for tx-ids in DynamoDB (unless there is a different approach that can give us good ordering if docs, but I don't think there is).
  • Would like a blocking notification when new txes are added to the log (right now it just polls DynamoDB).
    • Possibly use DynamoDB streams -> lambda -> SNS?