More documentation

docs/concepts/architecture.md

@@ -5,4 +5,12 @@ title: "Architecture"
 
 # Architecture
 
-TODO
+In the below diagram you can see the different layers of library and its dependencies.
+
+- The user can choose to call the High Level Type Safe API which is based on types from Scala case classes (in green) or the Low Level Type Safe API (in blue) which is based on DynamoDB specific abstractions like `AttributeValue`. 
+- **High Level Type Safe API** has a dependency on [ZIO Schema](https://zio.dev/zio-schema/) for compile time type information that is used to automatically generate codecs and in turn calls the ...
+- **Low Level API** for query creation and execution, and which has in turn has a dependency on [ZIO AWS](https://zio.dev/zio-aws/), which in turn has a dependency on the AWS Java SDK.
+
+![architecture diagram](architecture.png)
+
+Although there are two APIs, the High Level API is the recommended way to interact with the library as it provides a type safe way to interact with DynamoDB together with a huge amount of boilerplate reduction.

docs/concepts/high-level-api.md

@@ -3,5 +3,63 @@ id: high-level-api
 title: "High Level API"
 ---
 
-# High Level API
-TODO
+The high level API relies on two mechanisms to provide type safety and reduce boilerplate code:
+
+1. Automatic ZIO Schema derivation for case classes
+2. Semi automatic generation of `ProjectionExpression`'s for case classes and sealed traits 
+
+## ZIO Schema derivation
+
+The High Level API methods rely on a ZIO Schema for a particular type being in implicit scope. This is achieved using the `DeriveSchema.gen` macro. Internally codecs are automatically generated for the case classes based on the meta data provided by the `Schema`'s.
+
+```scala
+final case class Person(id: String, name: String)
+object Person {
+  implicit val schema: Schema.CaseClass2[String, String, Person] = DeriveSchema.gen[Person]
+}
+```
+
+## Projection Expression generation
+
+The automated generation of `ProjectionExpression`'s is achieved using the `ProjectionExpression.accessors` method. For classes this method generates a `ProjectionExpression` for each field. For sealed traits it generates a `ProjectionExpression` for each child.
+
+```scala
+final case class Person(id: String, name: String)
+object Person {
+  implicit val schema // ...
+
+  val (id, name) = ProjectionExpression.accessors[Person]
+}
+```
+
+## Using a `ProjectionExpression` as a springboard to creating further expressions
+
+In the above example `Person.id` and `Person.name` are `ProjectionExpression`s automatically generated by the `ProjectionExpression.accessors` method.
+They are used as a springboard for creating further type safe APIs eg 
+- `Person.id === "1"` creates a `ConditionExpression`
+- `Person.name.set("Smith")` creates an `UpdateExpression`
+- `Person.id.partitionKey === "1"` creates a primary key expression
+
+## `DynamoDBQuery` CRUD methods
+
+There are also type safe query creation methods in the `DynamoDBQuery` companion object such as `get`, `put`, `update`, `deleteFrom`, `queryAll` and all these take expressions as arguments. So taking our example further we can see how all these APIs can be used together to create a type safe CRUD queries: 
+
+```scala
+final case class Person(id: String, name: String)
+object Person {
+  implicit val schema: Schema.CaseClass2[String, String, Person] = DeriveSchema.gen[Person]
+  val (id, name) = ProjectionExpression.accessors[Person]
+}
+
+val table = "person-table"
+val person = Person("1", "John")
+for {
+  _ <- DynamoDBQuery.put(table, person).where(!Person.id.exists).execute
+  found <- DynamoDBQuery.get(table)(Person.id.partitionKey === "1").execute.absolve
+  _ <- DynamoDBQuery.update(table)(Person.id.partitionKey === "1")(
+    Person.name.set("Smith")
+  ).execute
+  _ <- ZIO.debug(found == person) // true
+} yield ()
+```
+

docs/concepts/low-level-api.md

@@ -3,5 +3,24 @@ id: low-level-api
 title: "Low Level API"
 ---
 
-# Low Level API
-TODO
+The low level API provides low level query creation and execution while still offering a large reduction in boilerplate. It's is based one to one on DynamoDB abstractions and the surface area consists of:
+- `AttrMap` which is a convenience container with automatic conversion between Scala values and `AttributeValue`'s. It also has type aliases of `PrimaryKey` and `Item` (see reference section for more details)
+- `$` function for creating untyped `ProjectionExpression`s eg `$("person.address.houseNumber")`
+- query methods in the `DynamoDBQuery` companion object that all contain the word `Item` such as `getItem`, `putItem`, `updateItem`, `deleteItem`, `queryAllItem`. 
+- Expressions - most of these query methods take expressions as arguments. Once we have a `ProjectionExpression` via the dollar function we can use it as a springboard to create further expressions such as `ConditionExpression`, `UpdateExpression` and `PrimaryKeyExpression`:
+  - `ConditionExpression`'s - `$("name") === "John"`
+  - `UpdateExpression`'s - `$("name").set("Smith")`
+
+However there are some caveats to using the Low Level API:
+- It is not type safe - subtle runtime errors can occur - for example if there are typos in the field names, or if incompatible types are used in expressions.
+- Serialization and deserialization of case classes is the responsibility of the user - this is usually a major burden.
+
+An example of a complete low level query is shown below:
+```scala
+for {
+  _ <- DynamoDBQuery.putItem("person-table", Item("id" -> "1", "name" -> "John", "age" -> 42)).execute
+  maybeFound <- DynamoDBQuery.getItem("person-table")(PrimaryKey("id" -> "1")).execute
+  _ <- DynamoDBQuery.updateItem("person-table")(PrimaryKey("id" -> "1"))($("name").set("Smith") + $("age").set(21)).execute
+  _ <- DynamoDBQuery.deleteItem("person-table")(PrimaryKey("id" -> "1")).execute
+} yield ()
+```

docs/concepts/transactions.md

@@ -1,7 +1,6 @@
 ---
 id: transactions
 title: "DynamoDB Transactions"
-sidebar_label: "Transactions"
 ---
 
 [Transactions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transaction-apis.html) are as simple as calling the `.transaction` method on a `DynamoDBQuery`. As long as every component of the query is a valid transaction item and the `DynamoDBQuery` does not have a mix of get and write transaction items. A list of valid items for both types of queries is listed below.

docs/guides/cheat-sheet.md

@@ -3,8 +3,6 @@ id: cheat-sheet
 title: "High Level API Cheat Sheet"
 ---
 
-# High Level API Cheat Sheet
-
 Note this guide assumes the reader has some basic knowledge of [AWS DynamoDB API](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html).
 
 Assuming the below model

docs/guides/codec-customization.md

@@ -3,8 +3,6 @@ id: codec-customization
 title: "Codec Customization"
 ---
 
-# Default encoding
-
 ## Sealed trait members that are case classes
 
 ```scala
@@ -120,7 +118,7 @@ Mapping for `Box(Amber(42))` would be `Map(trafficLightColour -> Map(String(red_
 
 ## Recommendations
 For greenfield development it is recommended to use:
-- the default encoding which uses an intermediate map ([see above](#default-encoding)) - (note this mapping does not work with top level sum types as it requires an intermediate map and partition keys must be scalar values) *or*
+- the default encoding which uses an intermediate map (see the above default encoding) - (note this mapping does not work with top level sum types as it requires an intermediate map and partition keys must be scalar values) *or*
 - `@discriminatorName` encoding ([see above](#customising-encodings-via-annotations)) if you would like a more compact encoding - note this *must* be used for top level sum types
 
 

docs/guides/data-modelling.md

@@ -0,0 +1,55 @@
+---
+id: data-modelling
+title: "Data Modelling Limitations"
+---
+
+The High Level API relies heavily on ZIO Schema and inherits some of it's limitations, namely:
+
+- A maximum of **22** fields per case class 
+- Deep Object-Oriented hierarchies with abstract classes/fields are not supported - only hierarchies one level deep are supported
+
+At first glance these seem limitations seem quite restrictive, however the next sections describe how these can be overcome.
+
+## A maximum of **22** fields per case class
+The high level API uses the Reified Optics feature of ZIO Schema to generate optics for case classes. This feature has a limitation of 22 fields per case class. 
+
+This limitation is something to be aware of when designing your models. In practice however this limitation can be overcome by using nested case classes and in the case of deeply nested hierarchies, by using product and sum types (see section below). 
+
+## Deep OO Style hierarchies are not supported - modelling data using Product and Sum Types
+Deep Object-Oriented like hierarchies with abstract classes/fields are not supported - only hierarchies one level deep are supported - again this is a limitation of ZIO Schema - however these limitations can be overcome by using product and types rather than inheritance.
+
+
+```scala
+object OopStyle {
+  // ensure that all concrete classes have id field
+  sealed trait Invoice {
+    def id: Int
+  }
+  // (1) Intermediate abstraction with abstract fields - not supported by ZIO Schema
+  sealed trait Billed                                                 extends Invoice {
+    def amount: Double
+  }
+  final case class BilledMonthly(id: Int, amount: Double, month: Int) extends Billed
+  final case class BilledYearly(id: Int, amount: Double, year: Int)   extends Billed
+  final case class PreBilled(id: Int, count: Int)                     extends Invoice
+}
+
+// FP style modelling uses pure data (product and sum types) rather than inheritance hence avoiding classes like (1)
+object FPStyle {
+  sealed trait BilledBody
+  final case class BilledMonthly(month: Int) extends BilledBody
+  final case class BilledYearly(year: Int)   extends BilledBody
+
+  sealed trait InvoiceBody
+  // (3) extract product refactoring
+  final case class Billed(amount: Double, billedBody: BilledBody) extends InvoiceBody
+  final case class PreBilled(count: Int)                          extends InvoiceBody
+
+  // (2) extract product refactoring
+  final case class Invoice(int: Int, body: InvoiceBody)
+}
+```
+
+By using the FP approach to modelling we reduce the size of the concrete classes. 
+
+For brevity the above examples do not show the full integration with ZIO Schema - [for a full example see this IT test](https://github.com/zio/zio-dynamodb/blob/series/2.x/dynamodb/src/it/scala/zio/dynamodb/TypeSafeApiAlternateModeling.scala).

docs/guides/testing.md

@@ -0,0 +1,6 @@
+---
+id: testing
+title: "Testing ZIO DynamoDB Applications"
+---
+
+TODO

docs/reference/auto-batching-and-parallelisation.md

@@ -0,0 +1,64 @@
+---
+id: auto-batching-and-parallelisation
+title: "Auto batching and parallelisation"
+---
+
+When `DynamoDBQuery`'s are composed either manually via the `zip` combinator or automatically using the `DynamoDBQuery.forEach` function they become eligible for auto-batching and parallelisation in the `execute` method.
+
+```scala
+val batchedWrite1 = DynamoDBQuery.put("person", Person("1", "John", 21))
+        .zip(DynamoDBQuery.put("person", Person("2", "Jane", 22)))
+
+val batchedWrite2 = DynamoDBQuery.forEach(people)(person => put("person", person))
+
+for {
+  _ <- batchedWrite1.execute // PutItem operations will be batched
+  _ <- batchedWrite2.execute // PutItem operations will be batched
+} yield ()
+```
+
+## Rules for determining auto-batching vs parallelisation behaviour
+
+The rules for determining whether a query is auto-batched are determined by what query types are eligible for batching in the AWS API. The [BatchWriteItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html) operation can only deal with `PutItem` and `DeleteItem` operations. Furthermore, for both of these operations - condition expressions are not allowed. The AWS [BatchGetItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html) operation is used for batching `GetItems`'s .
+
+So the rules are follows: 
+
+- A query only qualifies for auto-batching  if it passes the following criteria: 
+  - The query is a `PutItem` or `DeleteItem` operation (`put` and `deleteFrom` in the High Level API)
+    - The query does not have a condition expression
+  - The query is a `GetItem` operation (`get` in the High Level API)
+    - The query's `projections` list contains the primary key - this is required to match the response data to the request. Note all fields are included by default so this is only a concern if you explicitly specify the projection expression.  
+- If a query does not qualify for auto-batching it will be parallelised automatically
+
+## Maximum batch sizes for `BatchWrireItem` and `BatchGetItem`
+
+When using the `zip` or `forEach` operations one thing to bear in mind is the maximum number of queries that the `BatchWrireItem` and `BatchGetItem` operations can handle:
+
+- `BatchWriteItem` can handle up to **25** `PutItem` or `DeleteItem` operations
+- `BatchGetItem` can handle up to **100** `GetItem` operations
+
+If these are exceeded then you will get a runtime AWS error. For further information please refer to the AWS documentation linked above.
+
+## Automatic retry of unprocessed batch items/keys
+
+Note that both the AWS `BatchWriteItem` and `BatchGetItem` operations return a list of unprocessed items/keys. If this list is non-empty then the operation are retried automatically by the ZIO DynamoDB library. 
+
+If retries do not succeed in eliminating the unprocessed items/keys then the whole batch is failed with a `BatchError.WriteError`/`BatchError.GetError` - both of which will contain a list of the unprocessed items/keys. 
+
+The default retry policy is:
+
+```scala
+Schedule.recurs(3) && Schedule.exponential(50.milliseconds)
+```
+
+This can be overridden by using the `withRetryPolicy` combinator:
+
+```scala
+batchedWrite2.withRetryPolicy(myCustomRetryPolicy).execute
+```
+
+## Integration Batching with ZIO Streams
+
+For examples of how to integrate batching with ZIO Stream please see the utility functions `batchWriteFromStream` and `batchGetFromStream` in the `zio.dynamodb` package.
+These functions take care of details mentioned above such as managing the maximum batch sizes and can also be used as examples for writing your own custom batched streaming operations.
+```scala

docs/reference/dynamodb-json.md

@@ -0,0 +1,4 @@
+---
+id: dynamodb-json
+title: "DynamoDB JSON Module"
+---

docs/reference/dynamodb-query.md

@@ -3,6 +3,19 @@ id: dynamodb-query
 title: "DynamoDBQuery"
 ---
 
-# DynamoDBQuery
+TODO
 
-TODO
+interaction with DynamoDBExecutor?
+
+- put
+- get
+- update
+- delete
+
+- Execution
+
+- combinators - maybe a table of them?
+  - where
+  - whereKey
+  - limit 
+  - etc

docs/reference/error-handling.md

@@ -0,0 +1,6 @@
+---
+id: error-handling
+title: "Error Handling"
+---
+
+TODO

docs/reference/hi-level-api/creating-models/collection-field-traversal.md

@@ -0,0 +1,4 @@
+---
+id: collection-field-traversal
+title: "Collection Field Traversal"
+---

docs/reference/hi-level-api/creating-models/index.md

@@ -0,0 +1,57 @@
+---
+id: index
+title: "Creating Models"
+sidebar_label: "Creating Models"
+---
+
+The High Level API provides automatic serialization and deserialization of Scala case classes to and from DynamoDB types.
+This is done by requiring that an implicit ZIO Schema instance is in scope for the case class. This schema instance is 
+generated semi-automatically by using the ZIO Schema `DeriveSchema.gen[A]` - placing this in the companion object of 
+the case class ensures that this implicit it is automatically in scope.
+
+```scala
+final case class Person(email: String, hobbies: Map[String, List[String]], dob: Instant)
+object Person {
+  implicit val schema: Schema.CaseClass3[String, Map[String, List[String]], Instant, Person] = 
+    DeriveSchema.gen[Person]
+}
+```
+
+This semi-automatically derived schema is used to automatically generate codecs for the case class (in the `Codecs` object)
+to perform the serialization and deserialization to and from DynamoDB types.
+
+All standard Scala types are supported by the codecs, as well as nested case classes and collections. Note that where 
+possible Scala types are mapped to corresponding DynamoDB types, for example Scala `Map`'s and `Set`'s are mapped to 
+native DynamoDB types.
+
+Scala Scalar Types | Native DynamoDB Type | Notes
+------------|----------------------|--------------
+`Unit`              | NULL                    
+`String`            | S                    
+Numeric Types       | N                    
+`Collection[byte]`  | B                    | Any Scala collection type of byte is serialized to a DynamoDB binary type 
+`Boolean`           | BOOL                 
+`java.time.*`       | S                    | There is no native date/time support. Instant is serialized to a string in ISO-8601 format
+`java.util.UUID`    | S                    | There is no native UUID support
+`java.util.Currency`| S                   | There is no native Currency support
+
+Note in the below table that types `A`, `K` and `V` can be collections or case classes as well as scalar types.
+
+Scala Collection Types | Native DynamoDB Type | Notes
+------------|----------------------|--------------
+`Option[A]`           |                      | None is represented by the absence of the field in the DynamoDB item
+`List[A]`             | L                    |
+`Set[String]`         | SS                   |
+`Set` of numeric type | NS                   |
+`Set` of binary type  | BS                   |
+`Set[A]` of other type| L                    | If type is not a string or a numeric then a list is used
+`Map[String, A]`      | M                    | if key type is a string then a native Map is used
+`Map[K, V]`           | L                    | otherwise a list of tuple of key value pair is used
+
+
+
+
+
+
+
+

docs/reference/hi-level-api/creating-models/optics.md

@@ -0,0 +1,4 @@
+---
+id: optics
+title: "Nested Field Traversal"
+---