From a636085be57e333b97ae9eb835a3a2e6eb0bd44f Mon Sep 17 00:00:00 2001 From: Yusuke Tsutsumi Date: Sun, 14 Apr 2024 23:15:07 -0700 Subject: [PATCH 1/3] feat(pagination): re-importing from AIP-158 AEP-158 forked from the general aip.dev, whose guidance diverged a bit from gooogle.aip.dev. This re-imports those, resulting in the following set of changes: - switching typescript examples to proto (with placeholder for openapi). - minor wording fixes. --- aep/general/0158/aep.md.j2 | 191 +++++++++++++++---------------------- aep/general/0158/aep.yaml | 1 + 2 files changed, 77 insertions(+), 115 deletions(-) diff --git a/aep/general/0158/aep.md.j2 b/aep/general/0158/aep.md.j2 index 43184e94..edc91ce0 100644 --- a/aep/general/0158/aep.md.j2 +++ b/aep/general/0158/aep.md.j2 @@ -1,100 +1,103 @@ # Pagination -APIs often need to provide collections of data, most commonly in the -[List][aip-132] standard method. However, collections can often be arbitrarily -sized, and also often grow over time, increasing lookup time as well as the -size of the responses being sent over the wire. Therefore, it is important that -collections be paginated. +APIs often need to provide collections of data, most commonly in the [List][] +standard method. However, collections can often be arbitrarily sized, and also +often grow over time, increasing lookup time as well as the size of the +responses being sent over the wire. Therefore, it is important that collections +be paginated. ## Guidance -Operations returning collections of data **must** provide pagination _at the -outset_, as it is a [backwards-incompatible change](#backwards-compatibility) -to add pagination to an existing method. +RPCs returning collections of data **must** provide pagination _at the outset_, +as it is a [backwards-incompatible change](#backwards-compatibility) to add +pagination to an existing method. -```typescript +{% tab proto %} + +```proto // The request structure for listing books. -interface ListBooksRequest { +message ListBooksRequest { // The parent, which owns this collection of books. // Format: publishers/{publisher} - parent: string; + string parent = 1 [ + (google.api.field_behavior) = REQUIRED, + (google.api.resource_reference) = { + child_type: "library.googleapis.com/Book" + }]; // The maximum number of books to return. The service may return fewer than // this value. // If unspecified, at most 50 books will be returned. // The maximum value is 1000; values above 1000 will be coerced to 1000. - maxPageSize: bigint; + int32 page_size = 2; // A page token, received from a previous `ListBooks` call. // Provide this to retrieve the subsequent page. // // When paginating, all other parameters provided to `ListBooks` must match // the call that provided the page token. - pageToken: string; + string page_token = 3; } // The response structure from listing books. -interface ListBooksResponse { +message ListBooksResponse { // The books from the specified publisher. - books: Book[]; + repeated Book books = 1; // A token that can be sent as `page_token` to retrieve the next page. // If this field is omitted, there are no subsequent pages. - nextPageToken: string; + string next_page_token = 2; } ``` -- Request definitions for collections **should** define an - `int32 max_page_size` field, allowing users to specify the maximum number of - results to return. - - If the user does not specify `max_page_size` (or specifies `0`), the API +{% tab oas %} + +**Note:** OAS example not yet written. + +{% endtabs %} + +- Request messages for collections **should** define an integer (`int32` for +protobuf) `page_size` field, allowing users to specify the maximum number of +results to return. + - If the user does not specify `page_size` (or specifies `0`), the API chooses an appropriate default, which the API **should** document. The API **must not** return an error. - - If the user specifies `max_page_size` greater than the maximum permitted by - the service, the service **should** coerce down to the maximum permitted - page size. - - If the user specifies a negative value for `max_page_size`, the API - **must** return a `400 Bad Request` error. - - The service **should** the number of results requested, unless the end of - the collection is reached. - - However, occasionally this is infeasible, especially within expected time - limits. In these cases, the service **may** return fewer results than the - number requested (including zero results), even if not at the end of the - collection. -- Request definitions for collections **should** define a `string page_token` + - If the user specifies `page_size` greater than the maximum permitted by the + API, the API **should** coerce down to the maximum permitted page size. + - If the user specifies a negative value for `page_size`, the API **must** + send an `INVALID_ARGUMENT` error. + - The API **may** return fewer results than the number requested (including + zero results), even if not at the end of the collection. +- Request messages for collections **should** define a `string page_token` field, allowing users to advance to the next page in the collection. - - If the user changes the `max_page_size` in a request for subsequent pages, - the service **must** honor the new page size. - - The user is expected to keep all other arguments to the operation request - the same; if any arguments are different, the API **should** send a - `400 Bad Request` error. + - The `page_token` field **must not** be required. + - If the user changes the `page_size` in a request for subsequent pages, the + service **must** honor the new page size. + - The user is expected to keep all other arguments to the RPC the same; if + any arguments are different, the API **should** send an `INVALID_ARGUMENT` + error. - The response **must not** be a streaming response. -- Services **may** support using page tokens across versions of a service, but - are not required to do so. -- Response definitions for collections **must** define a +- Response messages for collections **should** define a `string next_page_token` field, providing the user with a page token that may be used to retrieve the next page. - - The field containing pagination results **should** be the first field - specified. It **should** be a repeated field containing a list of resources - constituting a single page of results. + - The field containing pagination results **should** be the first field in + the message and have a field number of `1`. It **should** be a repeated + field containing a list of resources constituting a single page of results. - If the end of the collection has been reached, the `next_page_token` field **must** be empty. This is the _only_ way to communicate "end-of-collection" to users. - If the end of the collection has not been reached (or if the API can not - determine in time), the service **must** provide a `next_page_token`. -- Response definitions **may** include a `string next_page_url` field - containing the full URL for the next page. -- Response definitions for collections **may** provide an `int32 total_size` - field, providing the user with the total number of items in the list. - - This total **may** be an estimate (but the API **should** explicitly - document that). - -[rfc-8288]: https://tools.ietf.org/html/rfc8288 + determine in time), the API **must** provide a `next_page_token`. +- Response messages for collections **may** provide an integer (`int32` for +protobuf) `total_size` field, providing the user with the total number of items +in the list. + - This total **may** be an estimate. If so, the API **should** explicitly + document that. ### Skipping results -The request definition for a paginatied operation **may** define an -`int32 skip` field to allow the user to skip results. +The request definition for a paginated operation **may** define an integer +(`int32` for protobuf ) `skip` field to allow the user to skip results. The `skip` value **must** refer to the number of individual resources to skip, not the number of pages. @@ -111,9 +114,9 @@ If a `skip` value is provided that causes the cursor to move past the end of the collection of results, the response **must** be `200 OK` with an empty result set, and not provide a `next_page_token`. -### Opacity +### Page Token Opacity -Page tokens provided by services **must** be opaque (but URL-safe) strings, and +Page tokens provided by APIs **must** be opaque (but URL-safe) strings, and **must not** be user-parseable. This is because if users are able to deconstruct these, _they will do so_. This effectively makes the implementation details of your API's pagination become part of the API surface, and it becomes @@ -133,32 +136,21 @@ authorization to the underlying resources, and authorization **must** be performed on the request as with any other regardless of the presence of a page token. -### Expiring page tokens - -Many services store page tokens in a database internally. In this situation, -the service **may** expire page tokens a reasonable time after they have been -sent, in order not to needlessly store large amounts of data that is unlikely -to be used. It is not necessary to document this behavior. - -**Note:** While a reasonable time may vary between services, a good rule of -thumb is three days. - -### Consistency +### Page Token Expiration -When discussing pagination, consistency refers to the question of what to do if -the underlying collection is modified while pagination is in progress. The most -common way that this occurs is for a resource to be added or deleted in a place -that the pagination cursor has already passed. +Many APIs store page tokens in a database internally. In this situation, APIs +**may** expire page tokens a reasonable time after they have been sent, in +order not to needlessly store large amounts of data that is unlikely to be +used. It is not necessary to document this behavior. -Services **may** choose to be strongly consistent by approximating the -"repeatable read" behavior in databases, and returning exactly the records that -exist at the time that pagination begins. +**Note:** While a reasonable time may vary between APIs, a good rule of thumb +is three days. -### Backwards compatibility +## Backwards compatibility -Adding pagination to an existing operation is a backwards-incompatible change. -This may seem strange; adding fields to interface definitions is generally -backwards compatible. However, this change is _behaviorally_ incompatible. +Adding pagination to an existing RPC is a backwards-incompatible change. This +may seem strange; adding fields to proto messages is generally backwards +compatible. However, this change is _behaviorally_ incompatible. Consider a user whose collection has 75 resources, and who has already written and deployed code. If the API later adds pagination fields, and sets the @@ -167,48 +159,17 @@ now is only getting the first 50 (and does not know to advance pagination). Even if the API set a higher default limit, such as 100, the user's collection could grow, and _then_ the code would break. -For this reason, it is important to always add pagination to operations -returning collections _up front_; they are consistently important, and they can -not be added later without causing problems for existing users. +For this reason, it is important to always add pagination to RPCs returning +collections _up front_; they are consistently important, and they can not be +added later without causing problems for existing users. **Warning:** This also entails that, in addition to presenting the pagination fields, they **must** be _actually implemented_ with a non-infinite default value. Implementing an in-memory version (which might fetch everything then paginate) is reasonable for initially-small collections. -## Implementation +[list]: ./list -Page tokens **should** be versioned independently of the public API, so that -page tokens can be used with any version of the service. - -The simplest form of a page token only requires an offset. However, offsets -pose challenges when a distributed database is introduced, so a more robust -page token needs to store the information needed to find a "logical" position -in the database. The simplest way to do this is to include relevant data from -the last result returned. Primarily, this means the resource ID, but also -includes any other fields from the resource used to sort the results (for the -event where the resource is changed or deleted). - -This information is from the resource itself, and therefore might be sensitive. -Sensitive data **must** be encrypted before being used in a page token. -Therefore, the token also includes the date it was created, to allow for the -potential need to rotate the encryption key. - -This yields the following interface, which **may** be base64 encoded and used -as a page token: - -```typescript -interface PageTokenSecrets { - // The ID of the most recent resource returned. - lastId: string; - - // Any index data needed, generally 1:1 with the fields used for ordering. - indexData: Buffer[]; - - // When this token was minted. - createTime: Date; -} -``` +## Changelog -**Note:** This section does not preclude alternative page token implementations -provided they conform to the guidelines discussed in this document. +- **2024-04-14**: Imported from https://google.aip.dev/158. \ No newline at end of file diff --git a/aep/general/0158/aep.yaml b/aep/general/0158/aep.yaml index 128130de..500b34a5 100644 --- a/aep/general/0158/aep.yaml +++ b/aep/general/0158/aep.yaml @@ -3,6 +3,7 @@ id: 158 state: approved slug: pagination created: 2019-02-18 +updated: 2024-04-14 placement: category: design-patterns order: 60 From 39468ff5f520794dbd8b36b19371c5c3bc08bd3b Mon Sep 17 00:00:00 2001 From: Yusuke Tsutsumi Date: Sat, 27 Apr 2024 21:33:09 -0700 Subject: [PATCH 2/3] Apply suggestions from code review Co-authored-by: Richard Frankel --- aep/general/0158/aep.md.j2 | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/aep/general/0158/aep.md.j2 b/aep/general/0158/aep.md.j2 index edc91ce0..5e343e55 100644 --- a/aep/general/0158/aep.md.j2 +++ b/aep/general/0158/aep.md.j2 @@ -8,7 +8,7 @@ be paginated. ## Guidance -RPCs returning collections of data **must** provide pagination _at the outset_, +Methods returning collections of data **must** provide pagination _at the outset_, as it is a [backwards-incompatible change](#backwards-compatibility) to add pagination to an existing method. @@ -29,7 +29,7 @@ message ListBooksRequest { // this value. // If unspecified, at most 50 books will be returned. // The maximum value is 1000; values above 1000 will be coerced to 1000. - int32 page_size = 2; + int32 max_page_size = 2; // A page token, received from a previous `ListBooks` call. // Provide this to retrieve the subsequent page. @@ -68,7 +68,7 @@ results to return. send an `INVALID_ARGUMENT` error. - The API **may** return fewer results than the number requested (including zero results), even if not at the end of the collection. -- Request messages for collections **should** define a `string page_token` +- Request schemas for collections **should** define a `string page_token` field, allowing users to advance to the next page in the collection. - The `page_token` field **must not** be required. - If the user changes the `page_size` in a request for subsequent pages, the From e4c9237c3e447da3b1bce9a25b7fd2b98f6788c0 Mon Sep 17 00:00:00 2001 From: Yusuke Tsutsumi Date: Sat, 27 Apr 2024 21:34:34 -0700 Subject: [PATCH 3/3] addressing more feedback - adding definition for schema - renaming page_size -> max_page_size - normalize verbase to use method vs operation / rpc --- aep/general/0003/aep.md.j2 | 6 ++++++ aep/general/0158/aep.md.j2 | 24 +++++++++++++----------- 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/aep/general/0003/aep.md.j2 b/aep/general/0003/aep.md.j2 index 9a0669be..987de638 100644 --- a/aep/general/0003/aep.md.j2 +++ b/aep/general/0003/aep.md.j2 @@ -99,6 +99,12 @@ Examples of complexities that declarative clients abstract away include: [Terraform][] is an example of such a client. +### Schema + +A schema describes the structure of the request or response of an [API +method](#api-method), or a [resource](#api-resource). It refers both to an +OpenAPI schema as well as a protobuf message. + ### User A human being which is using an API directly, such as with cURL. This term is diff --git a/aep/general/0158/aep.md.j2 b/aep/general/0158/aep.md.j2 index 5e343e55..be3d908f 100644 --- a/aep/general/0158/aep.md.j2 +++ b/aep/general/0158/aep.md.j2 @@ -50,6 +50,9 @@ message ListBooksResponse { } ``` +- The field containing pagination results **should** be the first field in + the message and have a field number of `1`. + {% tab oas %} **Note:** OAS example not yet written. @@ -57,31 +60,30 @@ message ListBooksResponse { {% endtabs %} - Request messages for collections **should** define an integer (`int32` for -protobuf) `page_size` field, allowing users to specify the maximum number of +protobuf) `max_page_size` field, allowing users to specify the maximum number of results to return. - - If the user does not specify `page_size` (or specifies `0`), the API + - If the user does not specify `max_page_size` (or specifies `0`), the API chooses an appropriate default, which the API **should** document. The API **must not** return an error. - - If the user specifies `page_size` greater than the maximum permitted by the + - If the user specifies `max_page_size` greater than the maximum permitted by the API, the API **should** coerce down to the maximum permitted page size. - - If the user specifies a negative value for `page_size`, the API **must** + - If the user specifies a negative value for `max_page_size`, the API **must** send an `INVALID_ARGUMENT` error. - The API **may** return fewer results than the number requested (including zero results), even if not at the end of the collection. - Request schemas for collections **should** define a `string page_token` field, allowing users to advance to the next page in the collection. - The `page_token` field **must not** be required. - - If the user changes the `page_size` in a request for subsequent pages, the + - If the user changes the `max_page_size` in a request for subsequent pages, the service **must** honor the new page size. - - The user is expected to keep all other arguments to the RPC the same; if + - The user is expected to keep all other arguments to the method the same; if any arguments are different, the API **should** send an `INVALID_ARGUMENT` error. - The response **must not** be a streaming response. -- Response messages for collections **should** define a +- Response messages for collections **must** define a `string next_page_token` field, providing the user with a page token that may be used to retrieve the next page. - - The field containing pagination results **should** be the first field in - the message and have a field number of `1`. It **should** be a repeated + - The field containing pagination results **must** be a repeated field containing a list of resources constituting a single page of results. - If the end of the collection has been reached, the `next_page_token` field **must** be empty. This is the _only_ way to communicate @@ -146,9 +148,9 @@ used. It is not necessary to document this behavior. **Note:** While a reasonable time may vary between APIs, a good rule of thumb is three days. -## Backwards compatibility +### Backwards compatibility -Adding pagination to an existing RPC is a backwards-incompatible change. This +Adding pagination to an existing method is a backwards-incompatible change. This may seem strange; adding fields to proto messages is generally backwards compatible. However, this change is _behaviorally_ incompatible.