Providing structure between spec, status, and top-level fields.

[toumorokoshi]

Although usually we look at relatively incremental guidance that improves API design, I think we should look toward larger changes that significantly improve the UX of APIs. To that end, I want to propose that the AEPs provide a recommendation around clearly delineated user-settable, server-settable, and first-class fields for a resource.

The TLDR is something like this, for a theoretical VM on a public cloud:

{
   "path": "/resource/path",
   // properties are all user-settable, and the server may not update these.
   "properties": {         
         "instance-type": "e.4xlarge",
   },
  // fields in status are not user-settable. The server generates these values, and they may mutate 
  // based on the runtime state of the resource.
  // custom methods *may* impact these (e.g. a restart custom method).
   "status": {
      "state": "RUNNING",
   }
}

The idea is that we explicitly namespace the properties of the resource based on ownership, as well as nest the resource-specific properties under objects to allow a uniform top-level fields.

This type of requirement is heavy-handed, but comes with several advantages:

• clearly disambiguates between what are user-settable values, and server-settable.
• allows room for the AEP schema to grow, adding new top-level fields as necessary that may mean specific things (e.g. a uniform way to retrieve resource state, or specify views).

Beyond the user-friendliness of this kind of separation, there are other advantage as well:

• differentiation between user-settable and server-settable values, along with top-level values, can all be included in an API platform that provides these features for free, and strictly enforces them at an organization.
• clients such as Terraform and Kubernetes can generate much better clients, since those declarative / IaC tools often need to differentiate between spec and state fields due to reading state and applying changes if the resource state has changed from a local copy.

I recognize this isn't going to actionable for any existing APIs - to that end, we could consider adding these as a "should", with some way to annotate that the resource matches the schema and requirements.

This isn't a particularly novel idea either - e.g. Kubernetes has a very similar convention: https://kubernetes.io/docs/concepts/overview/working-with-objects/

cc @mkistler

[hudlow]

The idea of wrapping resources in a standard envelope is one of those persistent ideas that pops up over and over again for HTTP/JSON APIs (yes, Kubernetes, but also JSON:API and Cloud Foundry and others before it). But it never gains traction, and I think there are some pretty good reasons for that.

It's very attractive to try to divvy up properties into the categories that seem most fundamental, but I think there are some fatal problems with the idea. One is that what seem like the obvious categories to split properties into tend to vary across contexts — so any decision you make is likely to make your standard awkward and undesirable across projects and over time.

It also raises the minimum complexity of a resource and doubles the depth-hierarchy of resource graphs. And, it constrains backward-compatible evolution of resources in unnecessary ways.

But, perhaps due to my own biases, what I find most undesirable about it is just the way that it raises the cognitive friction of someone approaching an API for the first time.

I will agree with a couple of points though. One is that a property on a given resource cannot be simultaneously user-managed and system-managed, and it's important to understand that distinction, especially as an API designer. Another is that we absolutely do need better schema tools to define these important characteristics of properties. (Not just system-managed vs user-managed, but also, for example, mutable/dynamic vs immutable/static.)

I hope for OpenAPI 4 to have better tools to define these aspects of schemas, and in the mean time, we could look at some AEP vendor extensions for OpenAPI 3.x as well as whatever standardization of annotations we might need for Protobuf. I think it would be a big mistake to try to define these aspects with the structure of serialization though. I think the structure of serialization needs to be reserved for how resources relate to each other, not for the finer points of semantics of various properties.

cc @meem

[toumorokoshi]

Thanks for the thoughtful reply! I think it's great that we agree that system-managed / user-managed is an important distinction in an API. It also sounds like we agree that there's a need to be able to describe that behavior in a programmatic description of an API.

If we have that, and we assume it's accurate, then it's big step forward in the usability of said APIs in clients.

More detailed thoughts:

The idea of wrapping resources in a standard envelope is one of those persistent ideas that pops up over and over again for HTTP/JSON APIs (yes, Kubernetes, but also JSON:API and Cloud Foundry and others before it). But it never gains traction, and I think there are some pretty good reasons for that.

It sounds like you're defining "traction" as universal adoption, but I will note that, at least in Kubernetes, this convention has been adopted in basically 100% of the resources exposes via it's API, of which there are thousands of resources within the ecosystem of third-party extensions / controllers.

One is that what seem like the obvious categories to split properties into tend to vary across contexts — so any decision you make is likely to make your standard awkward and undesirable across projects and over time.

out of curiosity, what other types of divisions do you imagine?

It also raises the minimum complexity of a resource and doubles the depth-hierarchy of resource graphs. And, it constrains backward-compatible evolution of resources in unnecessary ways.

depth-hierarchy is true, but why would it constrain backwards-compatible evolution? It seems like whether your resource is defined in the top-level or in a nested level, it would require the same guarantees?

I hope for OpenAPI 4 to have better tools to define these aspects of schemas, and in the mean time, we could look at some AEP vendor extensions for OpenAPI 3.x as well as whatever standardization of annotations we might need for Protobuf. I think it would be a big mistake to try to define these aspects with the structure of serialization though. I think the structure of serialization needs to be reserved for how resources relate to each other, not for the finer points of semantics of various properties.

+1 to some form of extension for this. I think it would be great if we could forge ahead - perhaps the work we can do inform OAS 4.0?