Proposal: Structured Previews

[Abbe98]

Abstract

This proposal introduces a JSON-based alternative to the existing HTML previews with the intent to allow clients to control the user experience and presentation of said previews. A secondary intent is to make it possible to utilize previews in environments where HTML rendering might not be convenient (terminals, etc).

Status of this Proposal

This proposal reflects our experimental usage rather than the expected end result. For concerns we already have in mind, see the open questions below. If the group considers this a worthy proposal we intend to keep our experiments and this proposal in sync.

Background and Motivation

We use a few hundred reconciliation services and even though we run all services ourselves using a few shared frameworks the look and feel of the HTML previews have diverted over the years.

In addition to the issue above the client has no control(in a sane and safe way) over the look and functionality of the embedded HTML preview causing user settings related to; keyboard shortcuts, dark mode, font size, etc to differ between the client and the preview.

We can avoid some of these issues as we run all services ourselves and have our own clients, however, new problems arise with third-party clients as we do so. For example, many of our HTML previews now support dark-mode natively but OpenRefine(.org) does not, causing previews to use a different theme. With structured previews, we want to give the power of controlling theming and much more to clients like OpenRefine.

Another lesser use case is our wish to use the information provided by preview endpoints in our CLI, where HTML rendering is not an option.

Upstreaming this extension would allow our public reconciliation services to remain compatible with OpenRefine and other clients even as we deprecate HTML previews.

Open Questions and Known Issues

• The endpoint/capability is not currently announced by the service manifest.
• The image property should possibly be media and contain additional information needed to render it safely.
• One could argue that HTML previews should be deprecated altogether given the burden of sandboxing and CSPs. However, they are powerful and might still have their use cases.
• We imagine that there is a need for custom properties and that such an extension mechanism should maybe be a part of the specification.
• JSON schemas are missing.
• Error responses and error schemas.

[netlify]

✅ Deploy Preview for reconciliation-api-specs ready!

Name	Link
🔨 Latest commit	60befb7
🔍 Latest deploy log	https://app.netlify.com/sites/reconciliation-api-specs/deploys/671a8ac70dd9b30008488e7f
😎 Deploy Preview	https://deploy-preview-182--reconciliation-api-specs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[thadguidry]

• The image property should possibly be media and contain additional information needed to render it safely.

Hmm, can you speak more about the additional information and what this might look like? And what safety concerns are you thinking of?

Media is a very wide net, and on the Web, there's been much evolution and will continue to be so. As evidence, in Schema.org we've already experienced several iterations of https://schema.org/MediaObject
I could envision some clients wanting previews of Video, Audio, 3D models, hell, even ASCII Art.

[Abbe98]

Hmm, can you speak more about the additional information and what this might look like? And what safety concerns are you thinking of?

By "safety" I do not mean "securely" here but rather if you can determinate enough about the resource to render it correctly. From a purely web-perspective one should determinate be able to determinate what element that should be used for the browser to display the resource. I imagine by simply providing both the resource URL and a MIME type for that resource.

[thadguidry]

I think it makes sense that structured previews would use existing structure standards of the Linked Data web. That way, structured data to power previews could actually come from many systems around the world simply through... a link. Linked Data allows an app to start at one piece of Linked Data and follow and retrieve additional data through embedded links and JSON-LD is a lightweight syntax that allows existing JSON to be interpreted as Linked Data with minimal changes.

We also automatically get strings annotated with their language (we don't have to do backflips).
Any terms are then allowed to simply be mapped to IRI's through the use of @context, if desired.
I could go on and on.

It's also easy enough to generate expanded, compacted, table, etc. forms through existing JSON-LD client libraries.
It's dead simple for preview clients to use existing libraries and show full expanded context, or simply a nice tabular nested record, or anything in-between.

The world of Linked Data then automatically helps with preview generation through a simple...link, not even a query itself has to be formed in many cases. But there are SPARQL endpoints around the world that even do that...generate a JSON-LD serialization for their results.

A Few Examples:

• Getty's LOD initiative https://linked.art and https://www.getty.edu/research/tools/vocabularies/lod/index.html
  • https://vocab.getty.edu/ulan/500115493.jsonld
  • https://vocab.getty.edu/tgn/7011179.jsonld
• Library of Congress' various catalogs
• Nasa 20160830_Importing_the_SpaceSituationalAwarenessOntology_and_Exporting_JSONLD.pdf
• Australian Statistical Geography Linked Data API

Your example could thus look like this (where I've added context and "sameAs" also as an expanded example).
You can paste this into https://json-ld.org/playground and scroll down view the different previews of it.

JSON-LD (with a Schema.org context):

{
	"@context": "https://schema.org",
        "@id": "http://www.wikidata.org/entity/Q2",
	"name": "Earth",
	"description": "third planet from the Sun in the Solar System",
	"url": "https://www.wikidata.org/wiki/Q2",
	"image": "https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/The_Blue_Marble_%285052124705%29.jpg/330px-The_Blue_Marble_%285052124705%29.jpg",
	"additionalType": [
	    "http://www.wikidata.org/entity/Q128207",
            "http://www.wikidata.org/entity/Q3504248",
            "terrestrial planet",
            "inner planet of the Solar System"
	],
        "sameAs": "https://science.nasa.gov/earth"
}

[Abbe98]

I couldn't agree more @thadguidry, and JSON-LD based previews would actually make some of our services compatible simply by pointing to URIs of our various data services(example: https://fornpunkt.se/lamning/bjyWKNz.jsonld). Not to mention that the reconciliation specification would benefit of RDFs native extension support, alignment with RDF based service/data discovery vocabularies, native multilingual support, and much more.

However, I wouldn't want to make this proposal dependent on such a larger effort as I see this proposal as it could be be beneficial on its own and the easily merged with a JSON-LD effort as illustrated by your example.

[wetneb]

Thanks for this proposal!

Previews are one of the main things I would prioritize improving if I wanted to continue developing my Wikibase reconciliation service, as people have been asking for adding much more data to them. I would essentially add all statements of the Wikibase item, displayed in a more compact way (perhaps with a way to prioritize certain properties if there is a sensible way to do so). For all the corresponding data to fit in a response like the one you're proposing, it would essentially amount to returning all entity data in the response.

So, I see the interest in returning structured data instead of HTML from an API purity point of view, but I wonder what makes those structured previews different from fetching the data of the entity, for instance via content negotiation on the URI or via the data extension API. To me, it's not clear that there is sufficient difference between the two use cases to really justify having a separate API endpoint in the specs for that.

I would also be reluctant to deprecate HTML previews, because of the freedom they give to the service to present its data in a suitable way. The data model of the reconciliation API is really basic, so in a lot of cases services will need to coerce their data into it at a loss. I think those previews are a welcome opportunity for the service to render its data in its "native" form, with the appropriate context and rendering conventions.

[Abbe98]

So, I see the interest in returning structured data instead of HTML from an API purity point of view, but I wonder what makes those structured previews different from fetching the data of the entity, for instance via content negotiation on the URI or via the data extension API. To me, it's not clear that there is sufficient difference between the two use cases to really justify having a separate API endpoint in the specs for that.

I see three challenges using content negotiation on the URI(which to some extent also applies to the data extension API):

1. It's incredibly hard to render RDF(or JSON) in an user friendly way if the data model is not know by the client, even if you resolve contexts and have your client ingest the most common vocabularies. You bring up the tricky question of the importance/ranking of statements but beyond that even knowing the predicates is hard. Imagine you want to render schema:image as an image, first you are very lucky to work with a ontology your client is aware of, now other problems start; that schema:image predicate can either point to schema:ImageObject node, an URI resolving to a schema:ImageObject node or a plain URL pointing to an image. The common solution to this is to define profiles of data models, at which point we are back to this proposal and future work on defining a schema.org subset for use in a preview-service.

2. Not all services are RDF enabled, CSVs, OpenRefine projects, some third-party service you don't control, etc.

3. The majority of reconciliation services in the wild are third-party ones. If you add a dependency on URIs you end up in a situation in which you need to proxy through fake(third-party) URIs(which would bring you back to this proposal). Consider for example the GeoNames service over at fornpunkt.se, even if a client would support previewing the geonames.org we would need to setup our own "fake" URIs not to rely on geonames.org, or if we would be okay with relying on geonames.org we would need to open up our CSP to geonames.org.

I would also be reluctant to deprecate HTML previews, because of the freedom they give to the service to present its data in a suitable way. The data model of the reconciliation API is really basic, so in a lot of cases services will need to coerce their data into it at a loss. I think those previews are a welcome opportunity for the service to render its data in its "native" form, with the appropriate context and rendering conventions.

I completely agree and I argue for them not to be deprecated :-)

[acka47]

We discussed this in the last meeting and I realized I have some reservations with regard to standardizing structured previews:

1. The purpose of the entity information in the structured previews is to enable identifying the right match in a set of more or less similar entities,
2. Thus, I need those properties that actually help for identification.
3. Depending on the type of entities a reconciliation service is providing, the properties needed for identification differ, e.g. birth/death dates and places for persons, geo information for places etc.

Presumed these three statements, it probably is not possible and does not make sense to standardize identifying properties for preview on protocol level.

[thadguidry]

@acka47 exactly, don't standardize at the protocol level, but instead around the basic structure of a preview, as @Abbe98 and I are encouraging. That way, any service can throw into the structured preview, any and as many "disambiguating properties" that they want clients to help disambiguate between entities.

[Abbe98]

We discussed this in the last meeting and I realized I have some reservations with regard to standardizing structured previews:

1. The purpose of the entity information in the structured previews is to enable identifying the right match in a set of more or less similar entities,

2. Thus, I need those properties that actually help for identification.

3. Depending on the type of entities a reconciliation service is providing, the properties needed for identification differ, e.g. birth/death dates and places  for persons, geo information for places etc.

Presumed these three statements, it probably is not possible and does not make sense to standardize identifying properties for preview on protocol level.

I think that the off-topic discussion about JSON-LD might have lead to some confusion because this is exactly what this proposal addresses(and I properly contributed to that confusion).

A preview "description", "title", etc is not conceptually the same as that of an entity. Consider a location as an entity ; the entity would have properties about the name of the location, a higher level administrative area etc. The preview service would pull several layers of administrative areas from various entities into the description to help with identification, it might also provide a dynamically generated map-thumbnail.

Given the negative summary form the meeting I wonder if the actual use cases this proposal addresses were discussed and if the group considers any alternative approaches to solve them?

[fsteeg]

Given the negative summary form the meeting I wonder if the actual use cases this proposal addresses were discussed and if the group considers any alternative approaches to solve them?

I think the basic idea in the meeting was that such structured previews would be a kind of entity data access, for which don't have an API yet, but which might make sense to add. The discussion started like this: "we have no access to the content of an entity, like data extension, but without having to ask for specific properties", the action item is "comment about possible entity API". (The discussion moved into ways of configuring such an API, which perhaps moved the original "without having to ask for specific properties" part into the background.) So from my understanding, a possible solution would be to add a new kind of entity API, which in the simple, no-config case would return data suitable for previews?

Since it's been a bit hidden in the discussion here (it's in the diff only), this is the example from the original proposal (by @Abbe98):

{
  "id": "http://www.wikidata.org/entity/Q2",
  "name": "Earth",
  "description": "third planet from the Sun in the Solar System",
  "url": "https://www.wikidata.org/wiki/Q2",
  "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/The_Blue_Marble_%285052124705%29.jpg/330px-The_Blue_Marble_%285052124705%29.jpg",
  "tags": [
    ["terrestrial planet", "http://www.wikidata.org/entity/Q128207"],
    ["inner planet of the Solar System", "http://www.wikidata.org/entity/Q3504248"]
  ]
}

If we remove the url (available via view.url), and replace the tags with type, we only have the additional image diverging from what an entity is according to the spec:

{
  "id": "http://www.wikidata.org/entity/Q2",
  "name": "Earth",
  "description": "third planet from the Sun in the Solar System",
  "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/The_Blue_Marble_%285052124705%29.jpg/330px-The_Blue_Marble_%285052124705%29.jpg",
  "type": [
    {"name": "terrestrial planet", "id": "http://www.wikidata.org/entity/Q128207"},
    {"name": "inner planet of the Solar System", "id": "http://www.wikidata.org/entity/Q3504248"}
  ]
}

This could then be returned e.g. for a simple (no-config) GET /entity/{id}.

When requesting specific properties, these could be added in an additional properties field (basically a non-batch data extension API):

{
  "id": "http://www.wikidata.org/entity/Q2",
  "name": "Earth",
  "description": "third planet from the Sun in the Solar System",
  "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/The_Blue_Marble_%285052124705%29.jpg/330px-The_Blue_Marble_%285052124705%29.jpg",
  "type": [
    {"name": "terrestrial planet", "id": "http://www.wikidata.org/entity/Q128207"},
    {"name": "inner planet of the Solar System", "id": "http://www.wikidata.org/entity/Q3504248"}
  ],
  "properties": [{
    "id": "P138",
    "name": "named after",
    "values": [
      {"name": "soil", "id": "Q36133"},
      {"name": "land", "id": "Q11081619"},
      {"name": "ball", "id": "Q838611"}
    ]
  }]
}

[Abbe98]

I think the basic idea in the meeting was that such structured previews would be a kind of entity data access, for which don't have an API yet, but which might make sense to add. The discussion started like this: "we have no access to the content of an entity, like data extension, but without having to ask for specific properties", the action item is "comment about possible entity API". (The discussion moved into ways of configuring such an API, which perhaps moved the original "without having to ask for specific properties" part into the background.) So from my understanding, a possible solution would be to add a new kind of entity API, which in the simple, no-config case would return data suitable for previews?

Thank you @fsteeg for the background. This proposal is only about presenting an entity for identification just like with the existing preview feature, with the difference being the format and a few standardized properties meant to move control of display and accessibility from the reconciliation endpoint to the client.

If we remove the url (available via view.url), and replace the tags with type, we only have the additional image diverging from what an entity is according to the spec:

In hindsight I maybe shouldn't have chosen so similar property names, nor exemplified with Wikidata content. The properties are not intended to represent entity data 1:1 but information that can be used for previewing and identification. To exemplify using the descriptions of tags and description from my proposal:

• description An optional description which can be provided to disambiguate namesakes, providing more context.
• tags An optional list of tags and optionally an URL associated with each tag. Tags can represent entity types, categories among other things;

I would like to emphasis that my proposal and initial PR does not mention data access nor is it in conflict with potential efforts around entity data access or JSON-LD. The proposal is solely about ensuring clients can control the display and accessibility of previews and to make it possibly to use previews in non-web environments.

[thadguidry]

In my current service prototype, I have disambiguating properties (a few properties used for identification to clearly expose differences between other entities). Previews have a high need for showcasing disambiguating properties (not all properties of an entity - Wikidata has P1963 but it's not purely for a limited set of disambiguating properties, but instead all common properties for a type). My need is for describing my limited set of properties directly within a structured preview.

Thus, the further problem that I have is when I need to give more information about those disambiguating properties themselves. For each Previews' disambiguating property, I was hoping to avoid data duplication and instead use a graph (preferably JSON-LD) so that I could describe the disambiguating properties once at the beginning of the preview.

Ex. 1 Disambiguating Property "affiliation" without using JSON-LD syntax or context, but just descriptors for universal understanding

{
    "affiliation": {
        "alias" : [
            "ally",
            "companion",
            "cohort"
        ],
        "description": "a loose alliance with another organization",
        "about": {
            "type" : "Relationship",
            "alias" : "Connection"
        },
        "sameAs" : "http://www.w3.org/ns/org#memberOf"
    }
}

My service would deal mostly with Organizations and having disambiguating properties such as "leader", "affiliation", "location", "industry".

Preview:

{ "id" : "1234",
  "name" : "Affiliated Metals",
  "affiliation" : "Reliance Steel & Aluminum Co.",
  "location" : "Salt Lake City, Utah",
  "industry" : "423510"
}