Migrate generated schemas pages under static website

[sbesson]

Context

During the migration of the OME website 2 years ago, the OME schemas kept on the legacy server and proxied to the new website.

One of the main issue of the architecture above is any downtime of www or www-legacy will cause schemas to be offline. The impact of such unavailability is the potential inability to validate any OME-TIFF file.
A second issue is the complexity of process for updating these pages either when releading new schemas, adding new representations of the OME specification (see ome/ome-model#89) or simply updating the static HTML pages.

Proposal

This PR adds a preliminary step to add the OME XSD schemas, XSLT transforms and landing pages to the source of this repository. The website generated by Jekyll would already include all the Data Model schemas. Changes can be reviewed via GitHub pages part, schema pages would be part of the release assets and no proxying would be necessary in production.

Similar to #269, the main issue is to prevent specifications out-of-sync with the upstream specification defined in the ome-model repository. An inital attempt (827c8b80) checked out the schema pages under version control with a validation script. Following initial feedback, the sources for the schemas, transforms and schema landing pages are now generated in a pre-build step using the generate_schemas.sh script.

Testing

With this PR included, http://snoopycrimecop.github.io/www.openmicroscopy.org/Schemas/ should now include the landing pages and schemas.

Next steps

If the strategy proposed above is suitable and no alternate proposal comes up,:

• migrate the publish script to this repository
• fix Schemas vs XMLSchemas
• investigate some minimal unification with the OME CSS style
• migrate the generated documentation under https://docs.openmicroscopy.org/
• deploy and remove the proxy

Outside the scope of this effort

• improve the visibility and integration within the website (navigation)
• improve the generated landing pages

[joshmoore]

In general, 👍 for piecing all this apart. I do wonder if there's not a more decoupled strategy that needing a www.o.org PR each time for release. Perhaps in the case of the schemas that's sufficiently low-cost, but thinking about the upcoming RDF work, etc. is it low cost enough?

[sbesson]

I have been considering whether the publish/check_schemas.py scripts should involve into a dynamic build step i.e. a generate-sources step similar to the way Bio-Formats documentation format pages are generated.

With regarding to the use case you describe (RDF work), this means a PR updating a new version of the specification would only have to bump _config.yml and point at a new tag. Like all code generation step, the downside is not having a usable diff without work for reviewers.

[joshmoore]

How does the diff review work for https://github.com/IDR/idr.openmicroscopy.org/blob/master/.travis.yml#L16 ?

[sbesson]

I do not think we have any diff review for tags of IDR/OME website. This usually happens at the PR level at the moment. That being said, it should be possible to create a diff with the last GH release and attach it as an asset.

[sbesson]

Update: in the light of recent work, current looking into a revised version of this work where:

• the schemas pages are built in a separate project which consumes org.openmicroscopy:specification, unzips it and builds the HTML pages via the publish script
• a Github action auto-commits the generated HTML pages on every push
• dependabot to track the upstream specification releases
• including this separate repository as a submodule similarly to Community meeting updates #403