Refresh production pipelines for Rev 5 update

[wendellpiez]

User Story:

As an OSCAL user, I would like access to the best available OSCAL rendering tools, including XSLTs that produce HTML and (Accessible) PDF, with production values comparable to an official publication (albeit not identical look/feel).

Various production pipelines for OSCAL and specifically SP800-53 are being refreshed and updated for the upcoming Rev 5 errata release. Pipelines supporting FISMA production are maintained in internal repositories, but they call code in this repository (xslt/publish subdirectory) that can benefit from the updates.

Updated versions would also be more cleanly factored and documented for maintenance and reuse.

Additionally, some of the logic available there (generating spin-offs such as Appendix tables, etc.) might be migrated into this repository.

Dependencies:

Follows on to ongoing work on the pipelines (in NIST repositories)

Requirements

Tracking here the details of requirements for 'finishing'.

itemized

• Drop any line "Related Controls: none" or "Control Enhancements: none" from Withdrawn controls and CEs (control enhancements) - although a Withdrawn control still shows its CEs (which should also be withdrawn).
• Adjust some fonts and colors as described below, including Appendix C tables wendellpiez/oscal-xslt@9a7620b
• Adjust bibliography sorting
• Review items below to confirm

transcribed (keep aligned with internal comms)

• Withdrawn controls and CEs should not have “related controls: none” when there are no related controls (examples are AU-6(10), AU-7(2))
• SA-12 and SA-14 should not show related controls, but should show all of the withdrawn CEs (and where they went).
• The control enhancements in SA-12 should not show related controls
• Control Enhancement titles should be all capitalized and slightly smaller, the same size as the adjacent | wendellpiez/oscal-xslt@89b8b49
• Special publications should be listed in ascending order by publication number (e.g., SP800-18 should come before the 100’s, etc.) wendellpiez/oscal-xslt@e43a1cf
• Update Appendix C:
  • Control Summary Tables font size can be made smaller and more consistent wendellpiez/oscal-xslt@9a7620b
  • Controls and control enhancements that are withdrawn, make sure that row is greyed out wendellpiez/oscal-xslt@36d0018

implicit

• Be ready to / remove 'PREVIEW' side bar on PDF if/as instructed
• Any followup requirements from NIST RMF team.

Acceptance Criteria

• The Adjustments list above is up to date and complete in a working branch of the usnistgov/oscal-xslt repository
• Resulting PDF (on updated catalog) has been deemed 'ready' by RMF team
• Adjustments and improvements made to XSLTs in that repository are ready in a PR
• New XSLTs have been added if appropriate
• Alternatively (as appropriate) additional work (not to be published) is stabilized/archived in a branch or repo
• Published XSLTs run without error over arbitrary (valid) OSCAL catalogs
• All README documentation affected by the changes in this issue have been updated
• Two or more developer colleagues have contributed changes to this PR
• The CI-CD build process runs without any reported errors on the PR - This can be confirmed by reviewing that all checks have passed in the PR

[aj-stein-nist]

Subject to dependency on FISMA Implementation Team's delivery schedule and release timelines. @wendellpiez will review, post the URL to the Gitlab repo, and mark the relevant goals as completed before close.

[wendellpiez]

Currently in a WIP branch I have some updates to the generic HTML and PDF production pipelines, made in connection with updates for the official (internal) production.

Of the two objectives of this Issue, this is the first, whereas the second is to capture any potential follow-on Issues such as publishing other code we have developed.

This would be addressed however by a PR into the https://github.com/usnistgov/oscal-xslt repository not this repository.

@aj-stein-nist should this Issue be moved there? (Interesting it is here instead of oscal-tools, where it might have been lost.)

[aj-stein-nist]

For Sprint 65, @wendellpiez is the driver for this issue.

[aj-stein-nist]

@wendellpiez needs to re-review A.J.'s translated checklist items for the tactical work in the PDF production. He also needs to sync with A.J. to discuss finer points around the oscal-xslt submodule work.

[aj-stein-nist]

@wendellpiez thanks for syncing up today. While you update the issue description and the AC checklist, I am going to look at the current version here:

• https://github.com/usnistgov/oscal-xslt/blob/4c4c2c301d8b35ac6536fd42a9eeea00708e2e6b/publish/nist-emulation/sp800-53A-catalog_html.xsl
• https://github.com/usnistgov/oscal-xslt/blob/4c4c2c301d8b35ac6536fd42a9eeea00708e2e6b/publish/make-catalog-pdf.xpl

[wendellpiez]

Perfect. WIP branch is here: https://github.com/wendellpiez/oscal-xslt/tree/catalog-view-updates-March2023

Entry points for adjusted HTML and PDF:

• https://github.com/wendellpiez/oscal-xslt/blob/catalog-view-updates-March2023/publish/nist-emulation/rev5update_html.xsl
• https://github.com/wendellpiez/oscal-xslt/blob/catalog-view-updates-March2023/publish/nist-emulation/rev5update-emulator_fo.xsl

They can be supported by a new pipeline or subbed in. Note that these both act as 'skins' around base XSLTs.

One of our tasks is to determine how much of this customization goes into the released 'emulator' XSLTs since as you will see in some ways it's a downgrade (for reasons a little painful to explain).

[aj-stein-nist]

OK making good progress on the recreating the setup for this work and testing out changes, now to start working down the issues on the list and asking Wendell as I encounter issues. More updates to follow.

[aj-stein-nist]

@wendellpiez I'll push up a commit but as for the second half of the first item in, removing "Control Enhancements: none" from Withdrawn controls and CEs (control enhancements) - although a Withdrawn control still shows its CEs (which should also be withdrawn)" I cannot find an example using the current data and just touching up the related links, has this second of the requirement already been handled?

[wendellpiez]

When I looked I was (and remain) hopeful that indeed more than one line item (requirement) would boil down into simple conditional logic. Maybe reviewing that outside the XSLT:

• Any control enhancements: none should be dropped
• Any **Related Controls: none" should be dropped at either level
• When control enhancements are present, they stay as they are

(Off the top of my head so let's review against the sheet.)

I also think all this could be done exclusive in the FO layer, not affecting the HTML, where arguably these "extra headers" are useful not spurious.

But we can look at all of it, including lining up fresh output against the request for changes, line by line.

[aj-stein-nist]

That totally makes sense, thanks @wendellpiez. I was saying for item one, I cannot find a current example where control enhancements: none in the HTML as I transform and edit, has this already been taken care of?

Additionally, I looked at all the SP references (or at least the first hundred or so) of 568 against the current 800-53 Revision 5 catalog, and they always seem to be in the correct order requested (SP 800-12, 800-18, 800-37, 800-137, 80011) and seems this was already done. That is why I am asking.

We can also just discuss it during our next pairing session later today.

[wendellpiez]

Indeed. All these are things that can and should be traced back. 🚀 Actually I think what you might see (if you look very closely wearing the right glasses) that there are elements in the HTML that the CSS is dropping ... this could be one.

We'll walk through it and look at all the layers on each step.

[aj-stein-nist]

🤦 and now looking at the PDF I see I didn't quite realize I was being dumb in not debugging one step after the other in both cases, woops!

[wendellpiez]

Yeah, I should have though of to mention - at this stage it is often easiest to start at the end and work backward. But we get to the same place all good. 🌞

[aj-stein-nist]

I met with Wendell and need to revert some initial changes to his current Rev 5.2 branch in wendellpiez/oscal-xslt#1 in my WIP PR. I learned more about the design and implementation approach: leaving data and presentation layer in HTML, removing elements in the FO PDF step. Since I went the opposite way, I will have to start over in the FO transform instead. We made good progress, so I think the current checked off items are still relatively current. Per agreement with Wendell as the driver, I will merge code into his branch, then we put a PR back to the oscal-xslt repo and the oscal-content once that is under wrap.

We will meet with the FISMA Team later this week and hopefully we can show that off and get good feedback. This task still looks very much on schedule.

[aj-stein-nist]

As discussed during standup, A.J. needs to make time to pair with Wendell on Thursday afternoon or Friday morning. It is likely this will not resolve by end of sprint, it is unclear it will be complete before Sprint 66 starts on Monday. TBD

[aj-stein-nist]

I had some setbacks on this issue and I did not complete this work as intended with Wendell as the driver, I wanted to do the work. I will discuss with him if we can or should move this to next sprint. I very much want to, as only a little of the work remains and with pairing it should take a few hours.

[wendellpiez]

@aj-stein-nist nice demo on this! Next up -

• Try without the 'rev5 update' customization layers for both HTML and PDF
• Examine the delta and determine which if any 'improvements" should migrate down
• Touch base w/ @nikitawootten-nist about old Issue Build CL invocation script for XSL + Apache FOP oscal-xslt#6

I believe we can close this Issue as soon as we have a PR ready with the changes we want. What I am proposing is that the changes may include improvements to the XSLTs, but they should not include the specialized logic for this delivery only. Our next job is to see the difference. The XSLTs are not huge so we could probably do this in an hour or two of pairing.

Let me know what you think -- this is going to be nice --

[aj-stein-nist]

This is a duplicate of usnistgov/oscal-xslt#3. Thanks to Wendell for the note, be sure to close both of the issues together when this work is done (transferring or merging will cause messiness).

[aj-stein-nist]

@wendellpiez I know the issue is under review but I updated the checklist accordingly. We still have some final items for close out, but I know those come after review of wendellpiez/oscal-xslt#1 and merging into unstgov/oscal-xslt, stakeholder review, and some discussion on success conditions for "Published XSLTs run without error over arbitrary (valid) OSCAL catalogs" and how I (probably) should document how you use this work once we merge it into usnistgov/oscal-xslt. For now, I mark as under review until you have time to review that PR tomorrow morning. 👋

[wendellpiez]

Two branches contain work to be merged:

• Updated templates for 'generic' processing: https://github.com/wendellpiez/oscal-xslt/tree/catalog-view-updates-March2023 behind Catalog view updates oscal-xslt#4
• Customizations specifically for delivery of current L/F using NIST-internal template - https://github.com/wendellpiez/oscal-xslt/tree/nistcsd-template-emulator2022 - can be pushed directly

[aj-stein-nist]

@wendellpiez I think we have a small amount of work to forward along documentation (shell or Oxygen) on how to operate the transforms as I had recommended, but other than that we are done, correct? I will move this forward to Sprint 67, but presume we ought to close this out quickly with a developer README, unless there is something else I missed from conversation on Friday. Let us know.

[aj-stein-nist]

To ensure understanding of the work let's make sure someone other than A.J. reviews this PR before close:

usnistgov/oscal-xslt#4

[aj-stein-nist]

I guess we will circle back on this during review, but is this complete given approvals of the related PR, @wendellpiez?

[nikitawootten-nist]

usnistgov/oscal-xslt#4 has been merged. Ok to close? @wendellpiez