Skip to content

v1.0.0
Compare
Choose a tag to compare
@david-waltermire david-waltermire released this 04 Aug 05:34
· 192 commits to develop since this release
v1.0.0
95b24a3

Changelog

This release contains unreleased commits from the NIST repository and changes that were requested, but not merged at the time of this release, including usnistgov/metaschema-java#374, usnistgov/metaschema-java#368, usnistgov/metaschema-java#367, usnistgov/metaschema-java#365, usnistgov/metaschema-java#363, usnistgov/metaschema-java#373, and usnistgov/metaschema#659. The commits for these PRs are also merged here.

This release includes:

  • significant cleanup of the code base, providing for easier long-term maintenance
  • a large number of new Metapath features and functions
  • robust support for external constraints that allows for further restrictions over an existing Metaschema module when validating content
  • many bug fixes that improve the reliability of this project
  • the ability to produce validation results in the Static Analysis Results Interchange Format (SARIF), which allows for integration with tools implementing this standard.

General

  • Added and improved Javadoc comments across all modules, with a focus on full documenting the core API.
  • Cleaned up many compile, code style, SpotBugs, CPD, and PMD warnings and errors.
  • Cleaned up orphaned code
  • Switched some tests using resources in the test classpath to use file resources to make testing more stable.
  • Refactored model interfaces in core and databind to reduce code duplication, to simplify interface heirachies, and to share more implementation code. This resulted in the removal of dozens of interfaces, which were no longer needed.
  • Cleaned up some accessor method names to be more consistent with their return types.
  • Added support for defining configuration feature names to better support debugging.
  • Cleaned up stream closure to ensure that streams are properly closed.
  • Added methods to handle making URIs relative to another URI.
  • Some work towards improving NullPointerExceptions to provide more context when missing data is the cause.

metaschema-core

Metaschema Module Model

  • Implemented choice group support based on usnistgov/metaschema#228, which provides a means to allow sequenced collections of objects of different types.
  • Cleaned up and documented loader interfaces, to include IDocumentLoader.
  • Refactored XML model parsing. Moved XML model classes to be hidden in the implementation.
  • Added support for QName in Metaschema data types, which are used in function signatures now instead of the Java class name.
  • Fixed XML choice model parsing to ensure instances are also added to the model instance list.
  • Refactored core model classes and support for collections
    • Relocated many class files and cleaned up many unneeded methods.
    • Reworked definition and instance model classes to reduce extra interfaces and to simplify and align implementations.
    • Added support for flag and field instances to provide a different default value than the definition.
    • Refactored instance class hierarchy to support grouped and ungrouped model instances, with minimal implementation.
      • Renamed model interfaces for greater consistency.
      • Refactored core model interfaces to reduce inheritance conflicts.
      • Adjusted the module interfaces to promote a cleaner, non-parameterized API.
    • Renamed many instance implementation classes to ensure consistent use of terminology. Prior the terms "property" and "instance" were used interchangeably, which could cause some confusion.
    • Refactored IBindingContext to move supporting interfaces to be child interfaces.
      • Removed the need for instances of IDataTypeHandler. Now IClassDataTypeHandler functionality is provided by IFeatureComplexItemValueHandler and JavaTypeAdapterDataTypeHandler is provided by IFeatureScalarItemValueHandler.
      • Refactored bound class hierarchy to be more consistent with needed methods.
      • Moved Java field handling to IFeatureJavaField from IBoundNamedModelInstance
      • Moved JSON key and item handling to IBoundModelInstance.
      • Reworked XML and JSON reading and writing.
      • Added IBoundChoiceGroupInstance interface for choice groups.
    • Cleaned up IModelPropertyInfo and implementations, relocating supporting classes as child classes.
      • Implemented a IModelInstance item parsing as IModelPropertyInfo.IReadHandler, allowing all item parsing to be implemented in the format-specific parsing classes, i.e. MetaschemaJsonReader, MetaschemaXmlReader.
      • Cleaned up IModelPropertyInfo readItems methods, which now return the parsed value. Eliminated the need for IPropertyCollector instances.
    • Refactored constraint ILet statement construction to use static factory methods.
    • Cleaned up JSON value key naming code.
    • Fixed default value handling which wasn't producing empty collection for unparsed model instances. Cleaned up getDefaultValue and getEffectiveDefaultValue methods.
    • Refactored JSON key handling to determine the JSON key on a per item basis. This is needed for ChoiceGroups.
    • Refactored container handling code to reuse the core model implementation where possible.
      • Refactored group-as namespace handling to allow for no namespace situations. Implemented getEffectiveGroupAsNamespace to handle the default case of using the module namespace where no group-as support is provided.

Metapath Interpreter

  • Added support for more XPath 3.1 syntax in Metapath:

    • Adjusted Metapath syntax to add support for for, if, range, simple map !, and extended QNames.
    • Added support for Metapath if expressions: if (expr) then (expr) else (expr).
    • Added support for Metapath range expressions: intValue to intValue.
    • Added support for Metapath quantified expressions: (some | every) $VarName in Expr1 (, $VarName in ExprN)* satisfies ExprS.

  • Added the following Metapath functions:

    • abs
    • avg
    • ceiling
    • concat
    • count
    • document-available
    • empty
    • head
    • insert-before
    • min
    • max
    • sum
    • tail
    • remove
    • reverse
    • array:get
    • array:put
    • array:append
    • array:subarray
    • array:remove
    • array:insert-before
    • array:join
    • array:head
    • array:tail
    • array:reverse
    • array:flatten
    • map:get
    • map:merge
    • map:entry
    • map:size
    • map:keys
    • map:contains
    • map:find
    • map:put
    • map:remove
    • meta:model which gets a nodeset based on the underlying Metaschema module model for the node.

  • Fixed defects in the following functions:

    • data
    • false
    • true
    • starts-with which was checking for contains not startsWith.

  • Added support for sub-DynamicContexts supporting variable definition in let and quantified expression statements.

  • Added support for map construction.

    • Added support for maps in function call, postfix and unary lookups.
    • Added missing characteristics on a number of existing Metapath functions.

  • Added support for Metapath arrays

    • Completed code to support Metapath array construction.
    • Implemented array postfix and unary lookups.

  • Changed ISequence to implement the List interface to allow sequences to better participate in collection-oriented Java methods.

  • Cleaned up ISequence implementations, moving getFirstItem to be a method member. Added a utility function to convert a sequence into an array member.

  • Refactored comparison methods into a new ComparisonFunctions class.

  • Reorganized and improved item implementations to better group interfaces and implementation classes and to be more robust.

    • Refactored atomic item interfaces and implementations.
    • Added support for casting an IItem of IAnyAtomicItem type to the type of another item using the castAsType method. This is useful for functions that require the return type to match the argument's type.
    • Added support for compareTo for all atomic items.
    • Added equals and hashCode to all item types.
    • Renamed FlagDefinitionNodeItemImpl to FlagGlobalDefinitionNodeItemImpl to better harmonize with sibling field and assembly implementations.
    • Added toString methods to some items to improve debugging and testing.
    • Refactored IDocumentNodeItem and IModuleNodeItem to better align the features of each. A document node item not exposes the root assembly, while a module node item exposes the global definitions and module namespace.
    • Ensured proper handling of Metapath errors during validation. Resolves usnistgov/oscal-cli#292

  • To improve maintenance of Metapath function test vectors, inline compilation of Metapaths are now used, making tests more readable while we working on usnistgov/metaschema#239.

  • Fixed a bug causing Metapath lexer errors to be silently ignored.

  • Reorganized and refactored the Metapath compact syntax tree and ANTLR-based abstract syntax tree implementation classes.

  • Fixed bug causing empty sequences to not be produced by the Metapath expression: ().

  • Cleaned up static and dynamic Metapath exceptions. Refactored StaticContext generation to ensure proper use of model-based methods.

  • Adjusted namespace of Metapath functions to http://csrc.nist.gov/ns/metaschema/metapath-functions.

  • Improved function testing to lookup and call the function. This will more fully exercise the function call in JUnit tests.

  • Fixed a defect in opNumericDivide that caused improper handling of integer division.

  • Adjusted the document node item to ensure it returns the root node when getValue is called. This fixes usnistgov/oscal-cli#216.

  • Added support for Expanded QNames

  • Fixed and tested mod expression handling to properly return an integer result

  • Fixed function chain arrow operator processing

    Made => operator eqname only in ANTLR grammar. See #68 for follow-on work.

    Per discussion with maintainers, adjust arrowfunctionspecifier to not parse the arrow function chain operator as a eqname or varref or a parenthesizedexpr, but only an eqname to fix parsing complexity. The relevant code to support this in BuildCSTVisitor has still not been implemented per the TODO comment.

  • Fixed BuildCSTVisitor context processing for usnistgov/metaschema-java#289

    The index and offset for BuildCSTVisitor for arrow chain operator context processing was off and therefore exceptions were occurring when handling => function calls during evaluation and therefore function calls were not instantiated the correct way. We also added tests to validate it works now, uncovering another issue in the use of fn:sum casting integers to decimals, that was also fixed.

Constraint processing

  • Added support for let statements in constraints.
  • Refactored constraint implementation, reducing code complexity, improving organization, and hiding implementation classes. Removed unused code.
  • Unified external constraint processing through use of an ExternalConstraintsModulePostProcessor.
  • Added XmlMetaConstraintLoader, which can load an XML-based <metaschema-meta-constraints> instance.
  • Fixed bug in the output generated by newCardinalityMinimumViolationMessage, to provide some context around the error. Before an empty array was always displayed, which was not helpful.
  • Refactored external constraint parsing, abstracting common data objects.
  • Added support for meta constraint imports.
  • Cleaned up ISource implementations, and ensured that URI-based sources are properly cached.
  • Added support for exposing parse locations in validation results.
  • Adjusted constraint result production to allow for pass results to be produced, which supports producing SARIF result that include both pass and fail statuses using an API-level configuration.
  • Refactored Metaschema validation API to allow for more flexibility in use.
  • Improved handling of missing index in index-has-key constraints
    • Added exception with explanation for missing index for #280
    • Tests demonstrating clear error reporting for #280
    • Report name of missing index in error message

metaschema-databind

  • Refactored constraint parsing to cleanup the API and to normalize processing between the legacy and meta constraint formats.
  • Implemented a Metaschema module for the Metaschema module format.
  • Added bindings for the legacy and new external constraint models.
  • Implemented a Metaschema module binding-based constraint loader: BindingConstraintLoader. This loader is capable of loading the legacy external constraint format and the new meta constraint format.
  • Added support for choice groups in serializers/deserializers, and code generator.
  • Refactored annotations to consolidate ValueConstraints, AssemblyConstraints, and GroupAs as child annotations of the core BoundAssembly, BoundField, BoundFlag, and BoundChoiceGroup annotations.
  • Cleaned up Metaschema module binding classes to align constraints with current constraints and added back legacy data types.
  • Refactored package names to better organize binding related code and to distinguish provided model bindings.
  • Added support for capturing parsed location information in bound objects. This will be useful for producing context for validation results.
  • Fixed bugs causing the ordering of generated classes to be chaotic. Also fixed bugs causing binding configurations to match based on minor URI differences caused by inconsistent behavior between file and path URI productions.
  • Added JSON parser test to test for the usnistgov/metaschema-java#308 regression.
  • Refactored XML and JSON item writing to use a shared IModelInstanceCollectionInfo.IWriteHandler implementation, allowing all item writing to be implemented in the format-specific classes, i.e. MetaschemaJsonWriter, MetaschemaXmlWriter.
  • Implemented choice group writing for XML, JSON, and YAML.
    • Moved JSON reading and writing code from IItemValueHandler to MetaschemaJsonReader and MetaschemaJsonWriter using dispatch methods. This allows are IO code to be implemented withing the JSON format-specific implementation.
    • Moved XML reading and writing code from IItemValueHandler to MetaschemaXmlReader and MetaschemaXmlWriter using dispatch methods. This allows are IO code to be implemented withing the XML format-specific implementation.
    • Cleaned up XML parsing code. Fixed a bunch of bugs causing parsing to not work.
      • Fixed all identified bugs in XML parsing.
      • Added support for parsing XML fields as a root element.
  • Cleaned up IBoundGroupedFieldInstance implementations to use only the complex form, since a grouped item must have a bound class to determine its item type.
  • Aligned generated annotations with class generation, adding support for many missing annotation entries.
  • Added binding support for choice groups.
  • Added Metaschema module bindings.
  • Adjusted the name used during code generation to use the value key name if available. This provides a more sensible name for Java property get and set methods created during code generation.
  • Added support for deprecated on allowed value enumerations.
  • Added support for Metaschema module loading that uses the Metaschema module binding instead of the XML parser. Metaschema modules in XML, JSON, and YAML can now be fully read!
    • Added Metaschema module loader support for entity resolution. Tested with OSCAL Metaschema modules.
    • Cleaned up bound module loading in DefaultBindingContext.
    • Relocated the IModulePostProcessor interface to the new IModuleLoader interface.
    • Refactored flag and field name handling for a cleaner implementation. This new approach uses "name" as the binding annotation value for flags, instead of "useName".
  • Improved testing of assemblies with various types of fields and flags.
    • Implemented new unit tests to test the model bindings.
  • Refactored DefaultJsonDeserializer to read a root element without the need to produce a Metapath item first.
  • Adjusted the class generation for grouped instances to allow for extended classes where definitions are used by reference.
  • Implemented subclass generation for choice group instances.
  • Moved Metaschema module binding classes to a subdirectory to help with change integration.
  • Refactored JSON schema generator. Many improvements to the indexing approach used to determine which definitions become global definitions in the JSON schema.
    • Improved JSON documentation to include titles and descriptions on properties.

metaschema-databind-modules

  • Added support for producing Static Analysis Results Interchange Format (SARIF) results based on schema and constraint validation results.
    • Added support for including rules and artifact information in SARIF results. SARIF files now work on commonly available viewers.
    • Added constraint formal-name and description to SARIF output, allowing human readers to better understand why the result was produced.
    • Added a GUID to SARIF output for each rule.

metaschema-maven-plugin

  • Added support for configuring external constraints in the maven plugin for the schema generation and code generation mojos.

metaschema-cli

  • Added SARIF CLI output option to validate command.
  • Improved abstract CLI validation commands to provide more options for model detection.
  • Added initial implementation of the metaschema-cli metapath list-functions command.
  • Added a better means to collect the metaschema-related CLI commands for reuse.
  • Fixed issue causing CLI errors when using --help or --version, while not providing a required command parameter. Resolves usnistgov/metaschema-java#293.
  • Added support for URL use in CLI arguments

    • Add UriUtils class for URI/Path translation

      Currently, we have the metaschema-cli only support local file paths for
      arguments for content inputs and outputs. This loader will support the
      use of MetaschemaLoader.load() with URIs and unify local file path and
      URL access (primarily https://, http://) with the same command line
      tooling for similar remote and local access.

    • CLI commands allow files or URIs for usnistgov/metaschema-java#297

      This PR adjusts the commands and supporting databind classes and ports
      them to use URIs. The commands will use the new UriUtils class and the
      toUri() function to use a remote URI supported as a URL and load the
      model, schema, or target file. If a local file, it will convert the file
      path to valid URI.

      Integration tests have been added to validate this functionality works
      with the update CLI commands.

    • Specific error handling for different URI failures

      Per discussion with @david-waltermire, update the exception handling and
      associated error messages to identify if a bad domain for a remote URI
      is used or a valid domain is used but a bad URL.

Repository

  • Updated workflows, templates, and core repository files to localize to this repository. Simplified code preamble to use SPDX tags.

  • Cleaned up retired modules that are no longer in use.

  • Fix dependency bounds issue for usnistgov/metaschema#265 (#266)

    More details with the reason for this PR can be found in the build
    mentioned below.

    Error:  Rule 0: org.apache.maven.plugins.enforcer.RequireUpperBoundDeps failed with message:
Failed while enforcing RequireUpperBoundDeps. The error(s) are [
Require upper bound dependencies error for org.xmlresolver:xmlresolver:5.2.0 paths to dependency are:
+-gov.nist.secauto.metaschema:metaschema-schema-generator:1.0.0-M2-SNAPSHOT
  +-net.sf.saxon:Saxon-HE:12.4
    +-org.xmlresolver:xmlresolver:5.2.0 (managed) <-- org.xmlresolver:xmlresolver:5.2.2
and
+-gov.nist.secauto.metaschema:metaschema-schema-generator:1.0.0-M2-SNAPSHOT
  +-net.sf.saxon:Saxon-HE:12.4
    +-org.xmlresolver:xmlresolver:5.2.0:data (managed) <-- org.xmlresolver:xmlresolver:5.2.2:data
]

  • Integrated Maven reporting profile into main build. No need to use -Preporting anymore to generate reports and Javadocs. These now build when the site goal is used.

  • Fixed many broken website links.

  • Normalized line endings for all text files. Added a .git attributes to ensure that line endings are properly maintained.

What's Changed

Full Changelog: https://github.com/metaschema-framework/metaschema-java/commits/v1.0.0

Contributors

david-waltermire, dependabot, and 2 other contributors
Assets 2
Loading