From 15c113980dcc37cb52774d5272827ab513d57354 Mon Sep 17 00:00:00 2001 From: Kevin Schneider Date: Thu, 14 Mar 2024 16:41:03 +0100 Subject: [PATCH] Update readme --- README.md | 205 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 145 insertions(+), 60 deletions(-) diff --git a/README.md b/README.md index e8a6a75..6a269de 100644 --- a/README.md +++ b/README.md @@ -1,78 +1,142 @@ -# arc-validate-package-registry +# Introduction -This repository contains: +The **arc-validate-package-registry (avpr)** repository contains: -- a staging area for authoring official validation packages intended for use with [`arc-validate`](). +- a staging area for authoring official validation packages intended for use with [`arc-validate`](https://github.com/nfdi4plants/arc-validate). - a web API for serving validation packages. This API is consumed by `arc-validate` to install and sync validation packages. +- a website for browsing validation packages. +- some domain types and utilities relevant for consuming libraries in the [AVPRIndex library](./src/AVPRIndex/) +- a .NET client library for consuming the web API in the [AVPRClient library](./src/AVPRClient/) + +Read more at [avpr.nfdi4plants.org/about](https://avpr.nfdi4plants.org/about) # Table of contents -- [The package index](#the-package-index) -- [Automated package testing](#automated-package-testing) -- [Local development](#local-development) -- [OpenAPI endpoint documentation via Swagger UI](#openapi-endpoint-documentation-via-swagger-ui) -- [package indexing](#package-indexing) -- [local development](#local-development-1) +- [General](#general) +- [Validation package staging area](#validation-package-staging-area) + - [Allowed validation package file formats](#allowed-validation-package-file-formats) + - [Automated package testing](#automated-package-testing) + - [The preview package index](#the-preview-package-index) + - [How to add packages](#how-to-add-packages) + - [Versioning packages](#versioning-packages) + - [Package publication workflow](#package-publication-workflow) - [Package metadata](#package-metadata) - [Mandatory fields](#mandatory-fields) - [Optional fields](#optional-fields) - [Objects](#objects) - - [Package publication workflow](#package-publication-workflow) - - [Versioning packages](#versioning-packages) + - [Author](#author) + - [Tag](#tag) +- [Web API (PackageRegistryService)](#web-api-packageregistryservice) + - [Local development](#local-development) + - [OpenAPI endpoint documentation via Swagger UI](#openapi-endpoint-documentation-via-swagger-ui) +- [Repository setup](#repository-setup) + - [local development](#local-development-1) + +# General + +This repo runs an [extensive CI/CD pipeline](.github/workflows/pipeline.yml) on every commit and PR on the `main` branch. The pipeline includes: + +- tests and pre-publish checks for every package in the [staging area](#validation-package-staging-area). +- a release pipeline for validation packages: + - publishing WIP packages to the `avpr-preview-index` on this repo's [preview-index](https://github.com/nfdi4plants/arc-validate-package-registry/releases/tag/preview-index) release + - publishing stable packages to the production instance of the web API at [avpr.nfdi4plants.org](https://avpr.nfdi4plants.org) +- tests and release pipelines for the `AVPRIndex` and `AVPRClient` libraries, as well as a docker container for the `PackageRegistryService` web API. + +```mermaid +flowchart TD + +setup("setup
(determines subsequent jobs
based on changed files)") +batp("Build and test projects
any of [AVPRIndex, AVPRClient, API]") +tsa("Test staging area
test all packages in the staging area") +sapc("Staging area pre-publish checks
hash verification, prevent double publication etc") +nr("Release (nuget)
any of [AVPRIndex, AVPRClient]") +dr("Release (docker image)
API") +upi("Update preview index
update the github release index json file") +ppp("Publish pending packages
Publish packages to production DB") + +setup --when relevant project
files change--> batp +setup --changes in the
staging area--> tsa +batp --when tests pass and
release notes change--> nr +batp --when tests pass--> dr +tsa --when tests pass--> sapc +sapc --when checks pass--> upi +sapc --when checks pass
and any new packages
are pending--> ppp +``` +[🔼 Back to top](#table-of-contents) # Validation package staging area -## The package index +The [package staging area](./StagingArea) is intended for development and testing of validation packages. + +Files in this folder must follow the naming convention `@...*` and contain a [yml frontmatter](#package-metadata) at the start of the file. These files must additionally be inside a subfolder exactly named as the package name. This leads to a folder structure like this: + +```no-highlight +StagingArea +│ +├── some-package +│ ├── some-package@1.0.0.fsx +│ ├── some-package@2.0.0.fsx +│ └── some-package@2.1.0.fsx +│ +└── some-other-package + ├── some-other-package@1.0.0.fsx + ├── some-other-package@2.0.0.fsx + └── some-other-package@3.0.0.fsx +``` + +## Allowed validation package file formats -This repo runs a [custom pre-commit hook](pre-commit.sh) that will run a [script](./update-index.fsx) automatically add any `.fsx` file in the [staging area](StagingArea/) to [the package index](src/PackageRegistryService/Data/arc-validate-package-index.json) when it is commited to the repo. +As all reference implementations are written in F#, the only currently allowed file format for validation packages is `.fsx` (F# script files). This can and will be expanded in the future. ## Automated package testing Tests located at [./tests](./tests) are run on every package in the index. Only if all packages pass these tests, the docker container will be built and pushed to the registry. -# Web API (PackageRegistryService) - -The `PackageRegistryService` project located in `/src` is a simple ASP.NET Core (8) web API that serves validation packages and/or associated metadata via a few endpoints. +## The preview package index -It is developed specifically for containerization and use in a docker environment. +Besides the published packages available at [avpr.nfdi4plants.org](https://avpr.nfdi4plants.org), -The service will eventually be continuously deployed to a public endpoint on the nfdi4plants infrastructure. +The pipeline includes a `Update preview index` CI step that extracts metadata from the [yml frontmatter](#package-metadata) of every `.fsx` file in the staging area and (if tests and sanity checks pass) adds it to the `avpr-preview-index.json` on this repo's [preview-index](https://github.com/nfdi4plants/arc-validate-package-registry/releases/tag/preview-index) release -## Local development +## How to add packages -To run the `PackageRegistryService` locally, ideally use VisualStudio and run the `Docker Compose` project in Debug mode. This will launch the stack defined at [`docker-compose.yml`](docker-compose.yml), which includes: - -- the containerized `PackageRegistryService` application -- a `postgres` database seeded with the [latest indexed packages](src/PackageRegistryService/Data/arc-validate-package-index.json) -- an `adminer` instance for database management (will maybe be replaced by pgAdmin in the future) +To add a package, follow these steps: -## OpenAPI endpoint documentation via Swagger UI +- fork this repo +- add a new `.fsx` file to the respective folder in the [staging area](StagingArea/). For more info on the staging area structure, see [Validation package staging area](#validation-package-staging-area). +- commit it to the repo +- open a PR to the `main` branch of this repo -The `PackageRegistryService` has a built-in Swagger UI endpoint for API documentation. It is served at `/swagger/index.html`. +All packages in the staging area are automatically tested on every PR. Additionally, all packages set to `publish: true` in their yml frontmatter will be pushed to the registry service if they pass all tests and are not already present in the registry (see [Package publication workflow](#package-publication-workflow) for more info). -# Setup +## Versioning packages -## package indexing +Packages SHOULD be versioned according to the [semantic versioning](https://semver.org/) standard. This means that the version number of a package should be incremented according to the following rules: -To install the pre-commit hook needed for automatic package indexing, run either `setup.cmd` or `setup.sh` depending on your platform to install the pre-commit hook. +- **Major version**: incremented when you make changes incompatible with previous versions +- **Minor version**: incremented when you add functionality in a backwards-compatible manner +- **Patch version**: incremented when you make backwards-compatible bug fixes -## local development +## Package publication workflow -install the following prerequisites: -- .NET 8 SDK -- Docker -- Docker Compose +Publishing a package to the registry is a multi-step process: -# How to add packages +Suppose you want to develop version 1.0.0 of a package called `my-package`. -To add a package to the staging area, make sure that you installed the pre-commit hook as described in the [Setup](#setup) section. +1. Add a new blank `my-package@1.0.0.fsx` file to the [staging area](./StagingArea/) in the folder `my-package`. +2. Develop the package, using a work-in-process pull request to use this repository's CI to perform automated integrity tests on it. +3. Once the package is ready, add `publish: true` to the yml frontmatter of the package file. This will trigger the CI to build and push the package to the registry once the PR is merged. +4. Once a package is published, it cannot be unpublished or changed. To update a package, create a new script with the same name and a higher version number. -Then, simply add a new `.fsx` file to the [staging area](StagingArea/), and commit it to the repo. The pre-commit hook will automatically add the new package to the package index. +| stage | availability | mutability | +| --- | --- | --- | +| staging: development in this repo | version of current HEAD commit in this repo via github API-based execution in `arc-validate` CLI | any changes are allowed | +| published: available in the registry | version of the published package via the registry API | no changes are allowed | -All packages in the staging area are automatically tested on every commit. Additionally, all packages set to `publish: true` in their yml frontmatter will be pushed to the registry service if they pass all tests and are not already present in the registry. +[🔼 Back to top](#table-of-contents) -## Package metadata +# Package metadata Package metadata is extracted from **yml frontmatter** at the start of the `.fsx` file, indicated by a multiline comment (`(* ... *)`)containing the frontmatter fenced by `---` at its start and end: @@ -84,7 +148,7 @@ Package metadata is extracted from **yml frontmatter** at the start of the `.fsx *) ``` -### Mandatory fields +## Mandatory fields | Field | Type | Description | | --- | --- | --- | @@ -95,7 +159,8 @@ Package metadata is extracted from **yml frontmatter** at the start of the `.fsx | Summary | string | a single sentence description (<=50 words) of the package | | Description | string | an unconstrained free text description of the package | -example (only mandatory fields): +
+Example: only mandatory fields ```fsharp (* @@ -115,7 +180,9 @@ let doSomeValidation () = () doSomeValidation () ``` -### Optional fields +
+ +## Optional fields | Field | Type | Description | | --- | --- | --- | @@ -124,8 +191,8 @@ doSomeValidation () | Tags | string[] | a list of tags with optional ontology annotations that describe the package. For more information about mandatory and optional fields in this object, see [Objects > Tag](#tag) | | ReleaseNotes | string[] | a list of release notes for the package indicating changes from previous versions | - -example (all fields): +
+Example: all fields ```fsharp (* @@ -160,14 +227,15 @@ ReleaseNotes: | - does it well" --- *) - let doSomeValidation () = () doSomeValidation () ``` -#### Objects +
+ +### Objects -##### Author +#### Author Author metadata about the people that create and maintain the package. Note that the @@ -178,7 +246,7 @@ Author metadata about the people that create and maintain the package. Note that | Affiliation | string | the affiliation (e.g. institution) of the author | no | | AffiliationLink | string | a link to the affiliation of the author | no | -##### Tag +#### Tag Tags can be any string with an optional ontology annotation from a controlled vocabulary: @@ -188,20 +256,37 @@ Tags can be any string with an optional ontology annotation from a controlled vo | TermSourceREF | string | Reference to a controlled vocabulary source | no | | TermAccessionNumber | string | Accession in the referenced controlled vocabulary source | no | -### Package publication workflow +[🔼 Back to top](#table-of-contents) -Publishing a package to the registry is a multi-step process: +# Web API (PackageRegistryService) -Suppose you want to develop version 1.0.0 of a package called `my-package`. +The `PackageRegistryService` project located in `/src` is a simple ASP.NET Core (8) web API that serves validation packages and/or associated metadata via a few endpoints. -1. Add a new blank `my-package@1.0.0.fsx` file to the [staging area](./StagingArea/) in the folder `my-package`. -2. Develop the package, using this repositories CI to perform automates integrity tests on it. -3. Once the package is ready, add `publish: true` to the yml frontmatter of the package file. This will trigger the CI to build and push the package to the registry. -4. Once a package is published, it cannot be unpublished or changed. To update a package, create a new script with the same name and a higher version number. +It is developed specifically for containerization and use in a docker environment. -| stage | availability | mutability | -| --- | --- | --- | -| staging: development in this repo | version of current HEAD commit in this repo via github API-based execution in `arc-validate` CLI | any changes are allowed | -| published: available in the registry | version of the published package via the registry API | no changes are allowed | +The service will eventually be continuously deployed to a public endpoint on the nfdi4plants infrastructure. + +## Local development + +To run the `PackageRegistryService` locally, ideally use VisualStudio and run the `Docker Compose` project in Debug mode. This will launch the stack defined at [`docker-compose.yml`](docker-compose.yml), which includes: + +- the containerized `PackageRegistryService` application +- a `postgres` database seeded with the [latest indexed packages](src/PackageRegistryService/Data/arc-validate-package-index.json) +- an `adminer` instance for database management (will maybe be replaced by pgAdmin in the future) + +## OpenAPI endpoint documentation via Swagger UI + +The `PackageRegistryService` has a built-in Swagger UI endpoint for API documentation. It is served at `/swagger/index.html`. + +[🔼 Back to top](#table-of-contents) + +# Repository setup + +## local development + +install the following prerequisites: +- .NET 8 SDK +- Docker +- Docker Compose -### Versioning packages \ No newline at end of file +[🔼 Back to top](#table-of-contents) \ No newline at end of file