From 523f76b594416cd431b2f89e0221f342d285bf07 Mon Sep 17 00:00:00 2001 From: Kevin Schneider Date: Tue, 12 Mar 2024 17:13:38 +0100 Subject: [PATCH] Move staging area into repo root and adapt accordingly: - add index package tests - release a new version of the index package with tested functionality exposed - this commit is quite complicated and will test all current ci pipeline capabilities --- .github/workflows/pipeline.yml | 4 +- README.md | 6 +- .../invenio/invenio@1.0.0.fsx | 0 .../pride/pride@1.0.0.fsx | 0 .../test/test@0.0.1.fsx | 0 .../test/test@1.0.0.fsx | 0 .../test/test@1.0.1.fsx | 0 .../test/test@2.0.0.fsx | 0 .../test/test@3.0.0.fsx | 0 StagingAreaTests/ReferenceObjects.fs | 2 +- StagingAreaTests/StagingAreaTests.fsproj | 3 +- StagingAreaTests/Utils.fs | 16 -- arc-validate-package-registry.sln | 8 +- scripts/pre-publish-checks.fsx | 1 + src/AVPRClient/AVPRClient.csproj | 5 +- src/AVPRIndex/Frontmatter.fs | 67 ++++-- src/AVPRIndex/Globals.fs | 3 +- src/AVPRIndex/RELEASE_NOTES.md | 2 + .../Data/DataInitializer.cs | 24 +- .../Data/arc-validate-package-index.json | 227 ------------------ .../PackageRegistryService.csproj | 6 +- tests/Common/Common.fsproj | 29 +++ tests/Common/Program.fs | 1 + .../{IndexTests/Tests.fs => Common/Utils.fs} | 0 tests/IndexTests/AVPRRepoTests.fs | 2 + tests/IndexTests/FrontmatterTests.fs | 68 ++++++ tests/IndexTests/IndexTests.fsproj | 15 +- tests/IndexTests/MetadataTests.fs | 52 ++++ tests/IndexTests/ReferenceObjects.fs | 213 ++++++++++++++++ tests/IndexTests/Utils.fs | 36 +++ .../IndexTests/ValidationPackageIndexTests.fs | 16 ++ tests/IndexTests/fixtures/invalid@0.0.fsx | 12 + tests/IndexTests/fixtures/valid@1.0.0.fsx | 13 + tests/IndexTests/fixtures/valid@2.0.0.fsx | 32 +++ 34 files changed, 571 insertions(+), 292 deletions(-) rename {src/PackageRegistryService/StagingArea => StagingArea}/invenio/invenio@1.0.0.fsx (100%) rename {src/PackageRegistryService/StagingArea => StagingArea}/pride/pride@1.0.0.fsx (100%) rename {src/PackageRegistryService/StagingArea => StagingArea}/test/test@0.0.1.fsx (100%) rename {src/PackageRegistryService/StagingArea => StagingArea}/test/test@1.0.0.fsx (100%) rename {src/PackageRegistryService/StagingArea => StagingArea}/test/test@1.0.1.fsx (100%) rename {src/PackageRegistryService/StagingArea => StagingArea}/test/test@2.0.0.fsx (100%) rename {src/PackageRegistryService/StagingArea => StagingArea}/test/test@3.0.0.fsx (100%) delete mode 100644 src/PackageRegistryService/Data/arc-validate-package-index.json create mode 100644 tests/Common/Common.fsproj create mode 100644 tests/Common/Program.fs rename tests/{IndexTests/Tests.fs => Common/Utils.fs} (100%) create mode 100644 tests/IndexTests/AVPRRepoTests.fs create mode 100644 tests/IndexTests/FrontmatterTests.fs create mode 100644 tests/IndexTests/MetadataTests.fs create mode 100644 tests/IndexTests/ReferenceObjects.fs create mode 100644 tests/IndexTests/Utils.fs create mode 100644 tests/IndexTests/ValidationPackageIndexTests.fs create mode 100644 tests/IndexTests/fixtures/invalid@0.0.fsx create mode 100644 tests/IndexTests/fixtures/valid@1.0.0.fsx create mode 100644 tests/IndexTests/fixtures/valid@2.0.0.fsx diff --git a/.github/workflows/pipeline.yml b/.github/workflows/pipeline.yml index d7e4448..ddb96cb 100644 --- a/.github/workflows/pipeline.yml +++ b/.github/workflows/pipeline.yml @@ -62,13 +62,11 @@ jobs: global: - '**' packages: - - src/PackageRegistryService/StagingArea/** + - StagingArea/** tests: - tests/** api: - src/PackageRegistryServive/** - - '!src/PackageRegistryServive/StagingArea/**' - - '!src/PackageRegistryServive/Data/**' client: - src/AVPRClient/** index: diff --git a/README.md b/README.md index 2b1edfb..e8a6a75 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ This repository contains: ## The package index -This repo runs a [custom pre-commit hook](pre-commit.sh) that will run a [script](./update-index.fsx) automatically add any `.fsx` file in the [staging area](src/PackageRegistryService/StagingArea/) to [the package index](src/PackageRegistryService/Data/arc-validate-package-index.json) when it is commited to the repo. +This repo runs a [custom pre-commit hook](pre-commit.sh) that will run a [script](./update-index.fsx) automatically add any `.fsx` file in the [staging area](StagingArea/) to [the package index](src/PackageRegistryService/Data/arc-validate-package-index.json) when it is commited to the repo. ## Automated package testing @@ -68,7 +68,7 @@ install the following prerequisites: To add a package to the staging area, make sure that you installed the pre-commit hook as described in the [Setup](#setup) section. -Then, simply add a new `.fsx` file to the [staging area](src/PackageRegistryService/StagingArea/), and commit it to the repo. The pre-commit hook will automatically add the new package to the package index. +Then, simply add a new `.fsx` file to the [staging area](StagingArea/), and commit it to the repo. The pre-commit hook will automatically add the new package to the package index. All packages in the staging area are automatically tested on every commit. Additionally, all packages set to `publish: true` in their yml frontmatter will be pushed to the registry service if they pass all tests and are not already present in the registry. @@ -194,7 +194,7 @@ Publishing a package to the registry is a multi-step process: Suppose you want to develop version 1.0.0 of a package called `my-package`. -1. Add a new blank `my-package@1.0.0.fsx` file to the [staging area](./src/PackageRegistryService/StagingArea/) in the folder `my-package`. +1. Add a new blank `my-package@1.0.0.fsx` file to the [staging area](./StagingArea/) in the folder `my-package`. 2. Develop the package, using this repositories CI to perform automates integrity tests on it. 3. Once the package is ready, add `publish: true` to the yml frontmatter of the package file. This will trigger the CI to build and push the package to the registry. 4. Once a package is published, it cannot be unpublished or changed. To update a package, create a new script with the same name and a higher version number. diff --git a/src/PackageRegistryService/StagingArea/invenio/invenio@1.0.0.fsx b/StagingArea/invenio/invenio@1.0.0.fsx similarity index 100% rename from src/PackageRegistryService/StagingArea/invenio/invenio@1.0.0.fsx rename to StagingArea/invenio/invenio@1.0.0.fsx diff --git a/src/PackageRegistryService/StagingArea/pride/pride@1.0.0.fsx b/StagingArea/pride/pride@1.0.0.fsx similarity index 100% rename from src/PackageRegistryService/StagingArea/pride/pride@1.0.0.fsx rename to StagingArea/pride/pride@1.0.0.fsx diff --git a/src/PackageRegistryService/StagingArea/test/test@0.0.1.fsx b/StagingArea/test/test@0.0.1.fsx similarity index 100% rename from src/PackageRegistryService/StagingArea/test/test@0.0.1.fsx rename to StagingArea/test/test@0.0.1.fsx diff --git a/src/PackageRegistryService/StagingArea/test/test@1.0.0.fsx b/StagingArea/test/test@1.0.0.fsx similarity index 100% rename from src/PackageRegistryService/StagingArea/test/test@1.0.0.fsx rename to StagingArea/test/test@1.0.0.fsx diff --git a/src/PackageRegistryService/StagingArea/test/test@1.0.1.fsx b/StagingArea/test/test@1.0.1.fsx similarity index 100% rename from src/PackageRegistryService/StagingArea/test/test@1.0.1.fsx rename to StagingArea/test/test@1.0.1.fsx diff --git a/src/PackageRegistryService/StagingArea/test/test@2.0.0.fsx b/StagingArea/test/test@2.0.0.fsx similarity index 100% rename from src/PackageRegistryService/StagingArea/test/test@2.0.0.fsx rename to StagingArea/test/test@2.0.0.fsx diff --git a/src/PackageRegistryService/StagingArea/test/test@3.0.0.fsx b/StagingArea/test/test@3.0.0.fsx similarity index 100% rename from src/PackageRegistryService/StagingArea/test/test@3.0.0.fsx rename to StagingArea/test/test@3.0.0.fsx diff --git a/StagingAreaTests/ReferenceObjects.fs b/StagingAreaTests/ReferenceObjects.fs index c5a90bb..9844e5e 100644 --- a/StagingAreaTests/ReferenceObjects.fs +++ b/StagingAreaTests/ReferenceObjects.fs @@ -21,7 +21,7 @@ let all_staged_packages_contents = all_staged_packages_paths |> Array.map (fun p -> p, - p |> (File.ReadAllText >> fun x -> x.ReplaceLineEndings()) + p |> (File.ReadAllText >> fun x -> x.ReplaceLineEndings("\n")) ) let all_staged_packages_metadata = diff --git a/StagingAreaTests/StagingAreaTests.fsproj b/StagingAreaTests/StagingAreaTests.fsproj index 15be53f..4bc587c 100644 --- a/StagingAreaTests/StagingAreaTests.fsproj +++ b/StagingAreaTests/StagingAreaTests.fsproj @@ -10,7 +10,7 @@ - + @@ -35,6 +35,7 @@ + diff --git a/StagingAreaTests/Utils.fs b/StagingAreaTests/Utils.fs index 4be820d..e2f44ef 100644 --- a/StagingAreaTests/Utils.fs +++ b/StagingAreaTests/Utils.fs @@ -25,22 +25,6 @@ type Assert with script.Contains(Frontmatter.frontMatterEnd) ) - static member MetadataValid(m: ValidationPackageMetadata) = - //test wether all required fields are present - Assert.NotNull(m) - Assert.NotNull(m.Name) - Assert.NotEqual(m.Name, "") - Assert.NotNull(m.Summary) - Assert.NotEqual(m.Summary, "") - Assert.NotNull(m.Description) - Assert.NotEqual(m.Description, "") - Assert.NotNull(m.MajorVersion) - Assert.True(m.MajorVersion >= 0) - Assert.NotNull(m.MinorVersion) - Assert.True(m.MinorVersion >= 0) - Assert.NotNull(m.PatchVersion) - Assert.True(m.PatchVersion >= 0) - static member FileNameValid(path:string) = let fileName = Path.GetFileName(path) let folderName = Path.GetDirectoryName(path) |> Path.GetFileName diff --git a/arc-validate-package-registry.sln b/arc-validate-package-registry.sln index e358137..53af25a 100644 --- a/arc-validate-package-registry.sln +++ b/arc-validate-package-registry.sln @@ -23,10 +23,9 @@ Project("{E53339B2-1760-4266-BCC7-CA923CBCF16C}") = "docker-compose", "docker-co EndProject Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = ".ci", ".ci", "{24F7CF58-94B9-4FB4-8A59-FD329B3C431D}" ProjectSection(SolutionItems) = preProject - .github\workflows\build-and-test.yml = .github\workflows\build-and-test.yml + .github\workflows\build-and-test-solution.yml = .github\workflows\build-and-test-solution.yml .github\workflows\pipeline.yml = .github\workflows\pipeline.yml - .github\workflows\publish-pending-packages.yml = .github\workflows\publish-pending-packages.yml - .github\workflows\update-docker-image.yml = .github\workflows\update-docker-image.yml + .github\workflows\release-package.yml = .github\workflows\release-package.yml EndProjectSection EndProject Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "AVPRClient", "src\AVPRClient\AVPRClient.csproj", "{D1FABAC1-D0F2-4F6C-B975-236E9969FB38}" @@ -41,6 +40,9 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "APITests", "tests\APITests\ EndProject Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "scripts", "scripts", "{C4DBD5F6-4A14-44F4-9E5A-AC6B72AAEB81}" ProjectSection(SolutionItems) = preProject + scripts\domain.fsx = scripts\domain.fsx + scripts\generate-preview-index.fsx = scripts\generate-preview-index.fsx + scripts\pre-publish-checks.fsx = scripts\pre-publish-checks.fsx scripts\publish-pending-packages.fsx = scripts\publish-pending-packages.fsx scripts\update-index.fsx = scripts\update-index.fsx EndProjectSection diff --git a/scripts/pre-publish-checks.fsx b/scripts/pre-publish-checks.fsx index 3779e83..1791cff 100644 --- a/scripts/pre-publish-checks.fsx +++ b/scripts/pre-publish-checks.fsx @@ -16,6 +16,7 @@ let current_preview_index = |> Request.send |> Response.deserializeJson with e as exn -> + printfn $"Failed to fetch current preview index: {exn.Message}" [||] let all_packages_in_staging_area = diff --git a/src/AVPRClient/AVPRClient.csproj b/src/AVPRClient/AVPRClient.csproj index e544d39..da01e55 100644 --- a/src/AVPRClient/AVPRClient.csproj +++ b/src/AVPRClient/AVPRClient.csproj @@ -24,13 +24,10 @@ README.md 0.0.4 - + - - - diff --git a/src/AVPRIndex/Frontmatter.fs b/src/AVPRIndex/Frontmatter.fs index 8c9d278..1ebae4c 100644 --- a/src/AVPRIndex/Frontmatter.fs +++ b/src/AVPRIndex/Frontmatter.fs @@ -1,6 +1,5 @@ namespace AVPRIndex - open Domain open System open System.IO @@ -9,8 +8,26 @@ open YamlDotNet.Serialization module Frontmatter = - let frontMatterStart = $"(*{System.Environment.NewLine}---" - let frontMatterEnd = $"---{System.Environment.NewLine}*)" + let [] frontMatterStart = "(*\n---" + let [] frontMatterEnd = "---\n*)" + + let containsFrontmatter (str: string) = + str.StartsWith(frontMatterStart, StringComparison.Ordinal) && str.Contains(frontMatterEnd) + + let tryExtractFromString (str: string) = + let norm = str.ReplaceLineEndings("\n") + if containsFrontmatter norm then + norm.Substring( + frontMatterStart.Length, + (norm.IndexOf(frontMatterEnd, StringComparison.Ordinal) - frontMatterEnd.Length)) + |> Some + else + None + + let extractFromString (str: string) = + match tryExtractFromString str with + | Some frontmatter -> frontmatter + | None -> failwith $"input has no correctly formatted frontmatter." let yamlDeserializer = DeserializerBuilder() @@ -19,24 +36,34 @@ module Frontmatter = type ValidationPackageMetadata with + static member extractFromString (str: string) = + let frontmatter = tryExtractFromString str + match frontmatter with + | Some frontmatter -> + let result = + yamlDeserializer.Deserialize(frontmatter) + result + | None -> + failwith $"string has no correctly formatted frontmatter." + + static member tryExtractFromString (str: string) = + try + ValidationPackageMetadata.extractFromString str |> Some + with e -> + printfn $"error parsing package metadata: {e.Message}" + None + static member extractFromScript (scriptPath: string) = - let script = File.ReadAllText(scriptPath).ReplaceLineEndings() - if script.StartsWith(frontMatterStart, StringComparison.Ordinal) && script.Contains(frontMatterEnd) then - let frontmatter = - script.Substring( - frontMatterStart.Length, - (script.IndexOf(frontMatterEnd, StringComparison.Ordinal) - frontMatterEnd.Length)) - try - let result = - yamlDeserializer.Deserialize(frontmatter) - result - with e as exn -> - printfn $"error parsing package metadata at {scriptPath}. Make sure that all required metadata tags are included." - printfn $"Error msg: {e.Message}." - ValidationPackageMetadata() - else - printfn $"script at {scriptPath} has no correctly formatted frontmatter." - ValidationPackageMetadata() + scriptPath + |> File.ReadAllText + |> ValidationPackageMetadata.extractFromString + + static member tryExtractFromScript (scriptPath: string) = + try + ValidationPackageMetadata.extractFromScript scriptPath |> Some + with e -> + printfn $"error parsing package metadata: {e.Message}" + None type ValidationPackageIndex with diff --git a/src/AVPRIndex/Globals.fs b/src/AVPRIndex/Globals.fs index c88acfa..d698711 100644 --- a/src/AVPRIndex/Globals.fs +++ b/src/AVPRIndex/Globals.fs @@ -2,5 +2,4 @@ module Globals = - let [] STAGING_AREA_RELATIVE_PATH = "src/PackageRegistryService/StagingArea" - let [] PACKAGE_INDEX_RELATIVE_PATH = "src/PackageRegistryService/Data/arc-validate-package-index.json" \ No newline at end of file + let [] STAGING_AREA_RELATIVE_PATH = "StagingArea" \ No newline at end of file diff --git a/src/AVPRIndex/RELEASE_NOTES.md b/src/AVPRIndex/RELEASE_NOTES.md index c2646cf..0744999 100644 --- a/src/AVPRIndex/RELEASE_NOTES.md +++ b/src/AVPRIndex/RELEASE_NOTES.md @@ -1,6 +1,8 @@ ## v0.0.5 - Add `getPreviewIndex` function that downloads the currently released preview index from the github release. +- Add `ValidationPackageMetadata.extractFromString` to extract metadata from a string +- Refactor and expose some Frontmatter parsing functions ## v0.0.4 diff --git a/src/PackageRegistryService/Data/DataInitializer.cs b/src/PackageRegistryService/Data/DataInitializer.cs index 51ba1a1..fb551b5 100644 --- a/src/PackageRegistryService/Data/DataInitializer.cs +++ b/src/PackageRegistryService/Data/DataInitializer.cs @@ -8,25 +8,20 @@ using AVPRIndex; using static AVPRIndex.Domain; using static AVPRIndex.Frontmatter; +using System.Reflection; namespace PackageRegistryService.Data { public class DataInitializer { - public static List ReadIndex() - { - var json = File.ReadAllText(@"Data/arc-validate-package-index.json"); - var index = JsonSerializer.Deserialize>(json); - return index ?? []; - } public static void SeedData(ValidationPackageDb context) { MD5 md5 = MD5.Create(); if (!context.ValidationPackages.Any()) { - var index = ReadIndex(); + var index = AVPRRepo.getPreviewIndex(); context.SaveChanges(); @@ -34,7 +29,12 @@ public static void SeedData(ValidationPackageDb context) index .Select((i) => { - var content = File.ReadAllBytes($"StagingArea/{i.Metadata.Name}/{i.FileName}"); + var path = + Path.Combine( + Path.GetDirectoryName(Assembly.GetEntryAssembly().Location), + $"StagingArea/{i.Metadata.Name}/{i.FileName}" + ); + var content = File.ReadAllBytes(path); return new ValidationPackage { @@ -58,7 +58,13 @@ public static void SeedData(ValidationPackageDb context) index .Select((i) => { - var content = File.ReadAllBytes($"StagingArea/{i.Metadata.Name}/{i.FileName}"); + var path = + Path.Combine( + Path.GetDirectoryName(Assembly.GetEntryAssembly().Location), + $"StagingArea/{i.Metadata.Name}/{i.FileName}" + ); + var content = File.ReadAllBytes(path); + var hash = Convert.ToHexString(md5.ComputeHash(content)); if (hash != i.ContentHash) { diff --git a/src/PackageRegistryService/Data/arc-validate-package-index.json b/src/PackageRegistryService/Data/arc-validate-package-index.json deleted file mode 100644 index b0f0aad..0000000 --- a/src/PackageRegistryService/Data/arc-validate-package-index.json +++ /dev/null @@ -1,227 +0,0 @@ -[ - { - "RepoPath": "src/PackageRegistryService/StagingArea/invenio/invenio@1.0.0.fsx", - "FileName": "invenio@1.0.0.fsx", - "LastUpdated": "2024-02-29T17:07:34+01:00", - "ContentHash": "066FBA6F80023DA281AF3AEB7C66C9A4", - "Metadata": { - "Name": "invenio", - "Summary": "Validates if the ARC contains the necessary metadata to be publishable via Invenio.", - "Description": "Validates if the ARC contains the necessary metadata to be publishable via Invenio.\nThe following metadata is required:\n - Investigation has title and description\n - All persons in Investigation Contacts must have a name, last name, affiliation and valid email\n", - "MajorVersion": 1, - "MinorVersion": 0, - "PatchVersion": 0, - "Publish": true, - "Authors": [ - { - "FullName": "Oliver Maus", - "Email": "", - "Affiliation": "DataPLANT", - "AffiliationLink": "" - } - ], - "Tags": [ - { - "Name": "ARC", - "TermSourceREF": "", - "TermAccessionNumber": "" - }, - { - "Name": "data publication", - "TermSourceREF": "", - "TermAccessionNumber": "" - } - ], - "ReleaseNotes": "Initial release" - } - }, - { - "RepoPath": "src/PackageRegistryService/StagingArea/pride/pride@1.0.0.fsx", - "FileName": "pride@1.0.0.fsx", - "LastUpdated": "2024-03-11T10:29:24+01:00", - "ContentHash": "EE4F9BD7D0249F272B79BAAD9DA02C0C", - "Metadata": { - "Name": "pride", - "Summary": "Validates if the ARC contains the necessary metadata to be publishable via PRIDE.", - "Description": "Validates if the ARC contains the necessary metadata to be publishable via PRIDE.\nThe following metadata is required:\n - Investigation has title and description\n - Investigation has Keywords comment in correct format\n - All persons in Investigation Contacts must have a first name, last name, affiliation and valid email\n - Study has protocol, tissue \u0026 species in correct format\n - Assay has protocol, technology type, instrument model, and fixed and/or variable modification in correct format\n", - "MajorVersion": 1, - "MinorVersion": 0, - "PatchVersion": 0, - "Publish": false, - "Authors": [ - { - "FullName": "Oliver Maus", - "Email": "maus@nfdi4plants.org", - "Affiliation": "RPTU Kaiserslautern", - "AffiliationLink": "http://rptu.de/startseite" - } - ], - "Tags": [ - { - "Name": "validation", - "TermSourceREF": "", - "TermAccessionNumber": "" - }, - { - "Name": "pride", - "TermSourceREF": "", - "TermAccessionNumber": "" - }, - { - "Name": "proteomics", - "TermSourceREF": "", - "TermAccessionNumber": "" - } - ], - "ReleaseNotes": "- initial release\n- metadata validation added:\n - Investigation has title and description\n - Investigation has Keywords comment in correct format\n - All persons in Investigation Contacts must have a first name, last name, affiliation and valid email\n - Study has protocol, tissue \u0026 species in correct format\n - Assay has protocol, technology type, instrument model, and fixed and/or variable modification in correct format\n" - } - }, - { - "RepoPath": "src/PackageRegistryService/StagingArea/test/test@0.0.1.fsx", - "FileName": "test@0.0.1.fsx", - "LastUpdated": "2024-03-08T11:37:58+01:00", - "ContentHash": "4030CC13BFD73E08144EF8910622786E", - "Metadata": { - "Name": "test", - "Summary": "this package is here for testing purposes only.", - "Description": "this package is here for testing purposes only.", - "MajorVersion": 0, - "MinorVersion": 0, - "PatchVersion": 1, - "Publish": true, - "Authors": [], - "Tags": [], - "ReleaseNotes": "" - } - }, - { - "RepoPath": "src/PackageRegistryService/StagingArea/test/test@1.0.0.fsx", - "FileName": "test@1.0.0.fsx", - "LastUpdated": "2024-02-29T17:07:34+01:00", - "ContentHash": "3E129B7298327351BAB43B72E70503EA", - "Metadata": { - "Name": "test", - "Summary": "this package is here for testing purposes only.", - "Description": "this package is here for testing purposes only.", - "MajorVersion": 1, - "MinorVersion": 0, - "PatchVersion": 0, - "Publish": true, - "Authors": [], - "Tags": [], - "ReleaseNotes": "" - } - }, - { - "RepoPath": "src/PackageRegistryService/StagingArea/test/test@1.0.1.fsx", - "FileName": "test@1.0.1.fsx", - "LastUpdated": "2024-02-29T17:07:34+01:00", - "ContentHash": "0141BBA7221725EAE90C9817447EA039", - "Metadata": { - "Name": "test", - "Summary": "this package is here for testing purposes only.", - "Description": "this package is here for testing purposes only.", - "MajorVersion": 1, - "MinorVersion": 0, - "PatchVersion": 1, - "Publish": true, - "Authors": [], - "Tags": [], - "ReleaseNotes": "" - } - }, - { - "RepoPath": "src/PackageRegistryService/StagingArea/test/test@2.0.0.fsx", - "FileName": "test@2.0.0.fsx", - "LastUpdated": "2024-02-29T17:07:34+01:00", - "ContentHash": "804D5FF5CE4D7266F5D1BDBA4CEF6B3A", - "Metadata": { - "Name": "test", - "Summary": "this package is here for testing purposes only.", - "Description": "this package is here for testing purposes only.", - "MajorVersion": 2, - "MinorVersion": 0, - "PatchVersion": 0, - "Publish": true, - "Authors": [ - { - "FullName": "John Doe", - "Email": "j@d.com", - "Affiliation": "University of Nowhere", - "AffiliationLink": "https://nowhere.edu" - }, - { - "FullName": "Jane Doe", - "Email": "jj@d.com", - "Affiliation": "University of Somewhere", - "AffiliationLink": "https://somewhere.edu" - } - ], - "Tags": [ - { - "Name": "validation", - "TermSourceREF": "", - "TermAccessionNumber": "" - }, - { - "Name": "my-package", - "TermSourceREF": "", - "TermAccessionNumber": "" - }, - { - "Name": "thing", - "TermSourceREF": "", - "TermAccessionNumber": "" - } - ], - "ReleaseNotes": "add authors and tags for further testing" - } - }, - { - "RepoPath": "src/PackageRegistryService/StagingArea/test/test@3.0.0.fsx", - "FileName": "test@3.0.0.fsx", - "LastUpdated": "2024-02-29T17:07:34+01:00", - "ContentHash": "618DC951075C63B2FDCE12450A584190", - "Metadata": { - "Name": "test", - "Summary": "this package is here for testing purposes only.", - "Description": "this package is here for testing purposes only.", - "MajorVersion": 3, - "MinorVersion": 0, - "PatchVersion": 0, - "Publish": true, - "Authors": [ - { - "FullName": "John Doe", - "Email": "j@d.com", - "Affiliation": "University of Nowhere", - "AffiliationLink": "https://nowhere.edu" - }, - { - "FullName": "Jane Doe", - "Email": "jj@d.com", - "Affiliation": "University of Somewhere", - "AffiliationLink": "https://somewhere.edu" - } - ], - "Tags": [ - { - "Name": "validation", - "TermSourceREF": "", - "TermAccessionNumber": "" - }, - { - "Name": "my-package", - "TermSourceREF": "", - "TermAccessionNumber": "" - }, - { - "Name": "thing", - "TermSourceREF": "", - "TermAccessionNumber": "" - } - ], - "ReleaseNotes": "add authors and tags for further testing" - } - } -] \ No newline at end of file diff --git a/src/PackageRegistryService/PackageRegistryService.csproj b/src/PackageRegistryService/PackageRegistryService.csproj index 4e0d392..4ed3d76 100644 --- a/src/PackageRegistryService/PackageRegistryService.csproj +++ b/src/PackageRegistryService/PackageRegistryService.csproj @@ -15,7 +15,11 @@ CS1591;$(NoWarn) - + + + + + diff --git a/tests/Common/Common.fsproj b/tests/Common/Common.fsproj new file mode 100644 index 0000000..6edcbbb --- /dev/null +++ b/tests/Common/Common.fsproj @@ -0,0 +1,29 @@ + + + + net8.0 + + false + false + true + + + + + + + + + + + + runtime; build; native; contentfiles; analyzers; buildtransitive + all + + + runtime; build; native; contentfiles; analyzers; buildtransitive + all + + + + diff --git a/tests/Common/Program.fs b/tests/Common/Program.fs new file mode 100644 index 0000000..0695f84 --- /dev/null +++ b/tests/Common/Program.fs @@ -0,0 +1 @@ +module Program = let [] main _ = 0 diff --git a/tests/IndexTests/Tests.fs b/tests/Common/Utils.fs similarity index 100% rename from tests/IndexTests/Tests.fs rename to tests/Common/Utils.fs diff --git a/tests/IndexTests/AVPRRepoTests.fs b/tests/IndexTests/AVPRRepoTests.fs new file mode 100644 index 0000000..9d6cadd --- /dev/null +++ b/tests/IndexTests/AVPRRepoTests.fs @@ -0,0 +1,2 @@ +namespace AVPRRepoTests + diff --git a/tests/IndexTests/FrontmatterTests.fs b/tests/IndexTests/FrontmatterTests.fs new file mode 100644 index 0000000..890123b --- /dev/null +++ b/tests/IndexTests/FrontmatterTests.fs @@ -0,0 +1,68 @@ +namespace FrontmatterTests + +open System +open Xunit +open AVPRIndex +open ReferenceObjects + +module InMemory = + + [] + let ``valid frontmatter capture guides lead to results`` () = + Assert.All( + [Frontmatter.validMandatoryFrontmatter; Frontmatter.validFullFrontmatter; Frontmatter.invalidMissingMandatoryFrontmatter], + (fun fm -> + Assert.True ((Frontmatter.tryExtractFromString fm).IsSome) + ) + ) + + [] + let ``valid frontmatter capture guides lead to correctly extracted substrings`` () = + Assert.All( + [ + Frontmatter.validMandatoryFrontmatter, Frontmatter.validMandatoryFrontmatterExtracted + Frontmatter.validFullFrontmatter, Frontmatter.validFullFrontmatterExtracted + Frontmatter.invalidMissingMandatoryFrontmatter, Frontmatter.invalidMissingMandatoryFrontmatterExtracted + ], + (fun (fm, expected) -> + let actual = Frontmatter.extractFromString fm + Assert.Equal(expected, actual) + ) + ) + + [] + let ``invalid frontmatter capture substrings are leading to None`` () = + Assert.All( + [Frontmatter.invalidEndFrontmatter; Frontmatter.invalidStartFrontmatter], + (fun fm -> + Assert.True ((Frontmatter.tryExtractFromString fm).IsNone) + ) + ) + +module IO = + + open System.IO + + [] + let ``valid frontmatter substring is extracted from valid mandatory field test file`` () = + + let actual = File.ReadAllText("fixtures/valid@1.0.0.fsx") |> Frontmatter.tryExtractFromString + + Assert.True actual.IsSome + Assert.Equal (Frontmatter.validMandatoryFrontmatterExtracted, actual.Value) + + [] + let ``valid frontmatter substring is correctly from all fields test file`` () = + + let actual = File.ReadAllText("fixtures/valid@2.0.0.fsx") |> Frontmatter.tryExtractFromString + + Assert.True actual.IsSome + Assert.Equal (Frontmatter.validFullFrontmatterExtracted, actual.Value) + + [] + let ``frontmatter substring is extracted although metadata is missing fields`` () = + + let actual = File.ReadAllText("fixtures/invalid@0.0.fsx") |> Frontmatter.tryExtractFromString + + Assert.True actual.IsSome + Assert.Equal (Frontmatter.invalidMissingMandatoryFrontmatterExtracted, actual.Value) diff --git a/tests/IndexTests/IndexTests.fsproj b/tests/IndexTests/IndexTests.fsproj index 6edcbbb..65d939d 100644 --- a/tests/IndexTests/IndexTests.fsproj +++ b/tests/IndexTests/IndexTests.fsproj @@ -1,4 +1,4 @@ - + net8.0 @@ -9,7 +9,14 @@ - + + + + + + + + @@ -26,4 +33,8 @@ + + + + diff --git a/tests/IndexTests/MetadataTests.fs b/tests/IndexTests/MetadataTests.fs new file mode 100644 index 0000000..0536650 --- /dev/null +++ b/tests/IndexTests/MetadataTests.fs @@ -0,0 +1,52 @@ +namespace MetadataTests + +open System +open Xunit +open AVPRIndex +open AVPRIndex.Domain +open AVPRIndex.Frontmatter +open ReferenceObjects +open Utils + +module InMemory = + + [] + let ``valid metadata is extracted from valid frontmatter`` () = + Assert.All( + [ + Frontmatter.validMandatoryFrontmatter, Metadata.validMandatoryFrontmatter + Frontmatter.validFullFrontmatter, Metadata.validFullFrontmatter + ], + (fun (fm, expected) -> + let actual = ValidationPackageMetadata.extractFromString fm + Assert.Equivalent(expected, actual) + ) + ) + +module IO = + + open System.IO + + [] + let ``valid metadata is extracted from valid mandatory field test file`` () = + + let actual = File.ReadAllText("fixtures/valid@1.0.0.fsx") |> ValidationPackageMetadata.extractFromString + + Assert.MetadataValid(actual) + Assert.Equivalent(Metadata.validMandatoryFrontmatter, actual) + + [] + let ``valid metadata is extracted from all fields test file`` () = + + let actual = File.ReadAllText("fixtures/valid@2.0.0.fsx") |> ValidationPackageMetadata.extractFromString + + Assert.MetadataValid(actual) + Assert.Equivalent(Metadata.validFullFrontmatter, actual) + + [] + let ``invalid metadata is extracted from testfile with missing fields`` () = + + let actual = File.ReadAllText("fixtures/invalid@0.0.fsx") |> ValidationPackageMetadata.extractFromString + + Assert.ThrowsAny(fun () -> Assert.MetadataValid(actual)) |> ignore + Assert.Equivalent(Metadata.invalidMissingMandatoryFrontmatter, actual) \ No newline at end of file diff --git a/tests/IndexTests/ReferenceObjects.fs b/tests/IndexTests/ReferenceObjects.fs new file mode 100644 index 0000000..c048139 --- /dev/null +++ b/tests/IndexTests/ReferenceObjects.fs @@ -0,0 +1,213 @@ +module ReferenceObjects + +open AVPRIndex +open AVPRIndex.Domain + +module Frontmatter = + + let validMandatoryFrontmatter = """(* +--- +Name: valid +MajorVersion: 1 +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell. +--- +*)""" .ReplaceLineEndings("\n") + + let validMandatoryFrontmatterExtracted = """ +Name: valid +MajorVersion: 1 +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell. +""" .ReplaceLineEndings("\n") + + + let validFullFrontmatter = """(* +--- +Name: valid +MajorVersion: 2 +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell. +Publish: true +Authors: + - FullName: John Doe + Email: j@d.com + Affiliation: University of Nowhere + AffiliationLink: https://nowhere.edu + - FullName: Jane Doe + Email: jj@d.com + Affiliation: University of Somewhere + AffiliationLink: https://somewhere.edu +Tags: + - Name: validation + - Name: my-tag + TermSourceREF: my-ontology + TermAccessionNumber: MO:12345 +ReleaseNotes: | + - initial release + - does the thing + - does it well +--- +*)""" .ReplaceLineEndings("\n") + + let validFullFrontmatterExtracted = """ +Name: valid +MajorVersion: 2 +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell. +Publish: true +Authors: + - FullName: John Doe + Email: j@d.com + Affiliation: University of Nowhere + AffiliationLink: https://nowhere.edu + - FullName: Jane Doe + Email: jj@d.com + Affiliation: University of Somewhere + AffiliationLink: https://somewhere.edu +Tags: + - Name: validation + - Name: my-tag + TermSourceREF: my-ontology + TermAccessionNumber: MO:12345 +ReleaseNotes: | + - initial release + - does the thing + - does it well +""" .ReplaceLineEndings("\n") + + let invalidStartFrontmatter = """( +--- +Name: invalid +MajorVersion: 1 +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell. +--- +*)""" .ReplaceLineEndings("\n") + + let invalidEndFrontmatter = """(* +--- +Name: invalid +MajorVersion: 1 +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell.---*)""".ReplaceLineEndings("\n") + + let invalidMissingMandatoryFrontmatter = """(* +--- +Name: invalid +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell. +--- +*)""" .ReplaceLineEndings("\n") + + let invalidMissingMandatoryFrontmatterExtracted = """ +Name: invalid +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell. +""" .ReplaceLineEndings("\n") + +module Metadata = + + + let validMandatoryFrontmatter = + + ValidationPackageMetadata( + Name = "valid", + MajorVersion = 1, + MinorVersion = 0, + PatchVersion = 0, + Summary = "My package does the thing.", + Description = """My package does the thing. +It does it very good, it does it very well. +It does it very fast, it does it very swell. +""".ReplaceLineEndings("\n") + ) + + let validFullFrontmatter = + ValidationPackageMetadata( + Name = "valid", + MajorVersion = 2, + MinorVersion = 0, + PatchVersion = 0, + Summary = "My package does the thing.", + Publish = true, + Description = """My package does the thing. +It does it very good, it does it very well. +It does it very fast, it does it very swell. +""".ReplaceLineEndings("\n"), + Authors = [| + Author( + FullName = "John Doe", + Email = "j@d.com", + Affiliation = "University of Nowhere", + AffiliationLink = "https://nowhere.edu" + ) + Author( + FullName = "Jane Doe", + Email = "jj@d.com", + Affiliation = "University of Somewhere", + AffiliationLink = "https://somewhere.edu" + ) + |], + Tags = [| + OntologyAnnotation(Name = "validation") + OntologyAnnotation(Name = "my-tag", TermSourceREF = "my-ontology", TermAccessionNumber = "MO:12345") + |], + ReleaseNotes = """- initial release + - does the thing + - does it well +""".ReplaceLineEndings("\n") + ) + + let invalidMissingMandatoryFrontmatter = + ValidationPackageMetadata( + Name = "invalid", + MajorVersion = -1, + MinorVersion = 0, + PatchVersion = 0, + Summary = "My package does the thing.", + Description = """My package does the thing. +It does it very good, it does it very well. +It does it very fast, it does it very swell. +""".ReplaceLineEndings("\n") + ) \ No newline at end of file diff --git a/tests/IndexTests/Utils.fs b/tests/IndexTests/Utils.fs new file mode 100644 index 0000000..90572bc --- /dev/null +++ b/tests/IndexTests/Utils.fs @@ -0,0 +1,36 @@ +module Utils + +open AVPRIndex +open AVPRIndex.Domain +open Xunit +open System +open System.IO + +type Assert with + static member MetadataValid(m: ValidationPackageMetadata) = + //test wether all required fields are present + Assert.NotNull(m) + Assert.NotNull(m.Name) + Assert.NotEqual(m.Name, "") + Assert.NotNull(m.Summary) + Assert.NotEqual(m.Summary, "") + Assert.NotNull(m.Description) + Assert.NotEqual(m.Description, "") + Assert.NotNull(m.MajorVersion) + Assert.True(m.MajorVersion >= 0) + Assert.NotNull(m.MinorVersion) + Assert.True(m.MinorVersion >= 0) + Assert.NotNull(m.PatchVersion) + Assert.True(m.PatchVersion >= 0) + + static member MetadataEqual(expected: ValidationPackageMetadata, actual: ValidationPackageMetadata) = + Assert.Equal(expected.Name, actual.Name) + Assert.Equal(expected.Summary, actual.Summary) + Assert.Equal(expected.MajorVersion, actual.MajorVersion) + Assert.Equal(expected.MinorVersion, actual.MinorVersion) + Assert.Equal(expected.PatchVersion, actual.PatchVersion) + Assert.Equal(expected.Publish, actual.Publish) + Assert.Equivalent(expected.Authors, actual.Authors, strict = true) + Assert.Equivalent(expected.Tags, actual.Tags, strict = true) + Assert.Equal(expected.ReleaseNotes, actual.ReleaseNotes) + Assert.Equivalent(expected, actual, strict = true) \ No newline at end of file diff --git a/tests/IndexTests/ValidationPackageIndexTests.fs b/tests/IndexTests/ValidationPackageIndexTests.fs new file mode 100644 index 0000000..bd84034 --- /dev/null +++ b/tests/IndexTests/ValidationPackageIndexTests.fs @@ -0,0 +1,16 @@ +namespace ValidationPackageIndexTests + +open System +open Xunit + +module InMemory = + + [] + let ``My test`` () = + Assert.True(true) + +module IO = + + [] + let ``My test`` () = + Assert.True(true) \ No newline at end of file diff --git a/tests/IndexTests/fixtures/invalid@0.0.fsx b/tests/IndexTests/fixtures/invalid@0.0.fsx new file mode 100644 index 0000000..67298a9 --- /dev/null +++ b/tests/IndexTests/fixtures/invalid@0.0.fsx @@ -0,0 +1,12 @@ +(* +--- +Name: invalid +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell. +--- +*) \ No newline at end of file diff --git a/tests/IndexTests/fixtures/valid@1.0.0.fsx b/tests/IndexTests/fixtures/valid@1.0.0.fsx new file mode 100644 index 0000000..86b90bf --- /dev/null +++ b/tests/IndexTests/fixtures/valid@1.0.0.fsx @@ -0,0 +1,13 @@ +(* +--- +Name: valid +MajorVersion: 1 +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell. +--- +*) \ No newline at end of file diff --git a/tests/IndexTests/fixtures/valid@2.0.0.fsx b/tests/IndexTests/fixtures/valid@2.0.0.fsx new file mode 100644 index 0000000..0f99576 --- /dev/null +++ b/tests/IndexTests/fixtures/valid@2.0.0.fsx @@ -0,0 +1,32 @@ +(* +--- +Name: valid +MajorVersion: 2 +MinorVersion: 0 +PatchVersion: 0 +Summary: My package does the thing. +Description: | + My package does the thing. + It does it very good, it does it very well. + It does it very fast, it does it very swell. +Publish: true +Authors: + - FullName: John Doe + Email: j@d.com + Affiliation: University of Nowhere + AffiliationLink: https://nowhere.edu + - FullName: Jane Doe + Email: jj@d.com + Affiliation: University of Somewhere + AffiliationLink: https://somewhere.edu +Tags: + - Name: validation + - Name: my-tag + TermSourceREF: my-ontology + TermAccessionNumber: MO:12345 +ReleaseNotes: | + - initial release + - does the thing + - does it well +--- +*) \ No newline at end of file