Add Modalities, Classification endpoints

[nagem]

New endpoints:

GET /api/modalities - GET list of all modalities and their allowable classifications.
GET /api/modalities/<modality_name> - GET a specific modality
POST /api/modalities - Add a new modality. Required: _id (readable name) and classification, a map of allowable classifications. Requires superuser. Example request:

POST /api/modalities HTTP/1.1
Content-Type: application/json
{
    "_id": "MR",
    "classifications": {
        "Contrast": ["B0", "B1", "T1", "T2", "T2*", "PD", "MT", "ASL", "Perfusion", "Diffusion", "Spectroscopy", "Susceptibility", "Velocity", "Fingerprinting"],
        "Intent": ["Structural", "Functional", "Localizer", "Shim", "Calibration"],
        "Features": ["Quantitative", "Multi-Shell", "Multi-Echo", "Multi-Flip", "Multi-Band", "Steady-State", "3D", "Compressed-Sensing", "Eddy-Current-Corrected", "Fieldmap-Corrected", "Gradient-Unwarped", "Motion-Corrected", "Physio-Corrected"]
    }
}

PUT /api/modalities/<modality_name> - REPLACE a specific modality. Requires superuser.
DELETE /api/modalities/<modality_name> - DELETE a specific modality. Requires superuser.

POST /api/<cont_name>/<cont_id>/files/<files_name>/classification - UPDATE an existing file's classification. Works similar to the info editing endpoint. Several options are allowed:

To push items to lists within classification, use add:

POST /api/acquisitions/0293840238/files/test.txt/classification
{
    "add": {
        "Contrast": ["B0"]
    }
}
# This will add "B0" to the "Contrast" key on the classification. It will create "Contrast" if the key does not exist and will not add "B0" if it is already in the list

To pull items from lists within classification, use delete:

POST /api/acquisitions/0293840238/files/test.txt/classification
{
    "delete": {
        "Intent": ["Structural"]
    }
}
# This will remove "Structural" from the "Intent" key on the classification. 

Note: add and delete can both be used in the same request.

To replace an entire classification, use replace:

POST /api/acquisitions/0293840238/files/test.txt/classification
{
    "replace": {
        "Contrast": ["B0", "Diffusion"]
        "Intent": ["Structural"]
    }
}
# This will set the `classification` key to the value of the `replace` key. 

Relevant Response Formats:

Success responses when editing a file (PUT /api/cont/cont_id/files/filename, POST /api/cont/cont_id/files/filename/info, POST /api/cont/cont_id/files/filename/classification) will be formatted like:

{
    'jobs_spawned': <int: number of jobs spawned because of this change>
}

Errors when attempting to edit a file's classification:

'Unknown modality. Classification must be set under "custom" key'

'Classification does not match format for modality <modality>. Unallowable key-value pairs: "key1:value1", "key1:value2", "key2:value1"'

TODO:

• finalize measurements -> classification handling in reaper/engine endpoints DONE

Breaking Changes:

• A measurements list will no longer be an accepted or outputted field on a file, it has been replaced by a classification map. SDK, search, and other clients will need to be updated.
• A few rule types have been renamed
• Session templates should no longer reference measurements. DB update replaces the key with classification. Format did not change, just a key rename.
• NOTE: The modality field on a file can be any string, not just to known list of modalities in the system
• NOTE: If the file has no modality or if the modality doesn't exist in the system, new classifications can only be entered under the Custom key

Review Checklist

• Tests were added to cover all code changes
• Documentation was added / updated
• Code and tests follow standards in CONTRIBUTING.md

[codecov-io]

Codecov Report

Merging #1010 into master will increase coverage by 0.08%.
The diff coverage is 94.77%.

@@            Coverage Diff             @@
##           master    #1010      +/-   ##
==========================================
+ Coverage   90.97%   91.05%   +0.08%     
==========================================
  Files          49       50       +1     
  Lines        7037     7145     +108     
==========================================
+ Hits         6402     6506     +104     
- Misses        635      639       +4

[kofalt]

@gsfr Just a head's up that this is a breaking change to gears and we should chat offline about our support strategy.

[ryansanford]

@nagem to make change to still pass measurements key for backwards compatibility. That behavior will be deprecated and removed in future.

[nagem]

@gsfr When should file modifications cause jobs to spawn? Info editing? Classification editing? Any editing?

[nagem]

@kofalt, @ambrussimon I am no longer fiddling with this and would appreciate your review when you have time available.

[ambrussimon]

Nice work. Easily used the tests and schemas to infer the feature/endpoint details 👍

[ambrussimon]

I believe I've seen the test for this edge case, please remove TODO 👍

[nagem]

Thanks 👍. We turned off the pylint warning for TODOs since there were so many in the system at that time. Would be nice to go back through and address any that can be addressed and move the rest to tickets so we can catch things that weren't meant to be longstanding TODOs. Not sure how much effort we want to put into formalizing what TODO means in the system, though 😉

[ambrussimon]

Does this still apply? Why can't validate_data use the newly added modality.json?

[nagem]

This is also a TODO that was intended to be finished before PR review. I ended up using a lot of TODO-type comments since this branch existed for so long.

[nagem]

This should be removed.

[nagem]

Also update here.