reimplement /data_objects/study/{study_id} using alldocs

[sujaypatil96]

Description

This PR seeks to reimplement the /data_objects/study/{study_id} endpoint that is currently part of the runtime API endpoints suite so that it accounts for the schema design considerations noted here: https://docs.google.com/document/d/1HglWENf1m9IDAuk6468OAFdCRvpC3wCQG8wwDHwll1M/edit?usp=sharing

Another point to note is that is that it also uses the alldocs collection for its implementation.

Fixes #401

TODO: write tests for this API endpoint.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration, if it is not simply make up-test && make test-run.

Definition of Done (DoD) Checklist:

A simple checklist of necessary activities that add verifiable/demonstrable value to the product by asserting the quality of a feature, not the functionality of that feature; the latter should be stated as acceptance criteria in the issue this PR closes. Not all activities in the PR template will be applicable to each feature since the definition of done is intended to be a comprehensive checklist. Consciously decide the applicability of value-added activities on a feature-by-feature basis.

• My code follows the style guidelines of this project (have you run black nmdc_runtime/?)
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (in docs/ and in https://github.com/microbiomedata/NMDC_documentation/?)
• I have added tests that prove my fix is effective or that my feature works, incl. considering downstream usage (e.g. https://github.com/microbiomedata/notebook_hackathons) if applicable.
• New and existing unit and functional tests pass locally with my changes (make up-test && make test-run)

[sujaypatil96]

@PeopleMakeCulture i'm having a little trouble writing a test in tests/test_api/test_endpoints.py because I seem to be having some trouble running the tests that I'm writing. Would you be able to help me write a test for this sometime? Perhaps after the roundtable updates at squad sync call tomorrow?

[dwinston]

I was thinking you could implement as an aggregation pipeline with a $graphLookup stage. The following, for example, yields for me 4,395 documents based on the prod db backup from 2024-07-29_20-12-07:

db.getCollection("alldocs").aggregate(

    // Pipeline
    [
        // Stage 1
        {
            $match: {
                "type": "Biosample",
                "part_of": "nmdc:sty-11-hdd4bf83"
            }
        },

        // Stage 2
        {
            $project: {
                "biosample_id": "$id"
            }
        },

        // Stage 3
        {
            $graphLookup: {
                from: "alldocs",
                startWith: "$biosample_id",
                connectFromField: "has_output",
                connectToField: "has_input",
                as: "process"
            }
        },

        // Stage 4
        {
            $unwind: {
                path: "$process"
            }
        },

        // Stage 5
        {
            $match: {
                "process.has_output.0": {"$exists": true}
            }
        },

        // Stage 6
        {
            $unwind: {
                path: "$process.has_output"
            }
        },

        // Stage 7
        {
            $project: {
                "has_output": "$process.has_output",
                "biosample_id": 1
            }
        },

        // Stage 8
        {
            $lookup: {
                from: "data_object_set",
                localField: "has_output",
                foreignField: "id",
                as: "data_object"
            }
        },

        // Stage 9
        {
            $match: {
                "data_object.0": {"$exists": true}
            }
        },

        // Stage 10
        {
            $project: {
                "biosample_id": 1,
                "data_object": {"$first": "$data_object"}
            }
        }
    ],
);

The form of the resulting documents in this case is e.g.:

{
    "_id" : ObjectId("66b4c7a8a2c847e76f4a1619"),
    "biosample_id" : "nmdc:bsm-11-1gzgce32",
    "data_object" : {
        "_id" : ObjectId("65a0688bfa6e7e39b884012f"),
        "id" : "nmdc:dobj-11-72cj2a39",
        "name" : "nmdc_wfmgas-11-dc2b0f84.1_contigs.fna",
        "description" : "Assembly contigs for nmdc:wfmgas-11-dc2b0f84.1",
        "alternative_identifiers" : [

        ],
        "file_size_bytes" : 127171635,
        "md5_checksum" : "090e8612f7bcf18530bb088f79a3cb54",
        "data_object_type" : "Assembly Contigs",
        "url" : "https://data.microbiomedata.org/data/nmdc:omprc-11-avd3n976/nmdc:wfmgas-11-dc2b0f84.1/nmdc_wfmgas-11-dc2b0f84.1_contigs.fna",
        "type" : "nmdc:DataObject"
    }
}

[sujaypatil96]

Wow, thanks for adding the tests and a prototype of the implementation with $graphLookup @dwinston!! i'll take a look at it. Looks super promising ❤️

[sujaypatil96]

@dwinston I ran:

make up-test
make test-build
docker compose --file docker-compose.test.yml run test -k 'test_find_data_objects_for_study'

The test test_find_data_objects_for_study() ran successfully, and returned 61 biosamples and data objects, which is the number of records that we expect to be returned. For the nmdc:sty-11-hdd4bf83 study, the current API endpoint as we have it also returns 61 biosamples and data objects so this works as expected. I believe it's ready to be merged.

[sujaypatil96]

Also, I will capture the code for the aggregation pipeline here on a separate issue, and we can work on benchmarking the two approaches -- Python (implemented in this PR), and Python / Mongo (here). We will ultimately pick the superior approach, and also use that to implement the general rollup endpoint.

[eecavanna]

I assume the name "ensure X" is being used to mean "ensure X exists". If that is the case, I'd prefer the function name be updated to say that specifically.

[eecavanna]

I am wondering whether there is documentation in this folder on the NERSC filesystem, that people there can use to know that the file is used by this code base. Maybe the file could be moved to a directory named nmdc-runtime-assets, for example.

[dwinston]

@eecavanna noted this issue. A move seems reasonable, and we should document this. Started microbiomedata/infra-admin#110

[aclum]

Is this PR going to be ready soon? I would like this updated endpoint to be out for the end of August release because I would like to get IMG switched over to using our API to find data instead of looking directly at the filesystem.

[sujaypatil96]

@aclum yes, this PR is actually ready to be merged. I'm addressing some of @eecavanna's review comments now.