Expand `genomes` results to include assembly and linked objects #148

bilalebi · 2024-06-25T09:05:26Z

JIRA Ticket

Parent ticket: https://www.ebi.ac.uk/panda/jira/browse/EA-1191

Description

What changes have you made?

Expand the query to include assembly and linked objects (assembly, organism, species and regions' data) (EA-1210)
Alter the arguments to be more specific (EA-1222)
Expand the query to include Datasets info (EA-1226)
- For genomes query
- And genome query
See if you can optimise the speed
⚠️ Point Thoas to the latest API tag before merging

Example:

How `genomes` query looked like before these changes (Expand to see)

Query

query {
  genomes(by_keyword: {keyword: "cat"}) {
    genome_id
    assembly_accession
    scientific_name
    release_number
    taxon_id
  }
}

Response

{
  "data": {
    "genomes": [
      {
        "genome_id": "14fe0744-4b47-4d31-bddf-dc2b6ffc7f4e",
        "assembly_accession": "GCA_000181335.4",
        "scientific_name": "Felis catus",
        "release_number": 110.1,
        "taxon_id": 9685
      }
    ]
  }
}

The New Query

Only one argument is needed, common_name is the one provided in the example below

query {
  genomes(by_keyword: {
    # tolid:""
    # assembly_accession_id:"GCA_902167145.1"
    # assembly_name:"Zm-B73-REFERENCE-NAM-5.0"
    # ensembl_name:"Zm-B73-REFERENCE-NAM-5.0"
    common_name:"maize",
    # scientific_name:"Zea mays"
    # scientific_parlance_name:"Maize"
    # species_taxonomy_id:"4577"
  })
  {
    genome_id
    assembly_accession
    scientific_name
    release_number
    release_date
    taxon_id
    assembly {
      assembly_id
      name
      accession_id
      regions {
        name
        length
        code
        topology
        metadata {
          ontology_terms {
            accession_id
            value
            url
            source {
              name
              url
              description
            }
          }
        }
      }
      organism {
        is_reference_organism
        scientific_name
        scientific_parlance_name
        species {
          taxon_id
        }
      }
    }
    dataset {
      dataset_id
      name
      release
      type
      source
      dataset_type
      version
      release_date
      release_type
    }
  }
}

Response

{
  "data": {
    "genomes": [
      {
        "genome_id": "4273b9f0-c927-4215-87bf-828ef65de984",
        "assembly_accession": "GCA_902167145.1",
        "scientific_name": "Zea mays",
        "release_number": 110.1,
        "release_date": "2023-10-18",
        "taxon_id": 4577,
        "assembly": {
          "assembly_id": "Zm-B73-REFERENCE-NAM-5.0",
          "name": "Zm-B73-REFERENCE-NAM-5.0",
          "accession_id": "GCA_902167145.1",
          "regions": [
            {
              "name": "8",
              "length": 182411202,
              "code": "chromosome",
              "topology": "linear",
              "metadata": {
                "ontology_terms": [
                  {
                    "accession_id": "SO:0000340",
                    "value": "chromosome",
                    "url": "www.sequenceontology.org/browser/current_release/term/SO:0000340",
                    "source": {
                      "name": "Sequence Ontology",
                      "url": "www.sequenceontology.org",
                      "description": "The Sequence Ontology is a set of terms and relationships used to describe the features and attributes of biological sequence. "
                    }
                  }
                ]
              }
            },
            .. very looooong list
          ],
          "organism": {
            "is_reference_organism": false,
            "scientific_name": "Zea mays",
            "scientific_parlance_name": "Maize",
            "species": {
              "taxon_id": 4577
            }
          }
        },
        "dataset": [
          {
            "dataset_id": "93633b44-0e59-49bf-8902-5acb41c5c74c",
            "name": "GCA_902167145.1",
            "release": 110.1,
            "type": "core_annotation",
            "source": "core",
            "dataset_type": "assembly",
            "version": "",
            "release_date": "10/18/2023",
            "release_type": "partial"
          },
          {
            "dataset_id": "b908f88f-a047-4da0-813e-f0a94b66877a",
            "name": "GCA_902167145.1_EXT01",
            "release": 110.1,
            "type": "core_annotation",
            "source": "core",
            "dataset_type": "genebuild",
            "version": "EXT01",
            "release_date": "10/18/2023",
            "release_type": "partial"
          },
          {
            "dataset_id": "cd0e290e-7986-4383-b545-5483ac3b1316",
            "name": "Zm-B73-REFERENCE-NAM-5.0",
            "release": 110.1,
            "type": "variation_annotation",
            "source": "vcf",
            "dataset_type": "variation",
            "version": "1.0",
            "release_date": "10/18/2023",
            "release_type": "partial"
          },
          {
            "dataset_id": "adf617f3-9e2b-49e0-921f-8bbe2c9ae750",
            "name": "Compara homologies ",
            "release": 110.1,
            "type": "compara_annotation",
            "source": "compara",
            "dataset_type": "homologies",
            "version": "1.0",
            "release_date": "10/18/2023",
            "release_type": "partial"
          },
          {
            "dataset_id": "b78f2b17-03e9-11ef-b152-005056b31e22",
            "name": "Compara homologies ",
            "release": 110.1,
            "type": "compara_annotation",
            "source": "compara",
            "dataset_type": "homologies",
            "version": "2.0",
            "release_date": "10/18/2023",
            "release_type": "partial"
          }
        ]
      }
    ]
  },
  "extensions": {
    "execution_time_in_seconds": 2.33,
    "metadata_api_version": "2.1.0a3"
  }
}

Deprecated Examples

Example 1: Expand the query to include assembly and linked objects

Query

query {
  genomes(by_keyword: {keyword: "cat"}) {
    genome_id
    assembly_accession
    scientific_name
    release_number
    taxon_id
    assembly {
      assembly_id
      name
      accession_id
      regions {
        name
        length
        code
        topology
        metadata {
          ontology_terms {
            accession_id
            value
            url
            source {
              name
              url
              description
            }
          }
        }
      }
      organism {
        is_reference_organism
        scientific_name
        scientific_parlance_name
        species {
          taxon_id
        }
      }
    }
  }
}

Response

{
  "data": {
    "genomes": [
      {
        "genome_id": "14fe0744-4b47-4d31-bddf-dc2b6ffc7f4e",
        "assembly_accession": "GCA_000181335.4",
        "scientific_name": "Felis catus",
        "release_number": 110.1,
        "taxon_id": 9685,
        "assembly": {
          "assembly_id": "Felis_catus_9.0",
          "name": "Felis_catus_9.0",
          "accession_id": "GCA_000181335.4",
          "regions": [
            {
              "name": "D2",
              "length": 90186660,
              "code": "chromosome",
              "topology": "linear",
              "metadata": {
                "ontology_terms": [
                  {
                    "accession_id": "SO:0000340",
                    "value": "chromosome",
                    "url": "www.sequenceontology.org/browser/current_release/term/SO:0000340",
                    "source": {
                      "name": "Sequence Ontology",
                      "url": "www.sequenceontology.org",
                      "description": "The Sequence Ontology is a set of terms and relationships used to describe the features and attributes of biological sequence. "
                    }
                  }
                ]
              }
            }
            ...
          ],
          "organism": {
            "is_reference_organism": false,
            "scientific_name": "Felis catus",
            "scientific_parlance_name": "Cat",
            "species": {
              "taxon_id": 9685
            }
          }
        }
      }
    ]
  }
}

Example 2: Alter the query params to be more specific

Query

Only one argument is needed, common_name is the one provided in the example below

query grpc_by_specific_keyword {
  genomes(by_keyword: {
    # tolid:""
    # assembly_accession_id:"GCA_902167145.1"
    # assembly_name:"Zm-B73-REFERENCE-NAM-5.0"
    # ensembl_name:"Zm-B73-REFERENCE-NAM-5.0"
    common_name:"maize",
    # scientific_name:"Zea mays"
    # scientific_parlance_name:"Maize"
    # species_taxonomy_id:"4577"
  })
  {
    genome_id
    assembly_accession
    scientific_name
    release_number
    release_date
    taxon_id
    assembly {
      assembly_id
      name
      accession_id
      regions {
        name
        length
        code
        topology
        metadata {
          ontology_terms {
            accession_id
            value
            url
            source {
              name
              url
              description
            }
          }
        }
      }
      organism {
        is_reference_organism
        scientific_name
        scientific_parlance_name
        species {
          taxon_id
        }
      }
    }
  }
}

Response

{
  "data": {
    "genomes": [
      {
        "genome_id": "4273b9f0-c927-4215-87bf-828ef65de984",
        "assembly_accession": "GCA_902167145.1",
        "scientific_name": "Zea mays",
        "release_number": 110.1,
        "release_date": "2023-10-18",
        "taxon_id": 4577,
        "assembly": {
          "assembly_id": "Zm-B73-REFERENCE-NAM-5.0",
          "name": "Zm-B73-REFERENCE-NAM-5.0",
          "accession_id": "GCA_902167145.1",
          "regions": [
            {
              "name": "8",
              "length": 182411202,
              "code": "chromosome",
              "topology": "linear",
              "metadata": {
                "ontology_terms": [
                  {
                    "accession_id": "SO:0000340",
                    "value": "chromosome",
                    "url": "www.sequenceontology.org/browser/current_release/term/SO:0000340",
                    "source": {
                      "name": "Sequence Ontology",
                      "url": "www.sequenceontology.org",
                      "description": "The Sequence Ontology is a set of terms and relationships used to describe the features and attributes of biological sequence. "
                    }
                  }
                ]
              }
            },
            ...
          ],
          "organism": {
            "is_reference_organism": false,
            "scientific_name": "Zea mays",
            "scientific_parlance_name": "Maize",
            "species": {
              "taxon_id": 4577
            }
          }
        }
      }
    ]
  }
}

I also sneaked in this ticket since Compara wanted to experiment with extracting all genes belonging to a given genome.

Changes

Altered resolve_genes() and made symbol in by_symbol optional
This will allow Compara to fetch all genes and related data (🔥 RIP Mongo! 🔥 ) for a given genome_uuid
The query might look something like this:

query get_protein_checksum {
  genes(by_symbol:{genome_id:"a7335667-93e7-11ec-a39d-005056b38ce3"})
  {
    stable_id
    transcripts {
      stable_id
      metadata {
        canonical {
          value
        }
      }
      product_generating_contexts {
        product {
          sequence {
            checksum
          }
        }
      }
    }
  }
}

… or not

kamaldodiya

Looks good to me

azangru · 2024-07-18T16:13:33Z

.gitignore

@@ -64,6 +64,7 @@ venv/
 ENV/
 env.bak/
 venv.bak/
+node_modules/


How did you get those? There isn't a package.json anywhere in the repo.

I got them I was playing around with graphql federation.

azangru · 2024-07-18T16:15:23Z

common/schemas/query.graphql


  genome(by_genome_uuid: GenomeUUIDInput!): Genome

 }

 input SymbolInput {
-  symbol: String!
+  symbol: String


How can this be optional/nullable if this is specifically an input to search by symbol? How can one search by symbol without providing a symbol?

This change is temporary, Botond from compara wants to experiment with fetching all genes data belonging to a given genome.

I figured the quickest way with minimal changes is to tweak the already existing query (Commit b942572)
We will definitely reverse the changes later.

Jira Ticket: https://www.ebi.ac.uk/panda/jira/browse/EA-1228

bilalebi added 2 commits June 24, 2024 15:14

first attempt

daf7b2a

clean up and refactor the newely added feature's code

8d6f18c

bilalebi self-assigned this Jun 25, 2024

bilalebi added 4 commits June 25, 2024 13:51

optimise the script by checking if assembly related data is requested…

b8732ea

… or not

fix tests

0a50ce4

fix pylint

f8d5885

remove uneccessary code

7fb9dad

bilalebi changed the title ~~Expand metadata~~ Expand genomes results to include assembly and linked objects Jun 25, 2024

bilalebi added 2 commits July 8, 2024 16:27

add release_date

b11e627

Alter by_keyword arguments to be more specific

1404283

bilalebi requested review from kamaldodiya and darefalola July 17, 2024 10:22

bilalebi added 3 commits July 17, 2024 11:52

include dataset info in 'genomes' and 'genome' queries

2b7d980

Improve the code logic

97c1532

Made symbol param optional in genes and transcript resolvers (EA-1228)

b942572

darefalola approved these changes Jul 18, 2024

View reviewed changes

kamaldodiya approved these changes Jul 18, 2024

View reviewed changes

azangru reviewed Jul 18, 2024

View reviewed changes

bilalebi added 4 commits July 19, 2024 15:13

🔀 fix merge conflicts

69356af

fix mypy errors

bc49c44

add mypy to github actions

04760e4

update metadata-api tag

1274e8c

bilalebi merged commit 31697f3 into develop Jul 22, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expand `genomes` results to include assembly and linked objects #148

Expand `genomes` results to include assembly and linked objects #148

bilalebi commented Jun 25, 2024 •

edited

Loading

kamaldodiya left a comment

azangru Jul 18, 2024

bilalebi Jul 19, 2024

azangru Jul 18, 2024 •

edited

Loading

bilalebi Jul 19, 2024

Expand genomes results to include assembly and linked objects #148

Expand genomes results to include assembly and linked objects #148

Conversation

bilalebi commented Jun 25, 2024 • edited Loading

JIRA Ticket

Description

Example:

Query

Response

The New Query

Response

Example 1: Expand the query to include assembly and linked objects

Query

Response

Example 2: Alter the query params to be more specific

Query

Response

Changes

kamaldodiya left a comment

Choose a reason for hiding this comment

azangru Jul 18, 2024

Choose a reason for hiding this comment

bilalebi Jul 19, 2024

Choose a reason for hiding this comment

azangru Jul 18, 2024 • edited Loading

Choose a reason for hiding this comment

bilalebi Jul 19, 2024

Choose a reason for hiding this comment

Expand `genomes` results to include assembly and linked objects #148

Expand `genomes` results to include assembly and linked objects #148

bilalebi commented Jun 25, 2024 •

edited

Loading

azangru Jul 18, 2024 •

edited

Loading