Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge Develop to Master, release v5.29.0 #194

Merged
merged 22 commits into from
Mar 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
67b90b2
update owlapi-wrapper
syphax-bouazzouni Aug 3, 2022
7eb3424
add concerns glob loader
syphax-bouazzouni Aug 3, 2022
32e6639
add MetadataExtractor concern with extract_version and ontology_iri
syphax-bouazzouni Aug 3, 2022
4996384
use the new extract_metadata and remove the old extract_version
syphax-bouazzouni Aug 3, 2022
c1e485b
update owlapi-wrapper to 1.4.1 upstream
syphax-bouazzouni Aug 3, 2022
c0f9d00
changes related to ncbo/ontologies_api#134 and ncbo/ontologies_api#135
mdorf Feb 13, 2024
55a4df1
changed the OWL API version to 1.4.2
mdorf Feb 13, 2024
81c0e33
Gemfile.lock update
mdorf Feb 13, 2024
bf438b1
merged PR #153 that adds the ontology IRI metadata attribute
mdorf Feb 13, 2024
e7ed676
merged PR #153 and fixed missing code from the PR
mdorf Feb 14, 2024
ef20f3e
fixed ncbo/ontologies_api#134, ncbo/ontologies_api#135
mdorf Feb 27, 2024
715abe0
Merge branch 'develop' of github.com:ncbo/ontologies_linked_data into…
mdorf Feb 27, 2024
226a2ab
added .bundle to gitignore
mdorf Feb 27, 2024
54925b4
Gemfile.lock update
mdorf Feb 27, 2024
4cdcf8d
Gemfile.lock update
mdorf Feb 27, 2024
68296aa
re-introduced a hard-coded value in create_ontologies_and_submissions…
mdorf Feb 28, 2024
2575a6e
Gemfile.lock update
mdorf Mar 2, 2024
b78284b
Gemfile.lock update
mdorf Mar 2, 2024
01046a5
Add password reset link expiration notice
alexskr Mar 6, 2024
6ed5338
Merge branch 'master' into develop
alexskr Mar 13, 2024
5a506fb
reset branch specifier to develop
alexskr Mar 13, 2024
a973d62
Reset branch specifier to develop
jvendetti Mar 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
*.swp
*.gemtags
.bundle
*.tags*
.DS_Store

Expand Down
4 changes: 2 additions & 2 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,5 @@ group :development do
end

# NCBO gems (can be from a local dev path or from rubygems/git)
gem 'goo', github: 'ncbo/goo', branch: 'master'
gem 'sparql-client', github: 'ncbo/sparql-client', branch: 'master'
gem 'goo', github: 'ncbo/goo', branch: 'develop'
gem 'sparql-client', github: 'ncbo/sparql-client', branch: 'develop'
39 changes: 20 additions & 19 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
GIT
remote: https://github.com/ncbo/goo.git
revision: 75436fe8e387febc53e34ee31ff0e6dd837a9d3f
branch: master
revision: 4ea0e70a4361fc694700e11f1012129452278c7d
branch: develop
specs:
goo (0.0.2)
addressable (~> 2.8)
Expand All @@ -15,8 +15,8 @@ GIT

GIT
remote: https://github.com/ncbo/sparql-client.git
revision: d418d56a6c9ff5692f925b45739a2a1c66bca851
branch: master
revision: 55e7dbf858eb571c767bc67868f9af61663859cb
branch: develop
specs:
sparql-client (1.0.1)
json_pure (>= 1.4)
Expand All @@ -38,10 +38,10 @@ GEM
ast (2.4.2)
base64 (0.2.0)
bcrypt (3.1.20)
bigdecimal (3.1.5)
bigdecimal (3.1.6)
builder (3.2.4)
coderay (1.1.3)
concurrent-ruby (1.2.2)
concurrent-ruby (1.2.3)
connection_pool (2.4.1)
cube-ruby (0.0.3)
daemons (1.4.1)
Expand Down Expand Up @@ -71,7 +71,7 @@ GEM
language_server-protocol (3.17.0.3)
launchy (2.5.2)
addressable (~> 2.8)
libxml-ruby (5.0.2)
libxml-ruby (5.0.3)
logger (1.6.0)
macaddr (1.7.2)
systemu (~> 2.6.5)
Expand All @@ -83,7 +83,7 @@ GEM
method_source (1.0.0)
mime-types (3.5.2)
mime-types-data (~> 3.2015)
mime-types-data (3.2023.1205)
mime-types-data (3.2024.0305)
mini_mime (1.1.5)
minitest (4.7.5)
minitest-reporters (0.14.24)
Expand All @@ -93,7 +93,7 @@ GEM
powerbar
multi_json (1.15.0)
net-http-persistent (2.9.4)
net-imap (0.4.9.1)
net-imap (0.4.10)
date
net-protocol
net-pop (0.1.2)
Expand All @@ -108,7 +108,7 @@ GEM
omni_logger (0.1.4)
logger
parallel (1.24.0)
parser (3.3.0.3)
parser (3.3.0.5)
ast (~> 2.4.1)
racc
pony (1.13.1)
Expand All @@ -120,16 +120,16 @@ GEM
method_source (~> 1.0)
public_suffix (5.0.4)
racc (1.7.3)
rack (2.2.8)
rack (2.2.8.1)
rack-test (0.8.3)
rack (>= 1.0, < 3)
rainbow (3.1.1)
rake (10.5.0)
rdf (1.0.8)
addressable (>= 2.2)
redis (5.0.8)
redis (5.1.0)
redis-client (>= 0.17.0)
redis-client (0.19.1)
redis-client (0.21.0)
connection_pool
regexp_parser (2.9.0)
rest-client (2.1.0)
Expand All @@ -141,19 +141,19 @@ GEM
rsolr (2.5.0)
builder (>= 2.1.2)
faraday (>= 0.9, < 3, != 2.0.0)
rubocop (1.59.0)
rubocop (1.62.1)
json (~> 2.3)
language_server-protocol (>= 3.17.0)
parallel (~> 1.10)
parser (>= 3.2.2.4)
parser (>= 3.3.0.2)
rainbow (>= 2.2.2, < 4.0)
regexp_parser (>= 1.8, < 3.0)
rexml (>= 3.2.5, < 4.0)
rubocop-ast (>= 1.30.0, < 2.0)
rubocop-ast (>= 1.31.1, < 2.0)
ruby-progressbar (~> 1.7)
unicode-display_width (>= 2.4.0, < 3.0)
rubocop-ast (1.30.0)
parser (>= 3.2.1.0)
rubocop-ast (1.31.2)
parser (>= 3.3.0.4)
ruby-progressbar (1.13.0)
ruby2_keywords (0.0.5)
rubyzip (1.3.0)
Expand Down Expand Up @@ -181,6 +181,7 @@ GEM
PLATFORMS
aarch64-linux
arm64-darwin-22
x86_64-darwin-21
x86_64-linux

DEPENDENCIES
Expand Down Expand Up @@ -212,4 +213,4 @@ DEPENDENCIES
thin

BUNDLED WITH
2.3.22
2.4.22
11 changes: 6 additions & 5 deletions config/solr/term_search/schema.xml
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,6 @@
<field name="_root_" type="string" indexed="true" stored="false" docValues="false" />
<field name="_text_" type="text_general" indexed="true" stored="false" multiValued="true"/>


<!-- NCBO specific fields -->
<field name="resource_id" type="string" indexed="true" stored="true" multiValued="false" required="true" />

Expand All @@ -139,8 +138,9 @@
<field name="synonymSuggest" type="text_suggest" indexed="true" stored="true" omitNorms="true" multiValued="true" />
<field name="synonymSuggestEdge" type="text_suggest_edge" indexed="true" stored="true" multiValued="true" />
<field name="synonymSuggestNgram" type="text_suggest_ngram" indexed="true" stored="true" omitNorms="true" multiValued="true" />

<field name="notation" type="text_general" indexed="true" stored="true" multiValued="false" />
<field name="notation" type="string_ci" indexed="true" stored="true" multiValued="false" />
<field name="oboId" type="string_ci" indexed="true" stored="true" multiValued="false" />
<field name="idAcronymMatch" type="boolean" indexed="true" stored="true" multiValued="false" default="false"/>

<field name="definition" type="string" indexed="true" stored="true" multiValued="true" />
<field name="submissionAcronym" type="string" indexed="true" stored="true" multiValued="false" />
Expand All @@ -151,8 +151,8 @@
<field name="childCount" type="int" indexed="true" stored="true" multiValued="false" />

<!-- Added cui and tui (semanticType) - NCBO-695 -->
<field name="cui" type="text_general" indexed="true" stored="true" multiValued="true" />
<field name="semanticType" type="text_general" indexed="true" stored="true" multiValued="true" />
<field name="cui" type="string_ci" indexed="true" stored="true" multiValued="true" />
<field name="semanticType" type="string_ci" indexed="true" stored="true" multiValued="true" />

<field name="property" type="text_general" indexed="true" stored="true" multiValued="true" />
<field name="propertyRaw" type="text_general" indexed="false" stored="true" multiValued="false" />
Expand Down Expand Up @@ -252,6 +252,7 @@
<copyField source="synonym" dest="synonymSuggestNgram" />

<copyField source="notation" dest="_text_"/>
<copyField source="oboId" dest="_text_"/>


<!-- field type definitions. The "name" attribute is
Expand Down
4 changes: 4 additions & 0 deletions lib/ontologies_linked_data.rb
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@
# Require all models
project_root = File.dirname(File.absolute_path(__FILE__))

models = Dir.glob("#{project_root}/ontologies_linked_data/concerns/**/*.rb").sort
models.each do |m|
require m
end
# We need to require deterministic - that is why we have the sort.
models = Dir.glob(project_root + '/ontologies_linked_data/models/**/*.rb').sort
models.each do |m|
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
module LinkedData
module Concerns
module OntologySubmission
module MetadataExtractor

def extract_metadata
version_info = extract_version
ontology_iri = extract_ontology_iri

self.version = version_info if version_info
self.uri = ontology_iri if ontology_iri

end

def extract_version
query = Goo.sparql_query_client.select(:versionInfo).distinct
.from(self.id)
.where([RDF::URI.new('http://bioportal.bioontology.org/ontologies/versionSubject'),
RDF::URI.new('http://www.w3.org/2002/07/owl#versionInfo'),
:versionInfo])

sol = query.each_solution.first || {}
sol[:versionInfo]&.to_s
end

def extract_ontology_iri
query = Goo.sparql_query_client.select(:uri).distinct
.from(self.id)
.where([:uri,
RDF::URI.new('http://www.w3.org/1999/02/22-rdf-syntax-ns#type'),
RDF::URI.new('http://www.w3.org/2002/07/owl#Ontology')])
sol = query.each_solution.first || {}
sol[:uri]&.to_s
end
end
end
end
end
49 changes: 43 additions & 6 deletions lib/ontologies_linked_data/models/class.rb
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@
doc = {}
path_ids = Set.new
self.bring(:submission) if self.bring?(:submission)
class_id = self.id.to_s

if to_set.nil?
begin
Expand All @@ -151,31 +152,33 @@
doc[:childCount] = self.childrenCount
rescue Exception => e
doc[:childCount] = 0
puts "Exception getting childCount for search for #{self.id.to_s}: #{e.class}: #{e.message}"
puts "Exception getting childCount for search for #{class_id}: #{e.class}: #{e.message}"

Check warning on line 155 in lib/ontologies_linked_data/models/class.rb

View check run for this annotation

Codecov / codecov/patch

lib/ontologies_linked_data/models/class.rb#L155

Added line #L155 was not covered by tests
end

begin
# paths_to_root = self.paths_to_root
# paths_to_root.each do |paths|
# path_ids += paths.map { |p| p.id.to_s }
# end
# path_ids.delete(self.id.to_s)
# path_ids.delete(class_id)
path_ids = retrieve_hierarchy_ids(:ancestors)
path_ids.select! { |x| !x["owl#Thing"] }
doc[:parents] = path_ids
rescue Exception => e
doc[:parents] = Set.new
puts "Exception getting paths to root for search for #{self.id.to_s}: #{e.class}: #{e.message}\n#{e.backtrace.join("\n")}"
puts "Exception getting paths to root for search for #{class_id}: #{e.class}: #{e.message}\n#{e.backtrace.join("\n")}"

Check warning on line 169 in lib/ontologies_linked_data/models/class.rb

View check run for this annotation

Codecov / codecov/patch

lib/ontologies_linked_data/models/class.rb#L169

Added line #L169 was not covered by tests
end

acronym = self.submission.ontology.acronym
doc[:id] = class_id
doc[:ontologyId] = self.submission.id.to_s
doc[:submissionAcronym] = self.submission.ontology.acronym
doc[:submissionAcronym] = acronym
doc[:submissionId] = self.submission.submissionId
doc[:ontologyType] = self.submission.ontology.ontologyType.get_code_from_id
doc[:obsolete] = self.obsolete.to_s

all_attrs = self.to_hash
std = [:id, :prefLabel, :notation, :synonym, :definition, :cui]
std = [:prefLabel, :notation, :synonym, :definition, :cui]

std.each do |att|
cur_val = all_attrs[att]
Expand All @@ -199,6 +202,20 @@
doc[:semanticType] = []
all_attrs[:semanticType].each { |semType| doc[:semanticType] << semType.split("/").last }
end

# mdorf, 2/4/2024: special handling for :notation field because some ontologies have it defined as :prefixIRI
if !doc[:notation] || doc[:notation].empty?
if all_attrs[:prefixIRI] && !all_attrs[:prefixIRI].empty?
doc[:notation] = all_attrs[:prefixIRI].strip
else
doc[:notation] = LinkedData::Utils::Triples::last_iri_fragment(doc[:id])
end
end
doc[:idAcronymMatch] = true if notation_acronym_match(doc[:notation], acronym)
# https://github.com/bmir-radx/radx-project/issues/46
# https://github.com/bmir-radx/radx-project/issues/46#issuecomment-1939782535
# https://github.com/bmir-radx/radx-project/issues/46#issuecomment-1939932614
set_oboid_fields(class_id, self.submission.uri, acronym, doc)
end

if to_set.nil? || (to_set.is_a?(Array) && to_set.include?(:properties))
Expand All @@ -209,10 +226,30 @@
doc[:propertyRaw] = props[:propertyRaw]
end
end

doc
end

def set_oboid_fields(class_id, ontology_iri, ontology_acronym, index_doc)
short_id = LinkedData::Utils::Triples.last_iri_fragment(class_id)
matched = short_id.match(/([A-Za-z]+)_([0-9]+)$/) do |m|
index_doc[:oboId] = "#{m[1]}:#{m[2]}"
index_doc[:idAcronymMatch] = true if m[1].upcase === ontology_acronym.upcase
true
end

if !matched && ontology_iri && class_id.start_with?(ontology_iri)
index_doc[:oboId] = "#{ontology_acronym}:#{short_id}"
index_doc[:idAcronymMatch] = true
end
end

def notation_acronym_match(notation, ontology_acronym)
notation.match(/^([A-Za-z]+)[_:]{1}/) do |m|
return m[1].upcase === ontology_acronym.upcase
end
false
end

def properties_for_indexing()
self_props = self.properties
return nil if self_props.nil?
Expand Down
Loading