UI unable to verify SSL certificate when connecting to backend after Let's Encrypt TDS Root CA X3 cert expired

[alexskr]

Seeing the following errors in the UI logs in prod/stage:

F, [2021-09-30T09:34:15.040790 #7246] FATAL -- : [bbdd49e5-065e-4465-9309-0c808a0abd96] Faraday::SSLError (Problem retrieving:
https://data.bioontology.org

Error: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:

            `Excon.defaults[:ssl_ca_path] = path_to_certs`
            `ENV['SSL_CERT_DIR'] = path_to_certs`
            `Excon.defaults[:ssl_ca_file] = path_to_file`
            `ENV['SSL_CERT_FILE'] = path_to_file`
            `Excon.defaults[:ssl_verify_callback] = callback`
                (see OpenSSL::SSL::SSLContext#verify_callback)
or:
            `Excon.defaults[:ssl_verify_peer] = false` (less secure).

Let's Encrypt TDS Root CA X3 cert expired on the same date which is why app started to fail.
https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

UI servers run on CentOS 7 and ca-certificates package is 2021.2.50-72 which includes updated root cert for letsencrypt which means that application is not using OS root ca but an outdated bundled-in cert somewhere.

a work around on CentOS 7 is to add ENV['SSL_CERT_FILE'] = '/etc/pki/tls/cert.pem' to config/bioportal_config_<env>.rb
However, this is not an ideal solution.

[alexskr]

a similar problem exists in ncbo_cron when ontology is pulled from a domain which uses letsencrypt TLS cert.

[alexskr]

related to nahi/httpclient#445

[alexskr]

a temporary fix for the API/ncbo_cron systems is done with overwriting/symlinking cacert.pem file in httpdclient gem with the system ca cert
ln -fs /etc/pki/tls/cert.pem $app_path/vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/cacert.pem

[jvendetti]

I looked at where the httpclient transitive dependency is coming from in our stack. The ontologies_api project uses an old version of the google-api-client gem (0.10.3 from March of 2017). Upgrading to the latest google-api-client won't help - it still depends on the same version of httpclient, which doesn't appear to be maintained anymore.

The same dependency structure exists in the ncbo_cron project.

[alexskr]

then we should switch to the modern google ruby client https://github.com/googleapis/google-cloud-ruby

[alexskr]

The issue is resolved.
google-api-client included a workaround.