Greenlight doesn't handle the Let's Encrypt CA switch

[trs80]

https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

2021-09-30 15:22:56 +0000: ERROR: [2b415b42-8abf-4a9a-be56-cf41696fb873] BigBlueButtonException: Connection error. Your URL is probably incorrect: "https://puppis.ccgs.wa.edu.au/bigbluebutton/api". Error: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (certificate has expired)

puppis.ccgs.wa.edu.au is scalelite and has a let's encrypt certificate valid until December.

[frederic100]

Hello, try to update greenlight

cd /root/greenlight
docker-compose down
docker pull bigbluebutton/greenlight:v2
docker-compose up -d

[trs80]

Mine is slightly customised and it's midnight here, will try tomorrow. Also school holidays fortunately.

[jannooo]

Unfortunately, this isn't enough. We are still experiencing the problem after pulling the current image.

[jannooo]

Nevermind. It was a second problem that had nothing to do with it. Your suggestion does the trick.

[kusznerr]

We are experiencing exactly the same problem now. Unfortunately we are running quite a customized version of a greenlight , and cannot simply rebuild it. Do you have any idea what is the true root cause ? Is there any cert that needs to be refreshed ?

would incorporating this commit help ? https://github.com//pull/2930

[Lutice]

this problem comes from this : https://scotthelme.co.uk/lets-encrypt-old-root-expiration/
and there is a mistake in the alpine container : gliderlabs/docker-alpine#30
I tried to update manually the certificates and I have this message : WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping

[Lutice]

@kusznerr I think this patch run for the last greenlight versions, but not for the older. In the fix ruby version is 2.7 but on some greenlight ruby version is 2.5 ... and normally a pull to upgrade greenlight should be a solution

[kusznerr]

@kusznerr I think this patch run for the last greenlight versions, but not for the older. In the fix ruby version is 2.7 but on some greenlight ruby version is 2.5 ... and normally a pull to upgrade greenlight should be a solution

Yes - exactly. We are still on 2.5. Do you have idea which http client (gem) is used there ?

[ofudre]

@kusznerr
Fix for CA certificate:

apt-get install snapd
apt-get remove -y certbot
snap install certbot --classic
certbot renew --force-renewal --preferred-chain "ISRG Root X1"

tested in 8 servers ubuntu 16.04 with customized greenlight

[hiroshisuga]

Yes, @kusznerr 's trick is working. But one needs to restart nginx by
sudo systemctl restart nginx

Can we keep this "classic" setting forever?

Or should we revert it to the apt's certbot when GL is upgraded to the latest version?
If so, would the following command do the job?
snap remove certbot
apt-get install certbot
certbot renew

[kusznerr]

@kusznerr
Fix for CA certificate:

apt-get install snapd
apt-get remove -y certbot
snap install certbot --classic
certbot renew --force-renewal --preferred-chain "ISRG Root X1"

tested in 8 servers ubuntu 16.04 with customized greenlight

Thank you for all timely responces , we were playing with these across greenlight, scalelite-api and bbb servers and continued to get wrong results. As a last resort decided on getting a new wildcard cert, which after installation worked perfectly in our ecosystem. Some lessons learned on our side for sure

[sualko]

I'm experience the same issues on a customized 2.9.2 Greenlight version. Any help would be appreciated.

EDIT: To verify that I'm on Ruby 2.7, I run docker exec greenlight-v2 ruby --version and it showed me 2.5. This means there have to be an issue with my custom image.

[hiroshisuga]

To revert

snap remove certbot
apt-get install certbot
certbot renew

the last one should be with --force-renewal. And everything may have to be executed by sudo.

This thread saved my life!

[strickes]

I am on Ubuntu 18.04 and on the Ruby 2.7 Version:

docker exec greenlight-v2 ruby --version
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux-musl]

unfortunately I have issues connecting to another server for OpenID Login where I get "Error 500"

I updated the ca-certificates on the main server and even forced a certificate renewal but still to no avail. This seems to correlate with the Letsencrypt problems as they occured exactly when the root certificate expired.

[belorenz]

Hello, try to update greenlight

cd /root/greenlight
docker-compose down
docker pull bigbluebutton/greenlight:v2
docker-compose up -d

Did the job for me. Thank you!

[strickes]

already on the newest version:

v2: Pulling from bigbluebutton/greenlight
Digest: sha256:17aab2a3d5469a3e74d2fd93c64e3e164911a6ed73d574bed6a4862e8b8cbbe8
Status: Image is up to date for bigbluebutton/greenlight:v2
docker.io/bigbluebutton/greenlight:v2

[Radnik]

None of the suggested fixes are working for me :(
Keep getting Invalid BigBlueButton Endpoint and Secret

#2930 ?
an issue with httpclient gem

[jannooo]

If anyone is experiencing problems with presentations after fixing the greenlight issue, nodejs likely is the reason. As far as I understand so far it ships its own certificate store. We worked around it by adding the following to /usr/share/meteor/bundle/systemd_start.sh:

export NODE_TLS_REJECT_UNAUTHORIZED=0

[anandology]

I faced the same issue. Updating the docker image of bigbluebutton/greenlight:v2 solved the issue for me.

$ docker pull bigbluebutton/greenlight:v2 
...

$ cd greenlight
$ docker-compose restart
...

Explanation

Big Blue Button documentation suggests the following to verify if you configuration is working correctly.

$ docker run --rm --env-file .env bigbluebutton/greenlight:v2 bundle exec rake conf:check

Checking environment: Passed
Checking Connection: Failed
Error connecting to BigBlueButton server - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (certificate has expired)

As you can see there is an SSL issue.

The Issue #1086 suggests the following way to verify if the installation is working correctly.

$ docker exec -it greenlight-v2  /usr/bin/curl https://join.[domain].de/bigbluebutton/api

Replace the URL(the last argument) with your Big Blue Button URL. You can also find your big-blue-button URL endpoint from .env file in the greenlight directory.

$ grep ENDPOINT .env
BIGBLUEBUTTON_ENDPOINT=https://bbb.example.com/bigbluebutton/

Please note that you can use either of '/bigbluebutton/' or '/bigbluebotton/api`. Both give exactly the same output.

You can also the following instead:

$ docker run --rm -it bigbluebutton/greenlight:v2 /usr/bin/curl https://bbb.example.com/bigbluebutton/api
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

If you see this error, curl is unable to identify the ceritficate provided by letsencrypt.

I'm not exactly sure what is the issue, but updating the docker image solved the issue.

$ docker pull bigbluebutton/greenlight:v2 
...

After this, we need to restart the docker containers.

$ docker-compose restart
...

After this verify again if your configuration is working correctly using:

$ docker run --rm --env-file .env bigbluebutton/greenlight:v2 bundle exec rake conf:check
Checking environment: Passed
Checking Connection: Passed
Checking Secret: Passed
Checking SMTP connection: Passed

[anandology]

I faced the same issue and updating the docker image of greenlight solved it.

$ docker pull bigbluebutton/greenlight:v2
...
$ docker-compose up -d
...

The last command has to be run in the greenlight directory.

Explanation

The bigbluebutton documentaion suggests the following command to verify the installation.

$ docker run --rm --env-file .env bigbluebutton/greenlight:v2 bundle exec rake conf:check
Checking environment: Passed
Checking Connection: Failed
Error connecting to BigBlueButton server - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (certificate has expired)

You may be seeing output similar to this if you are facing SSL issue.

Let's upgrade the greenlight docker image.

$ docker pull bigbluebutton/greenlight:v2
...

and recreate the docker container for greenlight.

$ docker-compose up -d
...

You can verify again if the configuration is working correctly.

$ docker run --rm --env-file .env bigbluebutton/greenlight:v2 bundle exec rake conf:check
Checking environment: Passed
Checking Connection: Passed
...

[farhatahmad]

Is this fixed by #2930?

[UnlimitedCookies]

Is this fixed by #2930?

Yes, but this is only a temporary fix. The issue needs to be fixed upstream, before this issue can eventually be closed.

[nazrimazlan]

Impacted with 2 instance.
Able to renew the certificates, but seems that I am unable to start any meeting.
Invalid Conference-E Endpoint and Secret

[maxbes]

If you are running a customized version based on an old Docker image

This fix will handle issues with OpenSSL 1.0 being unable to pick an alternate chain on its own:

rm /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
update-ca-certificates #ignore warnings, they are not a problem

And this fix will handle issues with the httpclient gem using a bundle that does not know about ISRG Root X1:
https://github.com/bigbluebutton/greenlight/pull/2930/files

Warning: if your docker image has BOTH OpenSSL 1.0.x and httpclient, be sure to run update-ca-certificates BEFORE overwriting the default httpclient certificate bundle

[1somedreamer]

We still have a problem with the certificates in greenlight.
Bigbluebutton 2.2
After

docker pull bigbluebutton/greenlight:v2
docker-compose up -d

We still have

docker exec -it greenlight-v2  /usr/bin/curl [Our ENDPOINT url]
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

And

docker run --rm --env-file .env [Our image name] bundle exec rake conf:check
Checking environment: Passed
Checking Connection: Failed
Error connecting to BigBlueButton server - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (certificate has expired)

So we checked the ruby version in the container

docker exec -it greenlight-v2  ruby --version
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-musl]

And we tried:

rm /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
update-ca-certificates

And after that

docker exec -it greenlight-v2  /usr/bin/curl [Our ENDPOINT url]
<response>
<returncode>SUCCESS</returncode>
<version>1.0</version>

But we still have

docker run --rm --env-file .env [Our image name] bundle exec rake conf:check
Checking environment: Passed
Checking Connection: Failed
Error connecting to BigBlueButton server - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (certificate has expired)

Please help us figure out what we're doing wrong

[JYPDWhite]

I also facing the issue that the presentation is not shown anymore. But all check tell me that all is fine.

root@streamy ~/greenlight # docker run --rm --env-file .env bigbluebutton/greenlight:v2 bundle exec rake conf:check

Checking environment: Passed
Checking Connection: Passed
Checking Secret: Passed
Checking SMTP connection: Passed

And I updated the whole system using bbb-setup and after that docker-compose pull and docker-compose up -d

Any ideas?