Various test failures when using OpenSSL 3.0

[dbenoit17]

Certain configurations of OpenSSL 3.0 (eg. CentOS Stream 9) deprecate certain crypto algortihms, causing tests to fail in FIPS mode.

https://github.com/dbenoit17/openssl-fips/actions/runs/3146401859/jobs/5114808045#step:14:1

[ueno]

I added diagnostic logging and it turned out that the failures are caused by stricter checks on RSA usage recently introduced in the openssl package in CentOS Stream 9 to adhere to the FIPS requirements, where:

1. key length must be longer than 1024-bit
2. RSAES-PKCS1-v1_5 (encryption and decryption) is generally not supported regardless of key length
3. SHA-1 usage in the MGF1 function used by RSASSA-PSS is not supported

(1) could be fixed by updating the key (and data) used in the tests, while we have no options for the others; in particular (3) cannot be worked around by envvar nor configuration option in my test.

I'm attaching the full log for further analysis.

[ueno]

The above comment is about crypto test; there is another failure in crypto/tls:

--- FAIL: TestBoringCertAlgs (2.55s)
    boring_test.go:344: basic: accept
    boring_test.go:377: basic (client cert): BAD reject (tls: failed to sign handshake: 
        EVP_PKEY_decrypt/encrypt failed
        openssl error(s):
        file: providers/implementations/asymciphers/rsa_enc.c
        line: 250
        function: rsa_decrypt
        flags: 0
        error string: error:1C800069:Provider routines::invalid key length
        )
    boring_test.go:352: basic (fips): BAD accept
    boring_test.go:373: basic (fips, client cert): reject
    boring_test.go:396: basic test failed, skipping exhaustive test

This is because the leaf certificate created using a 1024-bit RSA key.

[ueno]

3. SHA-1 usage in the MGF1 function used by RSASSA-PSS is not supported

This was actually incorrect; the offending SHA-1 usage is outer hash, while SHA-1 usage in MGF1 is still allowed (as it is the default in the spec).