TLS error using openssl 0.9.8

[ibc]

I have two Stud (master) compiled in two servers and running twice in each server:

• Server 1: openssl 0.9.8o-4squeeze7 (Debian)
  • Stud SSL: with --ssl option
  • Stud TLS: without --ssl option
• Server 2: openssl 1.0.0e-2ubuntu4.2 (Ubuntu)
  • Stud SSL: with --ssl option
  • Stud TLS: without --ssl option

Using "wget https://IP" or a Ruby SSL client I get these results:

• Connection from Server 1 to Stud SSL in Server 1 => OK
• Connection from Server 1 to Stud TLS in Server 1 => ERROR
• Connection from Server 1 to Stud SSL in Server 2 => OK
• Connection from Server 1 to Stud TLS in Server 2 => ERROR
• Connection from Server 2 to Stud SSL in Server 2 => OK
• Connection from Server 2 to Stud TLS in Server 2 => OK
• Connection from Server 2 to Stud SSL in Server 1 => OK
• Connection from Server 2 to Stud TLS in Server 1 => OK

ERROR is always the same:<pre
stud[28247]: {client} Unexpected SSL error (in handshake): 1

NOTE: After more checking I've realized that the error does not occur if I force the client to use TLSv1, so maybe this is an openssl "change" from 0.9.8 to 1.0.0?

[paneq]

Let me share a few thoughts about our issues that we recently discovered.

The client was Event Machine application connecting to the stud with start_tls (as described in EM documentation). Sometimes the client started the connection with TLSv1 header in first tcp packet, sometimes with SSLv2 header containing information inside that it want to use TLS protocol. Stud is compiled on the server with openssl 1.0.0. The EM clients are always using openssl 0.9.8 and as i said on different machines they use different protocol to initiate the secure connection even though same openssl 0.9.8 version is installed on the machines. Maybe this has something to do with different compilation options for openssl 0.9.8 deb package in different linux distros ? I don't know.

I am also not sure why the documentation says that --ssl SSLv3 (implies no TLSv1). Our tests shows that stud with --ssl option will accept connections starting with TLSv1 or SSLv2 header. In TLS mode connections started with SSLv2 header (even containing info about TLS protocol inside) are rejected with Unexpected SSL error (in handshake): 1.

[ibc]

Hi, what does it mean "sometimes with SSLv2 header containing information inside that it want to use TLS protocol"? is that valid in TLS protocol?

BTW it seems that EventMachine does not set the TLS/SSL version to use so it probably relies on the openssl default behavior, which seems to change between 0.9.8 and 1.0.0.

"--ssl SSLv3 (implies no TLSv1)": would it make sense? why not to allow a better protocol (TLSv1) when SSLv3 is offered? I'm not an expert in SSL/TLS protocol, but IMHO it makes sense that a server accepts any TLS/SSL protocol better than a given one.

[paneq]

"is that valid in TLS protocol?" - I don't know. I am not a security speciality neither. I will just let wireshark speak for myself. Both are screenshots from communication of the same Event Machine client using start_ssl. EM linked to openssl 0.9.8 on both machines.

From Debian Squeeze :

From Ubuntu 11.04:

[ibc]

Interesting. The same occurs if I use Ruby OpenSSL bindings (Ruby OpenSSL::SSLSocket class). If I don't set the SSL/TLS version in the class constructor, then same as above happens, which depends on the compiled openssl version.

So IMHO this is not a stud issue.