Server accept hangs when the client rejects the certificate (since 0.19.2) #399
Description
When the client rejects the server certificate, it sends nothing more: no alert and no close. If the client keeps its TCP socket open, the server blocks in accept forever.
require "openssl"
require "socket"
key = OpenSSL::PKey::RSA.new(2048)
cert = OpenSSL::X509::Certificate.new
cert.version = 2; cert.serial = 1
cert.subject = cert.issuer = OpenSSL::X509::Name.parse("/CN=localhost")
cert.public_key = key
cert.not_before = Time.now - 60; cert.not_after = Time.now + 3600
cert.sign(key, "SHA256")
sctx = OpenSSL::SSL::SSLContext.new(:TLSv1_2)
sctx.cert = cert; sctx.key = key
tcp_server = TCPServer.new("127.0.0.1", 0)
server = OpenSSL::SSL::SSLServer.new(tcp_server, sctx)
t = Thread.new { server.accept rescue $! }
cctx = OpenSSL::SSL::SSLContext.new(:TLSv1_2)
cctx.verify_mode = OpenSSL::SSL::VERIFY_PEER
cctx.cert_store = OpenSSL::X509::Store.new # trusts nothing
sock = TCPSocket.new("127.0.0.1", tcp_server.addr[1]) # left open
begin
OpenSSL::SSL::SSLSocket.new(sock, cctx).connect
rescue OpenSSL::SSL::SSLError => e
puts "client: #{e.message}"
end
puts t.join(10) ? "server: #{t.value.inspect}" : "server: accept still blocked after 10s"0.19.1: client: Socket closed
server: #<OpenSSL::SSL::SSLError: Socket closed>
0.19.2: client: Socket closed
server: accept still blocked after 10s
0.19.4 and the FIPS variant 0.19.4.1 also hang.
The client never sent the alert, even before 0.19.2. Capturing the traffic shows that 0.19.1 re-sends its ClientHello plus zero bytes on shutdown, which makes the server fail. From 0.19.2 the client sends nothing at all.
The likely cause: BCJSSE buffers the alert and throws the stored exception on the first wrap(). doShutdown stops after that first wrap(), so the alert is never flushed. Both 28bc75b and 8ee241d landed in 0.19.2; I haven't bisected which one changed this.
Activity
[fix] send the alert after a failed client handshake (jruby#399)
[fix] flush buffered alert on failed handshake shutdown (#399)
[compat] report cert verify error on failed handshake (#399)
[fix] flush buffered alert on failed handshake shutdown (#399)
[compat] report cert verify error on failed handshake (#399)