SSL FD Leak

[Ben Maurer]

Ben Maurer wrote:
> Ben Maurer wrote:
>> Hi,
>>
>> On a server that has quite a few SSL connections, I started to notice 
>> that FDs were leaking. I set the load balancer in front of nginx to 
>> stop sending new requests to one server for a few minutes (to let the 
>> keepalive time expire) and found that there were a few thousand FDs 
>> open. netstat says that there are many sockets in the CLOSE_WAIT and 
>> ESTABLISHED state for the SSL server. Many of them have data in 
>> receive queue.
>>
>> Any ideas what might cause this? This is an up-to-date 0.5.x install.
> 
> Some progress on debugging this -- it may have to to do with the 
> deferred setting.
> 
> I've managed to get straces like this:
> 
> accept(6, {sa_family=AF_INET, sin_port=htons(35327), 
> sin_addr=inet_addr("127.0.0.1")}, [16]) = 92
> ioctl(92, FIONBIO, [1])                 = 0
> recv(92, 0xbf9c6c2b, 1, MSG_PEEK)       = -1 EAGAIN (Resource 
> temporarily unavailable)
> 
> by using:
> 
> ab -c500 -n2000 https://localhost:8095/
> 
> and aborting in the middle. It seems that these straces are the ones 
> that result in leaked FDs. The trace really doesn't make much sense to 
> me. Deferred accept promises that the socket only goes into accept once 
> it has data or if it's ready to be closed. Neither of these should 
> result in an EAGAIN. Regardless, it seems the problem is that the FD 
> never gets added to epoll at this point.

It seems like commenting out the check for HTTP requests on the socket 
made everything work. There's probably a way to do this more correctly 
(eg, get the event added back into the epoll structure). With that said, 
maybe it'd be possible to avoid the MSG_PEEK call completely. Openssl is 
  good at detecting this error:

2007/12/30 17:16:43 [crit] 18303#0: *2 SSL_do_handshake() failed (SSL: 
error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request) while 
reading client request line, client: 127.0.0.1, server: warp10

The only disadvantage to doing this is that currently, nginx seems to 
wait until the header has been fully parsed to send an error. However, 
the client wouldn't be able to notice a difference unless:

1. The initial request didn't fit in a single write() (it'd have to be a 
few KB for this to happen)
2. It was super-picky about getting input before output.

Given that this is a rare error condition anyways, it doesn't seem worth 
the extra system call to handle this case...

- Ben