Intermittant SSL Problems

[Maxim Dounin]

Hello!

On Wed, Nov 20, 2013 at 06:03:11PM -0800, Will Pugh wrote:

> Hi folks,
> 
> We are using Nginx for SSL termination, and then it proxies to an ATS or
> Haproxy server depending on our environment.
> 
> We're running into a problem where every now and then, Nginx closes a
> connection due to a timeout.  When investigating, it looks like the
> connections that are being timed-out are not being forwarded to the backend
> service.  The scenario when we were able to best reproduce this is one
> where one of our Java client was running about 100 REST requests that were
> fairly similar.  I've attached files that contain both the tcpdump from the
> client side as well as the debug log on the nginx side.
> 
> I tried comparing a successful and unsuccessful request next to each
> other.  From the client side, it looks like the messages back and forth
> look very consistent.  On the nginx side, the first difference seems to
> happen when reading in the Http Request.  The requests that fail, all seem
> to do a partial read:

[...]

I think I see what happens here:

- Due to a partial read the c->read->ready flag is reset.

- While processing request headers rest of the body becomes available in a 
  socket buffer.

- The ngx_http_do_read_client_request_body() function calls c->recv(),
  ngx_ssl_recv() in case of SSL, and rest of the data is read from the 
  kernel to OpenSSL buffers.  The c->recv() is called with a limited 
  buffer space though (in the debug log provided, only 16 bytes - as this 
  happens during reading http chunk header), and only part of the data 
  becomes available to nginx.

- As c->read->ready isn't set, ngx_http_do_read_client_request_body() arms 
  a read event and returns to the event loop.

- The socket buffer is empty, so no further events are reported by 
  the kernel till timeout.

Please try the following patch:

--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -1025,6 +1025,7 @@ ngx_ssl_recv(ngx_connection_t *c, u_char
             size -= n;
 
             if (size == 0) {
+                c->read->ready = 1;
                 return bytes;
             }
 
@@ -1034,6 +1035,10 @@ ngx_ssl_recv(ngx_connection_t *c, u_char
         }
 
         if (bytes) {
+            if (c->ssl->last != NGX_AGAIN) {
+                c->read->ready = 1;
+            }
+
             return bytes;
         }
 
-- 
Maxim Dounin
http://nginx.org/en/donation.html