Intermittant SSL Problems

[Will Pugh]

Cool.  Thanks!

Initial testing looks like this fixed it.

   --Will


On Thu, Nov 21, 2013 at 3:34 AM, Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> On Wed, Nov 20, 2013 at 06:03:11PM -0800, Will Pugh wrote:
>
> > Hi folks,
> >
> > We are using Nginx for SSL termination, and then it proxies to an ATS or
> > Haproxy server depending on our environment.
> >
> > We're running into a problem where every now and then, Nginx closes a
> > connection due to a timeout.  When investigating, it looks like the
> > connections that are being timed-out are not being forwarded to the
> backend
> > service.  The scenario when we were able to best reproduce this is one
> > where one of our Java client was running about 100 REST requests that
> were
> > fairly similar.  I've attached files that contain both the tcpdump from
> the
> > client side as well as the debug log on the nginx side.
> >
> > I tried comparing a successful and unsuccessful request next to each
> > other.  From the client side, it looks like the messages back and forth
> > look very consistent.  On the nginx side, the first difference seems to
> > happen when reading in the Http Request.  The requests that fail, all
> seem
> > to do a partial read:
>
> [...]
>
> I think I see what happens here:
>
> - Due to a partial read the c->read->ready flag is reset.
>
> - While processing request headers rest of the body becomes available in a
>   socket buffer.
>
> - The ngx_http_do_read_client_request_body() function calls c->recv(),
>   ngx_ssl_recv() in case of SSL, and rest of the data is read from the
>   kernel to OpenSSL buffers.  The c->recv() is called with a limited
>   buffer space though (in the debug log provided, only 16 bytes - as this
>   happens during reading http chunk header), and only part of the data
>   becomes available to nginx.
>
> - As c->read->ready isn't set, ngx_http_do_read_client_request_body() arms
>   a read event and returns to the event loop.
>
> - The socket buffer is empty, so no further events are reported by
>   the kernel till timeout.
>
> Please try the following patch:
>
> --- a/src/event/ngx_event_openssl.c
> +++ b/src/event/ngx_event_openssl.c
> @@ -1025,6 +1025,7 @@ ngx_ssl_recv(ngx_connection_t *c, u_char
>              size -= n;
>
>              if (size == 0) {
> +                c->read->ready = 1;
>                  return bytes;
>              }
>
> @@ -1034,6 +1035,10 @@ ngx_ssl_recv(ngx_connection_t *c, u_char
>          }
>
>          if (bytes) {
> +            if (c->ssl->last != NGX_AGAIN) {
> +                c->read->ready = 1;
> +            }
> +
>              return bytes;
>          }
>
> --
> Maxim Dounin
> http://nginx.org/en/donation.html
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20131121/4353cbc4/attachment.html>