Extra RTT on large certificates (again?)

[Albert Casademont]

Hi Maxim,

Yes, as we were already compiling our own nginx we apply a patch in openssl
before compilation increasing the buffer size to 5120 bytes as a workaround.

As for the patch, we already had "tcp_nodelay on" set in our http {} config
and we kept seeing the extra RTT, is this a different setting or I am
missing something?

I believe the optimal solution would be that openssl exposed an API to
dynamically adjust the buffer size, I'll try to work on that...

Best,

Albert

On Tue, May 23, 2017 at 6:02 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> On Mon, May 22, 2017 at 10:34:11PM +0200, Albert Casademont wrote:
>
> > Seems like the openssl devs are aware of the issue and welcoming PRs,
> AFAIK
> > nothing's been done yet.
> >
> > https://mta.openssl.org/pipermail/openssl-users/2016-
> November/004835.html
>
> Thanks for the link, it confirms what I already concluded from the
> code.  As mentioned there, quick-and-dirty solution would be
> recompile OpenSSL with larger buffer size.
>
> > > Thanks for the prompt response. Yes, we're using Openssl 1.1.0e at the
> > > moment...That is unfortunate, what would you suggest doing? Report
> this to
> > > the openssl devs? An extra RTT is quite painful.
>
> With OpenSSL 1.1.0+ it is no longer possible to adjust handshake
> buffer size as nginx used to do, and OpenSSL changes are needed to
> make it possible again.
>
> Another approach might be to mitigate extra RTT using TCP_NODELAY.
> While result will be still non-optimal (as there will be
> incomplete packets), it should be better than nothing.
>
> Patch:
>
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1495555095 -10800
> #      Tue May 23 18:58:15 2017 +0300
> # Node ID 472c23c0a788646403074b359c30c4bbe860cbf6
> # Parent  7943298d4ac09aae83ca8eef09bcf0a12c310505
> SSL: set TCP_NODELAY on SSL connections earlier.
>
> With OpenSSL 1.1.0+, the workaround for handshake buffer size as introduced
> in a720f0b0e083 (ticket #413) no longer works, as OpenSSL no longer exposes
> handshake buffers, see https://github.com/openssl/
> openssl/commit/2e7dc7cd688.
> Moreover, it no longer possible to adjust handshake buffers at all now.
>
> To avoid additional RTT if handshake uses more than 4k we now set
> TCP_NODELAY
> on SSL connections before handshake.  While this still results in
> sub-optimal
> network utilization due to incomplete packets being sent, it seems to be
> better than nothing.
>
> diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c
> --- a/src/http/ngx_http_request.c
> +++ b/src/http/ngx_http_request.c
> @@ -623,14 +623,16 @@ ngx_http_create_request(ngx_connection_t
>  static void
>  ngx_http_ssl_handshake(ngx_event_t *rev)
>  {
> -    u_char                   *p, buf[NGX_PROXY_PROTOCOL_MAX_HEADER + 1];
> -    size_t                    size;
> -    ssize_t                   n;
> -    ngx_err_t                 err;
> -    ngx_int_t                 rc;
> -    ngx_connection_t         *c;
> -    ngx_http_connection_t    *hc;
> -    ngx_http_ssl_srv_conf_t  *sscf;
> +    int                        tcp_nodelay;
> +    u_char                    *p, buf[NGX_PROXY_PROTOCOL_MAX_HEADER + 1];
> +    size_t                     size;
> +    ssize_t                    n;
> +    ngx_err_t                  err;
> +    ngx_int_t                  rc;
> +    ngx_connection_t          *c;
> +    ngx_http_connection_t     *hc;
> +    ngx_http_ssl_srv_conf_t   *sscf;
> +    ngx_http_core_loc_conf_t  *clcf;
>
>      c = rev->data;
>      hc = c->data;
> @@ -712,6 +714,36 @@ ngx_http_ssl_handshake(ngx_event_t *rev)
>              ngx_log_debug1(NGX_LOG_DEBUG_HTTP, rev->log, 0,
>                             "https ssl handshake: 0x%02Xd", buf[0]);
>
> +            clcf = ngx_http_get_module_loc_conf(hc->conf_ctx,
> +                                                ngx_http_core_module);
> +
> +            if (clcf->tcp_nodelay
> +                && c->tcp_nodelay == NGX_TCP_NODELAY_UNSET)
> +            {
> +                ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0,
> "tcp_nodelay");
> +
> +                tcp_nodelay = 1;
> +
> +                if (setsockopt(c->fd, IPPROTO_TCP, TCP_NODELAY,
> +                               (const void *) &tcp_nodelay, sizeof(int))
> +                    == -1)
> +                {
> +#if (NGX_SOLARIS)
> +                    /* Solaris returns EINVAL if a socket has been shut
> down */
> +                    c->log_error = NGX_ERROR_IGNORE_EINVAL;
> +#endif
> +
> +                    ngx_connection_error(c, ngx_socket_errno,
> +                                         "setsockopt(TCP_NODELAY)
> failed");
> +
> +                    c->log_error = NGX_ERROR_INFO;
> +                    ngx_http_close_connection(c);
> +                    return;
> +                }
> +
> +                c->tcp_nodelay = NGX_TCP_NODELAY_SET;
> +            }
> +
>              sscf = ngx_http_get_module_srv_conf(hc->conf_ctx,
>                                                  ngx_http_ssl_module);
>
> --
> Maxim Dounin
> http://nginx.org/
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170523/1e75b171/attachment.html>