Extra RTT on large certificates (again?)

[Maxim Dounin]

Hello!

On Mon, May 22, 2017 at 10:34:11PM +0200, Albert Casademont wrote:

> Seems like the openssl devs are aware of the issue and welcoming PRs, AFAIK
> nothing's been done yet.
> 
> https://mta.openssl.org/pipermail/openssl-users/2016-November/004835.html

Thanks for the link, it confirms what I already concluded from the 
code.  As mentioned there, quick-and-dirty solution would be 
recompile OpenSSL with larger buffer size.

> > Thanks for the prompt response. Yes, we're using Openssl 1.1.0e at the
> > moment...That is unfortunate, what would you suggest doing? Report this to
> > the openssl devs? An extra RTT is quite painful.

With OpenSSL 1.1.0+ it is no longer possible to adjust handshake 
buffer size as nginx used to do, and OpenSSL changes are needed to 
make it possible again.

Another approach might be to mitigate extra RTT using TCP_NODELAY.  
While result will be still non-optimal (as there will be 
incomplete packets), it should be better than nothing.

Patch:

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1495555095 -10800
#      Tue May 23 18:58:15 2017 +0300
# Node ID 472c23c0a788646403074b359c30c4bbe860cbf6
# Parent  7943298d4ac09aae83ca8eef09bcf0a12c310505
SSL: set TCP_NODELAY on SSL connections earlier.

With OpenSSL 1.1.0+, the workaround for handshake buffer size as introduced
in a720f0b0e083 (ticket #413) no longer works, as OpenSSL no longer exposes
handshake buffers, see https://github.com/openssl/openssl/commit/2e7dc7cd688.
Moreover, it no longer possible to adjust handshake buffers at all now.

To avoid additional RTT if handshake uses more than 4k we now set TCP_NODELAY
on SSL connections before handshake.  While this still results in sub-optimal
network utilization due to incomplete packets being sent, it seems to be
better than nothing.

diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c
--- a/src/http/ngx_http_request.c
+++ b/src/http/ngx_http_request.c
@@ -623,14 +623,16 @@ ngx_http_create_request(ngx_connection_t
 static void
 ngx_http_ssl_handshake(ngx_event_t *rev)
 {
-    u_char                   *p, buf[NGX_PROXY_PROTOCOL_MAX_HEADER + 1];
-    size_t                    size;
-    ssize_t                   n;
-    ngx_err_t                 err;
-    ngx_int_t                 rc;
-    ngx_connection_t         *c;
-    ngx_http_connection_t    *hc;
-    ngx_http_ssl_srv_conf_t  *sscf;
+    int                        tcp_nodelay;
+    u_char                    *p, buf[NGX_PROXY_PROTOCOL_MAX_HEADER + 1];
+    size_t                     size;
+    ssize_t                    n;
+    ngx_err_t                  err;
+    ngx_int_t                  rc;
+    ngx_connection_t          *c;
+    ngx_http_connection_t     *hc;
+    ngx_http_ssl_srv_conf_t   *sscf;
+    ngx_http_core_loc_conf_t  *clcf;
 
     c = rev->data;
     hc = c->data;
@@ -712,6 +714,36 @@ ngx_http_ssl_handshake(ngx_event_t *rev)
             ngx_log_debug1(NGX_LOG_DEBUG_HTTP, rev->log, 0,
                            "https ssl handshake: 0x%02Xd", buf[0]);
 
+            clcf = ngx_http_get_module_loc_conf(hc->conf_ctx,
+                                                ngx_http_core_module);
+
+            if (clcf->tcp_nodelay
+                && c->tcp_nodelay == NGX_TCP_NODELAY_UNSET)
+            {
+                ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "tcp_nodelay");
+
+                tcp_nodelay = 1;
+
+                if (setsockopt(c->fd, IPPROTO_TCP, TCP_NODELAY,
+                               (const void *) &tcp_nodelay, sizeof(int))
+                    == -1)
+                {
+#if (NGX_SOLARIS)
+                    /* Solaris returns EINVAL if a socket has been shut down */
+                    c->log_error = NGX_ERROR_IGNORE_EINVAL;
+#endif
+
+                    ngx_connection_error(c, ngx_socket_errno,
+                                         "setsockopt(TCP_NODELAY) failed");
+
+                    c->log_error = NGX_ERROR_INFO;
+                    ngx_http_close_connection(c);
+                    return;
+                }
+
+                c->tcp_nodelay = NGX_TCP_NODELAY_SET;
+            }
+
             sscf = ngx_http_get_module_srv_conf(hc->conf_ctx,
                                                 ngx_http_ssl_module);
 
-- 
Maxim Dounin
http://nginx.org/