<div dir="ltr"><div><div><div><div>Hi Maxim,<br><br>Yes, as we were already compiling our own nginx we apply a
patch in openssl before compilation increasing the buffer size to 5120 bytes as a
workaround.<br><br></div>As for the patch, we already had "tcp_nodelay on" set in our http {} config and we kept seeing the extra RTT, is this a different setting or I am missing something? <br><br></div>I believe the optimal solution would be that openssl exposed an API to dynamically adjust the buffer size, I'll try to work on that...<br><br></div>Best,<br><br></div>Albert<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 23, 2017 at 6:02 PM, Maxim Dounin <span dir="ltr"><<a href="mailto:mdounin@mdounin.ru" target="_blank">mdounin@mdounin.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello!<br>
<span class=""><br>
On Mon, May 22, 2017 at 10:34:11PM +0200, Albert Casademont wrote:<br>
<br>
> Seems like the openssl devs are aware of the issue and welcoming PRs, AFAIK<br>
> nothing's been done yet.<br>
><br>
> <a href="https://mta.openssl.org/pipermail/openssl-users/2016-November/004835.html" rel="noreferrer" target="_blank">https://mta.openssl.org/<wbr>pipermail/openssl-users/2016-<wbr>November/004835.html</a><br>
<br>
</span>Thanks for the link, it confirms what I already concluded from the<br>
code. As mentioned there, quick-and-dirty solution would be<br>
recompile OpenSSL with larger buffer size.<br>
<span class=""><br>
> > Thanks for the prompt response. Yes, we're using Openssl 1.1.0e at the<br>
> > moment...That is unfortunate, what would you suggest doing? Report this to<br>
> > the openssl devs? An extra RTT is quite painful.<br>
<br>
</span>With OpenSSL 1.1.0+ it is no longer possible to adjust handshake<br>
buffer size as nginx used to do, and OpenSSL changes are needed to<br>
make it possible again.<br>
<br>
Another approach might be to mitigate extra RTT using TCP_NODELAY.<br>
While result will be still non-optimal (as there will be<br>
incomplete packets), it should be better than nothing.<br>
<br>
Patch:<br>
<br>
# HG changeset patch<br>
# User Maxim Dounin <<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>><br>
# Date 1495555095 -10800<br>
# Tue May 23 18:58:15 2017 +0300<br>
# Node ID 472c23c0a788646403074b359c30c4<wbr>bbe860cbf6<br>
# Parent 7943298d4ac09aae83ca8eef09bcf0<wbr>a12c310505<br>
SSL: set TCP_NODELAY on SSL connections earlier.<br>
<br>
With OpenSSL 1.1.0+, the workaround for handshake buffer size as introduced<br>
in a720f0b0e083 (ticket #413) no longer works, as OpenSSL no longer exposes<br>
handshake buffers, see <a href="https://github.com/openssl/openssl/commit/2e7dc7cd688" rel="noreferrer" target="_blank">https://github.com/openssl/<wbr>openssl/commit/2e7dc7cd688</a>.<br>
Moreover, it no longer possible to adjust handshake buffers at all now.<br>
<br>
To avoid additional RTT if handshake uses more than 4k we now set TCP_NODELAY<br>
on SSL connections before handshake. While this still results in sub-optimal<br>
network utilization due to incomplete packets being sent, it seems to be<br>
better than nothing.<br>
<br>
diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c<br>
--- a/src/http/ngx_http_request.c<br>
+++ b/src/http/ngx_http_request.c<br>
@@ -623,14 +623,16 @@ ngx_http_create_request(ngx_<wbr>connection_t<br>
static void<br>
ngx_http_ssl_handshake(ngx_<wbr>event_t *rev)<br>
{<br>
- u_char *p, buf[NGX_PROXY_PROTOCOL_MAX_<wbr>HEADER + 1];<br>
- size_t size;<br>
- ssize_t n;<br>
- ngx_err_t err;<br>
- ngx_int_t rc;<br>
- ngx_connection_t *c;<br>
- ngx_http_connection_t *hc;<br>
- ngx_http_ssl_srv_conf_t *sscf;<br>
+ int tcp_nodelay;<br>
+ u_char *p, buf[NGX_PROXY_PROTOCOL_MAX_<wbr>HEADER + 1];<br>
+ size_t size;<br>
+ ssize_t n;<br>
+ ngx_err_t err;<br>
+ ngx_int_t rc;<br>
+ ngx_connection_t *c;<br>
+ ngx_http_connection_t *hc;<br>
+ ngx_http_ssl_srv_conf_t *sscf;<br>
+ ngx_http_core_loc_conf_t *clcf;<br>
<br>
c = rev->data;<br>
hc = c->data;<br>
@@ -712,6 +714,36 @@ ngx_http_ssl_handshake(ngx_<wbr>event_t *rev)<br>
ngx_log_debug1(NGX_LOG_DEBUG_<wbr>HTTP, rev->log, 0,<br>
"https ssl handshake: 0x%02Xd", buf[0]);<br>
<br>
+ clcf = ngx_http_get_module_loc_conf(<wbr>hc->conf_ctx,<br>
+ ngx_http_core_module);<br>
+<br>
+ if (clcf->tcp_nodelay<br>
+ && c->tcp_nodelay == NGX_TCP_NODELAY_UNSET)<br>
+ {<br>
+ ngx_log_debug0(NGX_LOG_DEBUG_<wbr>HTTP, c->log, 0, "tcp_nodelay");<br>
+<br>
+ tcp_nodelay = 1;<br>
+<br>
+ if (setsockopt(c->fd, IPPROTO_TCP, TCP_NODELAY,<br>
+ (const void *) &tcp_nodelay, sizeof(int))<br>
+ == -1)<br>
+ {<br>
+#if (NGX_SOLARIS)<br>
+ /* Solaris returns EINVAL if a socket has been shut down */<br>
+ c->log_error = NGX_ERROR_IGNORE_EINVAL;<br>
+#endif<br>
+<br>
+ ngx_connection_error(c, ngx_socket_errno,<br>
+ "setsockopt(TCP_NODELAY) failed");<br>
+<br>
+ c->log_error = NGX_ERROR_INFO;<br>
+ ngx_http_close_connection(c);<br>
+ return;<br>
+ }<br>
+<br>
+ c->tcp_nodelay = NGX_TCP_NODELAY_SET;<br>
+ }<br>
+<br>
sscf = ngx_http_get_module_srv_conf(<wbr>hc->conf_ctx,<br>
ngx_http_ssl_module);<br>
<div class="HOEnZb"><div class="h5"><br>
--<br>
Maxim Dounin<br>
<a href="http://nginx.org/" rel="noreferrer" target="_blank">http://nginx.org/</a><br>
______________________________<wbr>_________________<br>
nginx-devel mailing list<br>
<a href="mailto:nginx-devel@nginx.org">nginx-devel@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx-devel" rel="noreferrer" target="_blank">http://mailman.nginx.org/<wbr>mailman/listinfo/nginx-devel</a><br>
</div></div></blockquote></div><br></div>