<div dir="ltr">Awesome, really glad to see this! A couple of followup questions...<div><br></div><div>(a) Is there any way to force a packet flush on record end? At the moment nginx will fragment multiple records across packet boundaries, which is suboptimal as it means that I need a minimum of two packets to decode any record - e.g. if I set my record size to 1370 bytes, the first packet will contain the first full record plus another 20-50 bytes of next record. </div>
<div><br></div><div>(b) Current NGX_SSL_BUFSIZE is set to 16KB which is effectively guaranteed to overflow the CWND of a new connection and introduce another RTT for interactive traffic - e.g. HTTP pages. I would love to see a lower starting record size to mitigate this problem -- defaults matter!</div>
<div><br></div><div>On the subject of optimizing record size, the GFE team at Google recently landed ~following logic:</div><div><br></div><div><div style="font-family:arial,sans-serif;font-size:13px">- new connections default to small record size</div>
<div style="font-family:arial,sans-serif;font-size:13px">-- each record fits into a TCP packet</div><div style="font-family:arial,sans-serif;font-size:13px">-- packets are flushed at record boundaries</div><div style="font-family:arial,sans-serif;font-size:13px">
- server tracks number of bytes written since reset and timestamp of last write</div><div style="font-family:arial,sans-serif;font-size:13px">-- if bytes written > {configurable byte threshold) then boost record size to 16KB</div>
<div style="font-family:arial,sans-serif;font-size:13px">-- if last write timestamp > now - {configurable time threshold} then reset sent byte count </div><div style="font-family:arial,sans-serif;font-size:13px"><br></div>
<div style="font-family:arial,sans-serif;font-size:13px">In other words, start with small record size to optimize for delivery of small/interactive objects (bulk of HTTP traffic). Then, if large file is being transferred bump record size to 16KB and continue using that until the connection goes idle.. when communication resumes, start with small record size and repeat. Overall, this is aimed to optimize delivery of small files where incremental delivery is a priority, and also for large downloads where overall throughput is a priority.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Both byte and time thresholds are exposed as configurable flags, and current defaults in GFE are 1MB and 1s.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">This would require a bit more work than the current patch, but I'd love to see a similar strategy in nginx. Hardcoding a fixed record size will inevitably lead to suboptimal delivery of either interactive or bulk traffic. Thoughts?</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">ig </div></div><div><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Fri, Dec 20, 2013 at 4:19 AM, Maxim Dounin <span dir="ltr"><<a href="mailto:mdounin@mdounin.ru" target="_blank">mdounin@mdounin.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
details: <a href="http://hg.nginx.org/nginx/rev/a297b7ad6f94" target="_blank">http://hg.nginx.org/nginx/rev/a297b7ad6f94</a><br>
branches:<br>
changeset: 5487:a297b7ad6f94<br>
user: Maxim Dounin <<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>><br>
date: Fri Dec 20 16:18:25 2013 +0400<br>
description:<br>
SSL: ssl_buffer_size directive.<br>
<br>
diffstat:<br>
<br>
src/event/ngx_event_openssl.c | 9 ++++++---<br>
src/event/ngx_event_openssl.h | 2 ++<br>
src/http/modules/ngx_http_ssl_module.c | 13 +++++++++++++<br>
src/http/modules/ngx_http_ssl_module.h | 2 ++<br>
4 files changed, 23 insertions(+), 3 deletions(-)<br>
<br>
diffs (121 lines):<br>
<br>
diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c<br>
--- a/src/event/ngx_event_openssl.c<br>
+++ b/src/event/ngx_event_openssl.c<br>
@@ -190,6 +190,8 @@ ngx_ssl_create(ngx_ssl_t *ssl, ngx_uint_<br>
return NGX_ERROR;<br>
}<br>
<br>
+ ssl->buffer_size = NGX_SSL_BUFSIZE;<br>
+<br>
/* client side options */<br>
<br>
SSL_CTX_set_options(ssl->ctx, SSL_OP_MICROSOFT_SESS_ID_BUG);<br>
@@ -726,6 +728,7 @@ ngx_ssl_create_connection(ngx_ssl_t *ssl<br>
}<br>
<br>
sc->buffer = ((flags & NGX_SSL_BUFFER) != 0);<br>
+ sc->buffer_size = ssl->buffer_size;<br>
<br>
sc->connection = SSL_new(ssl->ctx);<br>
<br>
@@ -1222,7 +1225,7 @@ ngx_ssl_send_chain(ngx_connection_t *c,<br>
buf = c->ssl->buf;<br>
<br>
if (buf == NULL) {<br>
- buf = ngx_create_temp_buf(c->pool, NGX_SSL_BUFSIZE);<br>
+ buf = ngx_create_temp_buf(c->pool, c->ssl->buffer_size);<br>
if (buf == NULL) {<br>
return NGX_CHAIN_ERROR;<br>
}<br>
@@ -1231,14 +1234,14 @@ ngx_ssl_send_chain(ngx_connection_t *c,<br>
}<br>
<br>
if (buf->start == NULL) {<br>
- buf->start = ngx_palloc(c->pool, NGX_SSL_BUFSIZE);<br>
+ buf->start = ngx_palloc(c->pool, c->ssl->buffer_size);<br>
if (buf->start == NULL) {<br>
return NGX_CHAIN_ERROR;<br>
}<br>
<br>
buf->pos = buf->start;<br>
buf->last = buf->start;<br>
- buf->end = buf->start + NGX_SSL_BUFSIZE;<br>
+ buf->end = buf->start + c->ssl->buffer_size;<br>
}<br>
<br>
send = buf->last - buf->pos;<br>
diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h<br>
--- a/src/event/ngx_event_openssl.h<br>
+++ b/src/event/ngx_event_openssl.h<br>
@@ -29,6 +29,7 @@<br>
typedef struct {<br>
SSL_CTX *ctx;<br>
ngx_log_t *log;<br>
+ size_t buffer_size;<br>
} ngx_ssl_t;<br>
<br>
<br>
@@ -37,6 +38,7 @@ typedef struct {<br>
<br>
ngx_int_t last;<br>
ngx_buf_t *buf;<br>
+ size_t buffer_size;<br>
<br>
ngx_connection_handler_pt handler;<br>
<br>
diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c<br>
--- a/src/http/modules/ngx_http_ssl_module.c<br>
+++ b/src/http/modules/ngx_http_ssl_module.c<br>
@@ -111,6 +111,13 @@ static ngx_command_t ngx_http_ssl_comma<br>
offsetof(ngx_http_ssl_srv_conf_t, ciphers),<br>
NULL },<br>
<br>
+ { ngx_string("ssl_buffer_size"),<br>
+ NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1,<br>
+ ngx_conf_set_size_slot,<br>
+ NGX_HTTP_SRV_CONF_OFFSET,<br>
+ offsetof(ngx_http_ssl_srv_conf_t, buffer_size),<br>
+ NULL },<br>
+<br>
{ ngx_string("ssl_verify_client"),<br>
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1,<br>
ngx_conf_set_enum_slot,<br>
@@ -424,6 +431,7 @@ ngx_http_ssl_create_srv_conf(ngx_conf_t<br>
<br>
sscf->enable = NGX_CONF_UNSET;<br>
sscf->prefer_server_ciphers = NGX_CONF_UNSET;<br>
+ sscf->buffer_size = NGX_CONF_UNSET_SIZE;<br>
sscf->verify = NGX_CONF_UNSET_UINT;<br>
sscf->verify_depth = NGX_CONF_UNSET_UINT;<br>
sscf->builtin_session_cache = NGX_CONF_UNSET;<br>
@@ -465,6 +473,9 @@ ngx_http_ssl_merge_srv_conf(ngx_conf_t *<br>
(NGX_CONF_BITMASK_SET|NGX_SSL_SSLv3|NGX_SSL_TLSv1<br>
|NGX_SSL_TLSv1_1|NGX_SSL_TLSv1_2));<br>
<br>
+ ngx_conf_merge_size_value(conf->buffer_size, prev->buffer_size,<br>
+ NGX_SSL_BUFSIZE);<br>
+<br>
ngx_conf_merge_uint_value(conf->verify, prev->verify, 0);<br>
ngx_conf_merge_uint_value(conf->verify_depth, prev->verify_depth, 1);<br>
<br>
@@ -572,6 +583,8 @@ ngx_http_ssl_merge_srv_conf(ngx_conf_t *<br>
return NGX_CONF_ERROR;<br>
}<br>
<br>
+ conf->ssl.buffer_size = conf->buffer_size;<br>
+<br>
if (conf->verify) {<br>
<br>
if (conf->client_certificate.len == 0 && conf->verify != 3) {<br>
diff --git a/src/http/modules/ngx_http_ssl_module.h b/src/http/modules/ngx_http_ssl_module.h<br>
--- a/src/http/modules/ngx_http_ssl_module.h<br>
+++ b/src/http/modules/ngx_http_ssl_module.h<br>
@@ -26,6 +26,8 @@ typedef struct {<br>
ngx_uint_t verify;<br>
ngx_uint_t verify_depth;<br>
<br>
+ size_t buffer_size;<br>
+<br>
ssize_t builtin_session_cache;<br>
<br>
time_t session_timeout;<br>
<br>
_______________________________________________<br>
nginx-devel mailing list<br>
<a href="mailto:nginx-devel@nginx.org">nginx-devel@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx-devel" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx-devel</a><br>
</blockquote></div><br></div>