Hi list:
This is v1 of the patchset the implementing the feature SSL Dynamic Record Sizing, inspiring by Google Front End (https://www.igvita.com/2013/10/24/optimizing-tls-record-size-and-buffering-…) .
There are 3 conditions, if true at the same time, may trigger SSL_write to send small record over the link, hard coded 1400 bytes at this time to keep it fit into MTU size. We just send out 3 of this small record at most to reduce framing overhead when serving large object, that is enough for browser to discovery other dependency of the page at top of html file. If the buffer chain is smaller than 4096 bytes, it will not justify the overhead of sending small record. After idle for 60s(hard coded at this moment), start all over again.
Any comments is welcome.
Regard
YM
hg export tip
# HG changeset patch
# User YM Chen <gzchenym(a)126.com>
# Date 1430828974 -28800
# Node ID 31bfe6403c340bdc4c04e8e87721736c07bceef8
# Parent 162b2d27d4e1ce45bb9217d6958348c64f726a28
[RFC] event/openssl: Add dynamic record size support for serving ssl trafic
SSL Dynamic Record Sizing is a long sought after feature for website that serving
huge amount of encrypted traffic. The rational behide this is that SSL record should
not overflow the congestion window at the beginning of slow-start period and by doing
so, we can let the browser decode the first ssl record within 1 rtt and establish other
connections to fetch other resources that are referenced at the top of the html file.
diff -r 162b2d27d4e1 -r 31bfe6403c34 src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c Wed Apr 29 14:59:02 2015 +0300
+++ b/src/event/ngx_event_openssl.c Tue May 05 20:29:34 2015 +0800
@@ -1508,6 +1508,11 @@
ngx_uint_t flush;
ssize_t send, size;
ngx_buf_t *buf;
+ ngx_msec_t last_sent_timer_diff;
+ ngx_uint_t loop_count;
+
+ last_sent_timer_diff = ngx_current_msec - c->ssl->last_write_msec;
+ loop_count = 0;
if (!c->ssl->buffer) {
@@ -1517,7 +1522,13 @@
continue;
}
- n = ngx_ssl_write(c, in->buf->pos, in->buf->last - in->buf->pos);
+ size = in->buf->last - in->buf->pos;
+
+ if(last_sent_timer_diff > 1000*60 && loop_count < 3 && size > 4096) {
+ size = 1400;
+ }
+
+ n = ngx_ssl_write(c, in->buf->pos, size);
if (n == NGX_ERROR) {
return NGX_CHAIN_ERROR;
@@ -1532,8 +1543,11 @@
if (in->buf->pos == in->buf->last) {
in = in->next;
}
+
+ loop_count ++;
}
+ c->ssl->last_write_msec = ngx_current_msec;
return in;
}
@@ -1614,9 +1628,14 @@
if (size == 0) {
buf->flush = 0;
c->buffered &= ~NGX_SSL_BUFFERED;
+ c->ssl->last_write_msec = ngx_current_msec;
return in;
}
+ if(last_sent_timer_diff > 1000*60 && loop_count < 3 && size > 4096) {
+ size = 1400;
+ }
+
n = ngx_ssl_write(c, buf->pos, size);
if (n == NGX_ERROR) {
@@ -1633,14 +1652,18 @@
break;
}
- flush = 0;
-
- buf->pos = buf->start;
- buf->last = buf->start;
+ if(buf->last == buf->pos) {
+ flush = 0;
+
+ buf->pos = buf->start;
+ buf->last = buf->start;
+ }
if (in == NULL || send == limit) {
break;
}
+
+ loop_count++;
}
buf->flush = flush;
@@ -1652,6 +1675,7 @@
c->buffered &= ~NGX_SSL_BUFFERED;
}
+ c->ssl->last_write_msec = ngx_current_msec;
return in;
}
diff -r 162b2d27d4e1 -r 31bfe6403c34 src/event/ngx_event_openssl.h
--- a/src/event/ngx_event_openssl.h Wed Apr 29 14:59:02 2015 +0300
+++ b/src/event/ngx_event_openssl.h Tue May 05 20:29:34 2015 +0800
@@ -51,6 +51,8 @@
ngx_buf_t *buf;
size_t buffer_size;
+ ngx_msec_t last_write_msec;
+
ngx_connection_handler_pt handler;
ngx_event_handler_pt saved_read_handler;
Hi,
I found following bug in nginx-1.8.0:
if aio is configured with threads support - sometime (one in thousands
requests) $bytes_sent contains only length of the header. I'm attaching my
nginx.conf, build params and simple python script I'm using the reproduce
this issue.
Here is the output of test script when the problem appears:
.
.
.
received: 101700000 from access_log : 101700000 on 26 iteration
127.0.0.1 - - [15/May/2015 17:27:45] "GET /test HTTP/1.0" 200 -
127.0.0.1 - - [15/May/2015 17:27:47] "GET /test HTTP/1.0" 200 -
received: 101700000 from access_log : 101700000 on 27 iteration
127.0.0.1 - - [15/May/2015 17:27:58] "GET /test HTTP/1.0" 200 -
127.0.0.1 - - [15/May/2015 17:28:00] "GET /test HTTP/1.0" 200 -
received: 101700000 from access_log : 101690000 on 28 iteration
test failed!!
also in access_log file"
.
.
.
10170 GET /test HTTP/1.1
10170 GET /test HTTP/1.1
10170 GET /test HTTP/1.1
170 GET /test HTTP/1.1
10170 GET /test HTTP/1.1
10170 GET /test HTTP/1.1
.
.
Best regards,
George
Hi,
enclosed you will find an attached changeset, that fixes a ASLR/DEP
problem on windows platforms (example Win 7/2008 x64).
To find shared addr offset with ASLR, we have successful used the same
resp. similar solution on various open source projects (ex.: postgresql
etc.). Also nginx with suggested patch works fine over 5 months in
production on several machines (Win-2k8-R2).
BTW(1): I have interest to fix a problem with multiple workers under
windows (ex.: http://forum.nginx.org/read.php?2,188714,188714#msg-188714
[1]).
@Maxim Dounin: can you possibly give me more information, what you mean
here, resp. what it currently depends on (see
http://forum.nginx.org/read.php?2,188714,212568#msg-212568 [2]).
BTW(2): I would like to fix iocp under windows also. Thanks in advance
for any information about.
P.S. speak fluently russian and german...
Regards,
Serg G. Brester (sebres)
-------------------------
Links:
------
[1] http://forum.nginx.org/read.php?2,188714,188714#msg-188714
[2] http://forum.nginx.org/read.php?2,188714,212568#msg-212568
We are seeing problems with Nginx (mostly)locking up the server when
running high loads of HTTPS traffic.
This scenario we had nginx configured to
bind to eth3 but our ssh sessions on eth0 were frozen during this condition as well.
The system restores itself after a few minutes, (the load generation would
have stopped after a minute or two of lockup, that may be what lets things
recover).
We tested different kernels (4.0.4+, 4.0.0+, 3.17.8+ with local patches,
and stock 3.14.27-100.fc19.x86_64, all with same results), different NICs (Intel 10G, Intel 40G),
and Apache as web server.
Apache can sustain about 10.8Gbps of HTTPS traffic and shows no
instability/lockups. nginx maxes out at 2.2Gbps (until it locks up machine).
Some kernel splats indicated some files writing to the file system
journal were blocked > 180 seconds, but they recover, so it is not
a hard lock. The system should not be doing any heavy disk access
since we have 32GB RAM. Swap shows no usage.
=== Scenario ===
Load testing box has a direct connection to eth3->eth3 over 10Gbps port.
Curl clients using https, keepalive, requesting a 1MB file:
1000 clients @ 0.25 req/sec = 243 req/sec, 2.2Gbps tx, load 8.3
400 clients @ 0.65 req/sec = 260 req/sec, 2.2Gbps tx, load 9.2
=== Environment ===
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz
> free
total used free shared buffers cached
Mem: 32840296 1394884 31445412 0 132792 632068
-/+ buffers/cache: 630024 32210272
Swap: 16457724 0 16457724
> cat /etc/issue
Fedora release 19 (Schrödinger’s Cat)
Kernel \r on an \m (\l)
# uname -a
Linux e5-1630-v3-qc 3.14.27-100.fc19.x86_64 #1 SMP Wed Dec 17 19:36:34 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# /usr/local/lanforge/nginx/sbin/nginx -v
nginx version: nginx/1.9.1
We are running small patch to allow nginx to bind to a particular interface. We
tried with this option disabled, and that causes the same trouble. The exact
source is found below:
https://github.com/greearb/nginx/commits/master
We are compiling nginx with these options:
./configure --prefix=/usr/local/lanforge/nginx/ --with-http_ssl_module --with-ipv6 --without-http_rewrite_module
=== Nginx Config ===
worker_processes auto;
worker_rlimit_nofile 100000;
error_log logs/eth3_error.log;
pid /home/lanforge/vr_conf/nginx_eth3.pid;
events {
use epoll;
worker_connections 8096;
multi_accept on;
}
http {
include /usr/local/lanforge/nginx/conf/mime.types;
default_type application/octet-stream;
access_log off;
sendfile on;
directio 1m;
disable_symlinks on;
gzip off;
tcp_nopush on;
tcp_nodelay on;
open_file_cache max=1000 inactive=10s;
open_file_cache_valid 600s;
open_file_cache_min_uses 2000;
open_file_cache_errors off;
etag off;
server {
listen 1.1.1.1:80 so_keepalive=on bind_dev=eth3;
server_name nginx.local nginx web.local web;
location / {
root /var/www/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
server {
listen 1.1.1.1:443 so_keepalive=on ssl bind_dev=eth3;
server_name nginx.local nginx web.local web;
ssl_certificate /usr/local/lanforge/apache.crt;
ssl_certificate_key /usr/local/lanforge/apache.key;
location / {
root /var/www/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
Any help or suggestions is appreciated.
Thanks,
Ben
--
Ben Greear <greearb(a)candelatech.com>
Candela Technologies Inc http://www.candelatech.com
Hi,
Since upgrading to nginx 1.8.0, we started seeing some requests being logged with status 500 and a response of 0 bytes ($bytes_sent). Not many, only one for about 36000 requests.
I've been trying to reliably reproduce this and/or figure out the cause without success, at least to know if this is an nginx bug or some module's. Any ideas or suggestions of how I can go about doing this?
These requests use proxy_pass (with cache) and header_filter_by_lua, so I've been able to figure out some things for these failed requests:
* They are always a cache hit;
* Headers created by the Lua code are correctly logged with $sent_http_*;
* Errors in Lua code result in status 500 with a non-empty response (our custom error page);
Also, the error log shows nothing along with these errors.
Thanks in advance,
--
Carlos Rodrigues
Hi,
I write a fastcgi server and use nginx to pass request to my server. It works till now.
But I find a problem. Nginx always set requestId = 1 when sending fastcgi record.
I was a little upset for this, cause according to fastcgi protocol, web server can send fastcgi records belonging to different request simultaneously, and requestIds are different and keep unique. I really need this feature, because requests can be handled simultaneously just over one connetion.
Can I find a way out?