Kernel stall while testing high-speed HTTPS traffic.

[Maxim Dounin]

Hello!

On Thu, May 28, 2015 at 12:26:55PM -0700, Ben Greear wrote:

> We are seeing problems with Nginx (mostly)locking up the server when
> running high loads of HTTPS traffic.
> 
> This scenario we had nginx configured to
> bind to eth3 but our ssh sessions on eth0 were frozen during this condition as well.
> The system restores itself after a few minutes, (the load generation would
> have stopped after a minute or two of lockup, that may be what lets things
> recover).

[...]

> === Scenario ===
> Load testing box has a direct connection to eth3->eth3 over 10Gbps port.
> 
> Curl clients using https, keepalive, requesting a 1MB file:
> 1000 clients @ 0.25 req/sec = 243 req/sec, 2.2Gbps tx, load 8.3
>  400 clients @ 0.65 req/sec = 260 req/sec, 2.2Gbps tx, load 9.2

[...]

>    directio            1m;

As you are testing with 1m file, and have "directio 1m" in your 
configuration, you are probably reading your files straigh off the 
disk, bypassing filesystem caches.  Given the fact that you are 
using default output_buffers, this is likely to cause lots of disk 
IO requests with such a workload, and likely cause the problem you 
see.  Adding something like "output_buffers 1 1m;" and/or removing 
"directio" should help.

(Note that you won't see the problem with plain HTTP, as you have 
sendfile enabled in your config, likely resulting in more 
effecient disk usage.  The problem appears with HTTPS because 
sendfile() can't be used with encrypted connections.)

(Either way, this topic looks unrelated to nginx-devel at .)

-- 
Maxim Dounin
http://nginx.org/