Kernel stall while testing high-speed HTTPS traffic.
mdounin at mdounin.ru
Mon Jun 1 16:04:46 UTC 2015
On Thu, May 28, 2015 at 12:26:55PM -0700, Ben Greear wrote:
> We are seeing problems with Nginx (mostly)locking up the server when
> running high loads of HTTPS traffic.
> This scenario we had nginx configured to
> bind to eth3 but our ssh sessions on eth0 were frozen during this condition as well.
> The system restores itself after a few minutes, (the load generation would
> have stopped after a minute or two of lockup, that may be what lets things
> === Scenario ===
> Load testing box has a direct connection to eth3->eth3 over 10Gbps port.
> Curl clients using https, keepalive, requesting a 1MB file:
> 1000 clients @ 0.25 req/sec = 243 req/sec, 2.2Gbps tx, load 8.3
> 400 clients @ 0.65 req/sec = 260 req/sec, 2.2Gbps tx, load 9.2
> directio 1m;
As you are testing with 1m file, and have "directio 1m" in your
configuration, you are probably reading your files straigh off the
disk, bypassing filesystem caches. Given the fact that you are
using default output_buffers, this is likely to cause lots of disk
IO requests with such a workload, and likely cause the problem you
see. Adding something like "output_buffers 1 1m;" and/or removing
"directio" should help.
(Note that you won't see the problem with plain HTTP, as you have
sendfile enabled in your config, likely resulting in more
effecient disk usage. The problem appears with HTTPS because
sendfile() can't be used with encrypted connections.)
(Either way, this topic looks unrelated to nginx-devel at .)
More information about the nginx-devel