nginx performance with aritificial latency - help?

[Jacob Burkamper]

Hello All!

I am attempting to do stress testing of a custom proxy server. Basically,
the scenario looks somewhat like the following:

performance testing software (client) ---> proxy server ---> nginx (server).

The nginx server is an Ubuntu 14.04 install. I have done a small amount of
reading on tuning nginx for performance, so I have increased the number of
worker_processes to fit my cpu, set worker_rlimit_nofile very high, told
nginx to use epoll, sendfile, tcp_nopush, and tcp_nodelay. I have also
increased the number of worker_connections to 4000, though in my case I did
not see any change from its previous default value of 768. The nginx server
is serving up static content only.

Under line-speed circumstances, everything works well and the nginx server
appears to be able to match (if not exceed) the abilities of the proxy
server. Performance issues and bugs worked out of the proxy server, I
decided I wanted to attempt to simulate an artificial latency. To do this,
I decided to use the linux tc plugin netem. I used the following command to
create an artificial latency between the nginx server and the proxy server:

tc qdisc add dev eth0 root netem delay 30ms 20ms distribution normal

Prior to running this command, I could fully saturate the 1 gbps connection
rate of the nginx server. Now that there is latency, it would make sense
that the connection rate would diminish. It dropped to a maximum of 450
mbps. To increase the speed, therefore, I decided to increase the number of
connections. To my surprise, this had no effect. The speed stayed at 450
mbps. I don't get any error messages on either end, the connections just
take longer to resolve. However, the load level on the nginx server does
not change.

Thinking something about the way netem worked had caused the line speed to
drop, I used the tool iperf to test the connection speed between the client
and the server, I got the full 1gbps.

At this point, I bypassed the proxy software thinking it was the problem.
This did leave the hardware with the proxy server on it routing the
traffic. It is linux based. Bypassing the proxy software did not change the
performance.

Does anyone have any suggestions on how I can solve this? I would have
expected an increase in the number of connections to yield an increase in
speed, but this was not the result. It seems with the artificial latency
nginx is only able to handle approximately 3,000 requests per second, where
without the latency I saw over 7,000.

Thank you all for your help, and I will try to provide more information if
that will help anyone help me solve this problem.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20141120/d3b89d96/attachment-0001.html>