Poor load balancer performance on rackspace and centos

[seanhess]

I am load testing different options for load balancing, and am getting
poor results from Nginx, haproxy, and varnish. I have one 4GB instance
at Rackspace, hitting 4x1GB app servers. 

I'm hitting a url called "/slow" which deliberately waits 500ms before
responding. If I hit an application server directly it can handle a
connection rate of 1600-1800 per second. 

If I hit the Nginx load balancer, it can only handle about 2000
connections. I was hoping for something closer to 4x1600 = 6000. Below
is the command I'm using to test it. This is run in parallel on 40 256
MB instances. I'm deliberately setting num_call to 1, because I want to
see connection performance. Any higher than this and I start to get a
lot of errors. 

    httperf --server 50.56.80.227 --port 1555 --uri /slow --rate 50
--num-call 1 --num-conn 100 --timeout 5

Here is my nginx config: https://gist.github.com/1299501

So, here's the weird thing, no matter whether I use nginx, haproxy, or
varnish, I get roughly the same results. However, I tested Rackspace's
new cloud balancers, and they get much better performance (doing fine at
7000/s). Since nginx and the others are all running on an instance I set
up, and the rackspace balancer isn't, I'm guessing there's something
about the system that is wrong. I'd rather use a balancer I control, so
I can add caching, gzip, ssl, and other stuff to it. 

How can I find out what the bottleneck is? Is there anything I should
tweak on the system to get performance to be better? Do I need more than
4GB of ram? (Ram usage is not high during the test). Any other random
ideas? 

Thank you for your help!
~sean

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,216935,216935#msg-216935