Dramatic slowdown on increased connections (fbsd/nginx/php-fpm)

[Antonio L.]

Hi folks --
I apologize in advance, as this isn't necessarily a clear-cut Nginx issue
per se, but I'd appreciate any pointers that anyone might have.

I'm transitioning a web site with app servers currently running
FreeBSD/Apache/mod_php into a new setup where they will run nginx+php-fpm. I
have been testing with a machine that is running FreeBSD 7.1-Release,
nginx/0.7.60, PHP 5.2.8/fpm-0.5.10.  It's a dual quad-core (Xeon 5405) with
12GB.

I offloaded some live traffic from our app servers to this test server to
see how it performed, gradually increasing the load.  We got to around 250
reqs/s (measured using the nginx stub status module -- and it's purely
dynamic traffic -- static stuff is served separately) with no strain on the
server (avg load on procs was < 1, no disk blocking in vmstat, etc.).

As soon as we passed the 250 req/s mark, we witnessed an instant slowdown.
 On the nginx status page, "Writing: " jumps from approx. 15-20 to over 2000
(continuing to pile up rapidly).  However, server load stays constant and I
didn't notice significant I/O changes (in vmstat/iostat) -- but maybe I
missed something.  Likewise, kern.openfiles rose to maybe ~ 2000 (well below
the limit) and kern.ipc.numopensockets was similar.

This occurs if we keep the load constant at around 200 reqs/s, just not as
fast -- after a couple of hours, though, the same thing happens -- and
traffic just piles up.  The only way we can resolve it is to restart nginx /
php-fpm and decrease the traffic load sent to the server.

This is what my /boot/loader.conf looks like:
kern.maxdsiz="11264M"
kern.dfldsiz="768M"
kern.maxssiz="256MB"
vm.kmem_size="1400M"
kern.maxbcache="64M"
kern.ipc.maxpipekva="4M"
kern.maxfiles="65536"
kern.ipc.nmbclusters="65536"

And /etc/sysctl.conf contains:
kern.ipc.somaxconn=8192
kern.ipc.nmbclusters=32768

In php-fpm.conf I've got rlimit_files set to 30000 and max_children to 200.

Does anyone have any advice on how to further investigate this issue?  I
would expect the load on the server to gradually increase, not pile up like
this, so I assume some system or software configuration value must be
holding this back.

Thanks!

Antonio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://nginx.org/pipermail/nginx/attachments/20090711/e91c86ec/attachment.html>