Fast CGI (spawn-fcgi / php-cgi) crashes/dies/hangs

[Phillip Oldham]

miradev wrote:
> We've been running nginx + spawn-fcgi for 6 months without any issues. Until one morning I come in and all our sites are down because there were no php-cgi processes running. Obviously I'm keen to not have this happen again. I could make use of something like monit to monitor the php-cgi processes, but even a few minutes down time (waiting for a monit cycle) is not good. 
>
> The only article on this I've been able to find so far is this one: http://blog.taragana.com/index.php/archive/how-to-stop-crashing-hanging-of-php-cgi-spawn-fcgi-with-nginx-lighttpd/
>   
I had a very similar problem when using lighttpd+spawn-fcgi. We had 
moved from Apache about 3 years ago because of load issues with mod_php 
to lighttpd+spawnfcgi+php. This worked fine for a while, but as our load 
increased we started to see PHP being unresponsive, and lighty returning 
503s.

After some digging I found the cause was the way spawnfcgi handled 
incomming requests in conjunction with PHP's internal 
PHP_FCGI_MAX_REQUESTS setting. This setting is set to 250 by default in 
PHP, and when this limit is hit PHP recycles the child thread. During 
this recycle spawnfcgi would continue to handle requests, pass them to 
the recycling thread, get no response, and in turn lighty would return a 
503. IIRC this would also cause the recycling thread to die, slowly 
killing off all the threads and PHP completely.

Playing with PHP_FCGI_MAX_REQUESTS helped in the short-term but we 
continued to experience the problem, so when I heard of nginx I quickly 
switched to nginx+php-fcgi (using the -b switch). Since then I've yet to 
see an issue, and we've had ~100 production sites running on that stack 
for over 2 years. Since starting to use supervisord we've begun the 
process of moving the rest of our ~500 domains/sites over to to that stack.

Using PHP `-b` in my experience is rock-solid. Running it under 
supervisord means that if it dies it is brought back up straight away. 
This does not affect nginx, and you can always show a nice error page 
should this case happen - I remember that if php went down hard under 
mod_php it would cause apache to fail also. Bad times.

My suggestion is to get rid of spawnfcgi and install supervisord. Here's 
the supervisord config I use on my dev machine:

[fcgi-program:php]
user=nginx
command=/usr/bin/php-cgi -c /etc/php5/php.ini -b /tmp/php.socket
socket=unix:///tmp/php.socket
autostart=true
startsecs=5
startretries=3
stopwaitsecs=10
environment=PHP_FCGI_MAX_REQUESTS=50,PHP_FCGI_CHILDREN=4

Some on the board suggest having nginx running under supervisord also, 
but I'm a little worried about that approach; if supervisord goes down 
for any reason on a live server everything goes down, whereas if nginx 
is kept separate and supervisord goes down nginx can continue to serve 
requests (albeit error pages) while supervisord comes back up and brings 
your app back online (php, database, memcached, etc).

Our production runs:
  supervisord > php-fcgi, mysqld, memcached, and a number of python 
"services" (our webapp layer)
  monit > nginx, supervisord, sshd, other "system" level important daemons
which means if supervisord goes down for any reason, it'll only be down 
for 1min until monit can bring it back up. Since memcached is killed our 
webapp is brought back up in a "clean" state. Users are asked to log in 
again, but this rarely (or, rather, is yet to) happens and is a mild 
inconvenience for them.

Give it a try. All these technologies are highly configurable and there 
are many permutations. To find your optimal configuration is going to 
take experimentation. It's not like Apache where there's only one 
"right" way to do things; you've got more control - make use of it!

-- 

*Phillip B Oldham*
ActivityHQ
phill at activityhq.com <mailto:phill at theactivitypeople.co.uk>

------------------------------------------------------------------------

*Policies*

This e-mail and its attachments are intended for the above named 
recipient(s) only and may be confidential. If they have come to you in 
error, please reply to this e-mail and highlight the error. No action 
should be taken regarding content, nor must you copy or show them to anyone.

This e-mail has been created in the knowledge that Internet e-mail is 
not a 100% secure communications medium, and we have taken steps to 
ensure that this e-mail and attachments are free from any virus. We must 
advise that in keeping with good computing practice the recipient should 
ensure they are completely virus free, and that you understand and 
observe the lack of security when e-mailing us.

------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://nginx.org/pipermail/nginx/attachments/20091209/d7343cf6/attachment.html>