Curious problem with PHP

[Steve Holdoway]

Hi,

On Sat, 2014-06-28 at 20:39 +0100, Ian wrote:
> Hi all,
> 
> I have been investigating a performance problem with my 
> nginx/init-fastcgi/PHP set up and I'm not having much success. All ideas 
> gratefully received.
> 
> It appears as if the PHP worker processes return their reply (in 100ms 
> or so), and then hang about for 500-700ms before being able to handle 
> the next message. This results in worker starvation under remarkably low 
> loads.
> 
> The application is a chat room. I am using the Nginx HTTP push module. 
> POSTed messages are passed to PHP, which uses its CURL module to write 
> to the /publish page. PHP also talks to MySQL for data storage.
> Javascript fetches from the /subscribe address, updates the screen and 
> requests again. GETs from /subscribe do not involve PHP.
> 
> I know the delay is after the reply, because Firebug shows the response 
> time is 100 to 200ms - as expected. Ping times are 100ms plus).
> 
> With 5 PHP worker processes and three people paste/sending as fast as 
> they can, it is very easy to overload the system.  (I can nearly do it 
> on my own).  Each browser stores the POSTs waiting for the previous 
> reply thus each browser can only have one message in PHP at a time. Thus 
> there cannot be more than three messages in the PHP system at once.  So 
> how can 5 workers be overloaded?
> 
> I have stopped and counted 5 of my own messages coming through over the 
> next 7-8 seconds.  It appears from comments made on the phone, that all 
> messages arrive together - just late.  No messages are dropped.
> 
> So it appears that worker starvation is happening - but not long 
> response times. Server load is also slight (machine load is pushed up 
> from 0.3% to 2% out of 400% - 4CPUs).
> 
> Conclusion: the worker processes are hanging about after sending their 
> reply for some considerable time (guess 500-700ms) before they can 
> handle a new message.
> 
> In an effort to identify what might cause the PHP worker process to hang 
> about I have:
>       released the session as early as possible - no change.
>       switched the MySQL processing from a transaction to auto-commit - 
> this made things slightly worse.
>       doubled the VPS memory - this had minimal effect but removes the 
> possibility of swapping
>       attempted to use keep-alive between nginx and the fastcgi process 
> (and failed. Gave up as this will probably going to make things worse - 
> no keep alive is the default).
> 
> Using nginx 1.4.4 , PHP 5.3.10
> 
> Has anyone got an idea as what might be happening? I'm stumped.
> 
> Thanks
> 
> Ian

First check: nginx/init-fastcgi/PHP setup... you *do* mean nginx /
php-fpm don't you??

There'll be at least some suggestions to get a bit more up-to-date with
nginx 1.6+, php 5.4+, but, ignoring those...

My first step ( I'm a sysadmin, so look at every application as a black
box and measure what's happening to the infrastructure ) would be to
install some kind of monitoring, munin being my preference, which has
loads of useful extras to monitor exactly what the server, nginx, mysql,
php-fpm, APC ( you have installed it I hope! ), and everything else is
doing. It's all well and good adding extra memory for example, but if
you don't configure the infrastructure to use it, then it may just get
wasted.

It could be a database problem. How much resource is allocated to this,
and are you saving session data in it? There's a simple script out there
called mysql_tuner.sh which will give you a quick overview of how the
database resources are being used.

Hopefully these will aid you in identifying the bottleneck.

Steve
-- 
Steve Holdoway BSc(Hons) MIITP
http://www.greengecko.co.nz
Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa