Debugging 504 Gateway Timeout and its actual cause and solution

[Keyur]

Hi,

We are running following stack on our web server Varnish + Nginx + FastCGI
(php-fpm) on RHEL 6.6

Its a dynamic website with different result sets everytime and has around 2
million url's indexed with Google.
* Its running on nginx/1.5.12 and PHP 5.3.3 (Will be upgraded to latest
nginx and PHP soon)
* Nginx connects to php-fpm running locally on same server on port 9000

We are getting 504 Gateway timeout intermittently on some pages which we are
unable to resolve. The URL's which give 504 works fine after sometime.
We get to know about 504 from our logs and we haven't been able to replicate
this as it randomly happens on any URL and works after sometime.

I have had couple of discussions with developer but as per him the
underlying php script hardly does anything and it should not take this long
(120 seconds) but still it is giving 504 Gateway timeout. 

Need to establish where exactly the issue occurs :
* Is it a problem with Nginx ?
* Is it a problem with php-fpm ?
* Is it a problem with underlying php scripts ?
* Is it possible that nginx is not able to connect to php-fpm ?
* Would it resolve if we use Unix socket instead of TCP/IP connection to ?

The URL times out after 120 seconds with 504

Below is the error seen :
2016/01/04 17:29:20 [error] 1070#0: *196333149 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 66.249.74.95,
server: x.x.x.x, request: "GET /Some/url HTTP/1.1", upstream:
"fastcgi://127.0.0.1:9000", host: "example.com"


Earlier with fastcgi_connect_timeout of 150 seconds - it used to give at 502
status code after 63 seconds with default net.ipv4.tcp_syn_retries = 5 on
RHEL 6.6 ; afterwards we set net.ipv4.tcp_syn_retries =  6 and then it
started giving 502 after 127 seconds.

Once I set fastcgi_connect_timeout = 120 it started giving 504 status code.
I understand fastcgi_connect_timeout with such high value is not good.

Need to findout why exactly we are getting 504 (I know its timeout but the
cause is unknown). Need to get to the root cause to fix it permanently. 

How do I confirm where exactly the issue is ?

If its poorly written code then I need to inform the developer that 504 is
happening due to issue in php code and not due to nginx or php-fpm and if
its due to Nginx or Php-fpm then need to fix that.



Thanks in Advance!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,263842,263842#msg-263842