Nginx + php-fpm "504 Gateway Time-out" error with almost zero load(on a test-server)

[Rahul Bansal]

Hi Reinis,

Thanks for your reply.

Debugging php-fpm log was first thing I did.

I am using "log_level = debug" and "request_slowlog_timeout = 2s" only already.
Only thing - "request_terminate_timeout" was not active which I am
setting now = 10s.

I am able to catch slow queries earlier and they didn't look like
creating issues.

I think problem is mainly because some open (hanged) sockets/process like below:
tcp     1337      0 localhost:9000          localhost:46680 CLOSE_WAIT  -
tcp        0      0 localhost:46680         localhost:9000 FIN_WAIT2   -

FIN_WAIT2 => Connection is closed, and the socket is waiting for a
shutdown from the remote end.
CLOSE_WAIT => The remote end has shut down, waiting for the socket to close.

With my pathetic patchwork, as I am restarting php5-fpm every minute,
above kind of "blockage" is not getting created.

On a sidenote, as I have 100s of virtual hosts, is there anyway - I
can temporary configure nginx to dump all error logs in one place so
that I can analyze them easily? As of now, each virtual host has its
logs files in its own dir.
I cannot rule-out some php code going wrong as this issue started
right in the middle of working hours and its really hard to find out
exactly which virtual host might have some bad codes.

By the way using netstats, output, can I determine, for which domain
the port was assigned?

Thanks for your inputs. I really appreciate them as this is toughest
time I am having in my career.

-Rahul


On Tue, Sep 7, 2010 at 4:03 PM, Reinis Rozitis <r at roze.lv> wrote:
>>
>> After debugging for 6-hours - I am giving this up :|
>
> First of all since this is clearly a php-fpm issue you should have checked (enabled) the php-fpm logs.
>
> In your php-fpm.conf there should be lines:
>
> error_log = /some/path/error.log
> log_level = notice
>
> and for debugging / performance purposes its good to add:
>
> request_terminate_timeout = 60s
> request_slowlog_timeout = 20s
> slowlog = /some/path/slow.log
>
>
> This way php-fpm will start to log/backtrace (print the codeparts in logfile) of requests which take longer than 20 seconds (you can adjust by your own needs) and kill those taking longer that minute.
>
> It should pretty much give you an idea what is happening and/or slowing down the php to level it hangs up (the typical case is usually some code involving sockets or db backends).
>
>
> rr
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://nginx.org/mailman/listinfo/nginx