sudden nginx hang -- restart fails, "98: Address already in use"

[Maxim Dounin]

Hello!

On Wed, Feb 05, 2014 at 01:53:14PM -0500, dwirth wrote:

> Hello, all. 
> 
> About an hour ago, out of the blue, my server stopped responding to webpage
> requests. We are using nginx + php-fpm on RHEL6.
> 
> # service nginx status
> nginx (pid 31600) is running...
> 
> # service nginx restart
> Stopping nginx:                                            [FAILED]
> Starting nginx: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address
> already in use)
> nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
> nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
> nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
> nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
> nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
> nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
> nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
> nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
> nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
> nginx: [emerg] still could not bind()
> 
> I killed the process and was able to restart nginx so the immediate crisis
> is over, but I need to know: What the hell happened? What would cause nginx
> to hang like this? I have googled around and I see several discussions about
> what to do when this happens but zilch about how to keep it from happening.

Such hangs can be caused either by bugs (either in nginx itself, 
or in 3rd party modules; take a look at nginx -V to find out how 
many 3rd party modules you have) or by some serious blocking at 
OS level.  E.g., serving files from an NFS share may easily result 
in such a hang if something happens with the NFS server.

It is impossible to say what happened in your case without 
additional information (at least "ps alx" output whould be 
helpful, and see also http://wiki.nginx.org/Debugging).  General 
recomendations are:

1. Make sure your nginx is up-to-date.  Note that some linux 
distros ship quite outdated versions in their repositories, make 
sure to check version against nginx.org.  Current versions are 
1.5.10 (mainline) and 1.4.4 (stable).

2. Make sure you aren't using things that can easily block, like 
NFS or other network filesystems, or some blocking code in 
embedded languages like embedded perl or lua.  Or, if you do use 
them, expect anything to die if something bad happens.

3. If you are using 3rd party modules, make sure you have good 
reasons to do so.

4. Examine logs for "crit", "alert", "emerge" messages.  If there 
are any, they require investigation, especially messages about 
worker processes "exited on signal".

-- 
Maxim Dounin
http://nginx.org/