Nginx reverse proxy crash when dns unavailable

[Gabriel Ramuglia]

Actually in your case, if nginx is fully exiting, you can have a
script to just check if nginx is up, and if not, restart it. In my
case, nginx stays in a zombie status, considered up but not working
right.

Here's a script I use to check if nginx and apache are up, and if
nginx is down, kill apache, start nginx, start apache. If apache is
down and nginx up, it just restarts apache.

Replace instances of "/usr/local/apache-php/bin/httpd" with wherever
you've installed apache.

#!/bin/bash

this=$(ps aux | grep httpd | awk '{print $11}' | grep
"/apache-php/bin/httpd" | head -1)
if [ "$this" = "/usr/local/apache-php/bin/httpd" ]; then
        echo "httpd (php) was found"
else
        echo "httpd (php) was not found"
        /usr/local/apache-php/bin/apachectl restart
        sleep 1
        /usr/local/apache-php/bin/apachectl start
        fi

thing=$(ps aux | grep nginx | awk '{print $14}' | grep nginx | head -1)
if [ "$thing" = "/usr/local/nginx/sbin/nginx" ]; then
        echo "nginx was found"
else
    echo "nginx not found"
    /usr/local/apache-php/bin/apachectl stop
    sleep 4
    killall /usr/local/apache-php/bin/httpd
    sleep 4
    killall -KILL /usr/local/apache-php/bin/httpd
    sleep 4
    ulimit -HSn 8192
    /usr/local/nginx/sbin/nginx
    sleep 4
    /usr/local/apache-php/bin/apachectl restart
    sleep 1
    /usr/local/apache-php/bin/apachectl start
    /usr/local/spri/spri -q
    fi

You can just put this in a cron every 2 minutes, and this would seem
to solve your issue.

On Thu, Oct 22, 2009 at 1:10 PM, Gabriel Ramuglia <gabe at vtunnel.com> wrote:
> I would say I'm seeing a similar issue. I don't know if it hard locks
> after one failed request or if it takes several, but, I do see a
> continued lockup on an nginx worker if that worker encounters a
> failure condition once. One would hope the worker would become
> responsive again once the upstream service (apache or dns) becomes
> responsive again, but it does not.
>
> However, in your case, if Nginx tries to start before it has an ip
> address to bind to, yes, you can expect it not to work. At the very
> least, nginx can only bind to the addresses on the system, so you'll
> need to start / restart nginx after you've completed DHCP.
>
> On Thu, Oct 22, 2009 at 12:53 PM, masom <nginx-forum at nginx.us> wrote:
>> But shoulnd't nginx start anyway if the end point is not responding and just try to reach it anyway?
>>
>>
>> I can't really see why it would need to stop or crash when either the endpoint (apache) or the dns system is unavailable.
>>
>> Yes it should display 5xx errors saying the endpoint is unreachable (dns or server failure / not-responding) but nginx should not "lock up" after 1 bad answer.
>>
>> Current problem:
>>
>> unit starts
>> dhcp kicks in
>> nginx get started before dhcp process is completed
>> nginx realize that content.dev.local is not reachable (dns settings are not yet set by dhcp)
>> nginx exits
>> Browser on unit starts, says address is unreachable (as nginx did not start).
>>
>>
>> Shouldn't nginx just attempt to connect to the end point as requests are coming in?
>>
>> The solution we consider is the hosts file that would always point to a static ip for the content server, but would be a little management problem as we are deploying in several different location with different networks.
>>
>> Posted at Nginx Forum: http://forum.nginx.org/read.php?2,15995,16042#msg-16042
>>
>>
>>
>