Nginx reverse proxy crash when dns unavailable

[Gabriel Ramuglia]

Alright, that makes sense. I'm more referring to resolving names after
startup, in the process of handling reproxy requests. In that case, I
believe I have to set a resolver in the nginx.conf, and if that
resolver is not working right, many times nginx will hang, and require
a restart, but without actually crashing which would be much better as
my uptime script would notice and restart it on its own.

2009/10/22 Maxim Dounin <mdounin at mdounin.ru>:
> Hello!
>
> On Thu, Oct 22, 2009 at 12:05:57PM -0700, Gabriel Ramuglia wrote:
>
>> I just re-read your reply. Are you saying that if I don't use a
>> resolver in the nginx.conf, it will use the system resolvers? If the
>> first system resolver doesn't work, will nginx automatically fall back
>> to other resolvers in the /etc/resolv.conf file without encountering
>> this failure condition (obviously in this case there will be some
>> delay waiting for the first resolver to time out)?
>
> While parsing configuration nginx just uses system resolver via
> gethostbyname() function.  It's up to system to handle this.
>
> Maxim Dounin
>
>>
>> On Thu, Oct 22, 2009 at 12:04 PM, Gabriel Ramuglia <gabe at vtunnel.com> wrote:
>> > I've seen this as well. If my DNS server becomes unavailable, even
>> > temporarily, at least one nginx worker will crash, using 100% cpu time
>> > and never responding to requests again. It used to be that my DNS
>> > server would be automatically restarted, and this problem would not
>> > necessarily affect all workers per incident, i.e. if I have 8 workers,
>> > maybe 1 or 2 would permanently lock up when the DNS server fails. The
>> > rest would continue processing requests once DNS became available, and
>> > the crashed workers would spin 1 core cpu at 100% until I restart
>> > nginx.
>> >
>> > It would be handy if this were solved, also, it would make sense if
>> > you could specify more than one resolver for nginx to use, or, if
>> > nginx would default to using the resolvers in the /etc/resolv.conf.
>> >
>> > Since we're on the subject, it seems the same behavior happens during
>> > an upstream failure. nginx children are working fine, then an upstream
>> > such as apache on the machine temporarily fails or is unresponsive,
>> > then nginx stops working properly even after apache rights itself, and
>> > the whole thing can only be solved by restarting nginx. In this case,
>> > the nginx workers will either not respond to requests at all (sending
>> > a blank page) or will respond 500 bad gateway, even though connecting
>> > to the gateway directly on it's own port works fine, and restarting
>> > nginx solves the issue.
>> >
>> > Nginx has definitely been great for me, and if these two things didn't
>> > tend to happen from time to time, I would consider it bulletproof.
>> >
>> > Thanks,
>> > Gabe
>> >
>> > On Thu, Oct 22, 2009 at 11:22 AM, Maxim Dounin <mdounin at mdounin.ru> wrote:
>> >> Hello!
>> >>
>> >> On Thu, Oct 22, 2009 at 02:02:14PM -0400, masom wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I am currently planning to use nginx on several thousand devices as a reverse-proxy caching system.
>> >>>
>> >>> It currently work as expected (thanks Igor!), caches files as they are being requested by the devices.
>> >>>
>> >>>
>> >>> The only problem we hit is when nginx starts faster than the dns sytem is available on the units. Nginx will crash saying it is unable to connect to the remote host being proxied.
>> >>>
>> >>>
>> >>>  1739#0: host not found in upstream "content.dev.local" in /usr/local/nginx/conf/nginx.conf 33
>> >>
>> >> Crashes and refuses to start is quite a different things.  As you
>> >> have no DNS available during start - nginx just can't proceed any
>> >> further since it doesn't know what your config means.  Once
>> >> started it won't depend on DNS anymore.
>> >>
>> >> To avoid such issues on start there are two basic options:
>> >>
>> >> 1. Use ip addresses in config instead of host names.
>> >>
>> >> 2. Make sure your OS resolving subsystem always returns meaningful
>> >> results to nginx - either by launching nginx once DNS is available
>> >> or by adding relevant entries to /etc/hosts.
>> >>
>> >> Maxim Dounin
>> >>
>> >>>
>> >>>
>> >>> Staring nginx again and it work (as the DNS is now responding properly).
>> >>>
>> >>>
>> >>> Any idea on how to work around this or should i fill a bug report (Nginx shouldn't crash when the remote is not available, but should try on requests to access it).
>> >>>
>> >>> Nginx does not die when the remote drops and come back (by pulling the network cable for example). It only crash when nginx is launched and the dns sytem is not yet available.
>> >>>
>> >>> Posted at Nginx Forum: http://forum.nginx.org/read.php?2,15995,15995#msg-15995
>> >>>
>> >>>
>> >>
>> >>
>> >
>>
>
>