proxy_pass behavior

[Jim Popovitch]

On Fri, Mar 28, 2014 at 2:36 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> Hello!
>
> On Fri, Mar 28, 2014 at 01:56:59PM -0400, Jim Popovitch wrote:
>
>> On Fri, Mar 28, 2014 at 8:20 AM, Maxim Dounin <mdounin at mdounin.ru> wrote:
>> > Hello!
>> >
>> > On Thu, Mar 27, 2014 at 08:18:48PM -0400, Jim Popovitch wrote:
>> >
>> >> On Thu, Mar 27, 2014 at 1:50 PM, Jim Popovitch <jimpop at gmail.com> wrote:
>> >> > On Thu, Mar 27, 2014 at 1:27 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
>> >> >> Hello!
>> >> >>
>> >> >> On Thu, Mar 27, 2014 at 01:12:18PM -0400, Jim Popovitch wrote:
>> >> >>
>> >> >>> On Thu, Mar 27, 2014 at 12:59 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
>> >> >>> > Hello!
>> >> >>> >
>> >> >>> > On Thu, Mar 27, 2014 at 12:29:03PM -0400, Jim Popovitch wrote:
>> >> >>> >
>> >> >>> >> Should proxy_pass retry ipv4 if ipv6 fails?   Currently (1.5.12), it
>> >> >>> >> does not, and there is no way to force proxy_pass to always use ipv4.
>> >> >>> >
>> >> >>> > In no particular order:
>> >> >>> >
>> >> >>> > - The proxy_next_upstream is expected to retry by default.
>> >> >>>
>> >> >>> Yes, that works if the host name has 2 or more A  OR  2 or more AAAA
>> >> >>> records.  proxy_next_upstream does not appear to transition to the
>> >> >>> next protocol (i.e. from ipv4 to ipv6)
>> >> >>
>> >> >> It doesn't care about address family.  As long as connect() to
>> >> >> one address fails, it will try next one.
>> >> >
>> >> > I will have to do some more tests, but so far 1.5.12 (debian wheezy)
>> >> > does not appear to try the next one if the next one is a different
>> >> > protocol family.
>> >>
>> >> So here is what happens:
>> >>
>> >> Nginx 1.5.12 on Debian Wheezy 7.4
>> >>
>> >> ~$ host www.acadiau.ca
>> >> www.acadiau.ca has address 131.162.201.18
>> >> www.acadiau.ca has IPv6 address 2620:21:c000:c8:0:aca:d1a:18
>> >>
>> >> ~$ grep proxy_pass /etc/nginx/sites-available/proxy
>> >> proxy_pass https://www.acadiau.ca;
>> >>
>> >> When an IPv6 client connects to the proxy, proxy_pass logs the error as
>> >>
>> >> 2014/03/27 21:28:28 [alert] 24862#0: *158336 connect() failed (13:
>> >> Permission denied) while connecting to upstream, client:
>> >> 2604:180::82c6:fe9, server: proxy-service.tld, request: "GET /
>> >> HTTP/1.1", upstream: "https://[2620:21:c000:c8:0:aca:d1a:18]:443/",
>> >> host: "proxy-service.tld"
>> >>
>> >> It will next try the ipv4 addr and succeed, however the reason for the
>> >
>> > So the original claim that it "does not appear to try the next
>> > one" is withdrawn, right?
>>
>> 99% of it is, I did see it fail once.
>>
>> >> "Permission denied" error appears to be the format of the URL (raw
>> >> ipv6 addr, not hostname).  Why is it using the ipv6 addr instead of
>> >> the hostname?
>> >
>> > No, it's an error from connect() syscall, returned by your OS for
>> > some reason (likely because there is no route available, or due to
>> > a firewall rule).  The address is one of the upstream server nginx
>> > uses at the moment - it's is vital for logging if there are more
>> > than one address.
>>
>> I agree it's vital for logging so that you can see which address
>> failed.  It's just not clear that the destination server was presented
>> with the hostname (SNI) because the "Permission denied" message seems
>> to idicate connect() was successful but the resource was denied (due
>> to permissions).   But I can live with bad OS level error messages as
>> long as the behavior is accurate.
>
> The message in question indicates that connect() syscall failed
> (note "connect() failed"), and errno was set to EPERM (note "13:
> Permission denied" in brackets).
>
> As previosly said, EPERM was seen to be returend from connect()
> due to either firewall rules in place that deny the connection, or
> due to lack of route to the destination address.

Yes. That is my understanding.  It is an OS level error message from
the connect() call, not an nginx generated message.

I'm still going to try and submit a patch for proxy_protocol in order
to force proxies to use a certain protocol rather than allowing any.
;-)

-Jim P.