nginx upgrade fails due bind error on 127.0.0.1 in a FreeBSD jail

[Maxim Dounin]

Hello!

On Mon, Dec 05, 2016 at 02:40:27PM +0000, Steven Hartland wrote:

> On 05/12/2016 13:27, Maxim Dounin wrote:
> > Hello!
> >
> > On Sun, Dec 04, 2016 at 09:39:59PM +0000, Steven Hartland wrote:

[...]

> >> I believe the change to add a localhost bind to the server in question
> >> was relatively recent so I suspect it has something to do with that.
> >>
> >> The config for this is simply:
> >> server {
> >>       listen 127.0.0.1:81;
> >>       server_name localhost;
> >>
> >>       location /status {
> >>           stub_status;
> >>       }
> >> }
> >>
> >> The upgrade in this case was:
> >> nginx: 1.10.1_1,2 -> 1.10.2_2,2
> >>
> >> Now this server is running under FreeBSD in a jail (10.2-RELEASE) and it
> >> has 127.0.0.1 available yet it seems nginx has incorrectly bound the
> >> address:
> >> netstat -na | grep LIST | grep 81
> >> tcp4       0      0 10.10.96.146.81        *.* LISTEN
> > In a FreeBSD jail with a single IP address any listening address
> > is implicitly converted to the jail address.  As a result, if you
> > write in config "127.0.0.1" - upgrade won't work, as it will see
> > inherited socket listening on the jail address (10.10.96.146 in
> > your case) and will try to create a new listening socket with the
> > address from the configuration and this will fail.
> 
> Thanks for the response Maxim.
> 
> In our case we don't have a single IP in the jail we have 4 addresses:
> 1 x localhost address (127.0.0.1)
> 2 x external
> 1 x private address (10.10.96.146)
> 
> We have a number of binds the externals are just port binds the internal 
> a localhost e.g.
> listen 443 default_server accept_filter=httpready ssl;
> listen 80 default_server accept_filter=httpready;
> ...
> listen 80;
> listen 443 ssl;
> ...
> listen 127.0.0.1:81;
> 
> We're expecting the none IP specified listens to bind to * (this is what 
> happens)
> 
> But the "listen 127.0.0.1:81" results in "10.10.96.146:81" instead.
> 
> Given your description I would only expect this 127.0.0.1 wasn't present 
> in the jail and 10.10.96.146 was the only IP available.
> 
> Did I miss-understand your description?

Given that the real local address of the listening socket as shown 
by netstat is 10.10.96.146, it means that the socket was created 
when there were no explicit 127.0.0.1 in the jail.

And, given that you are able to connect to it via "lwp-request 
http://127.0.0.1:81/status", it looks like that 127.0.0.1 is still 
not in the jail, but mapped to 10.10.96.146 instead.

Note that the fact that you can use 127.0.0.1 in a jail doesn't 
mean that it is a real address available.  Normally, 127.0.0.1 
will be implicitly converted to the main IP of the jail, and most 
utilities won't notice.

(Note well that since there is no real 127.0.0.1 in the jail, it 
doesn't provide any additional isolation compared to the jail IP 
address.  That is, a service which is listening on 127.0.0.1 is in 
fact listening on 10.10.96.146, and it is reachable from anywhere, 
not just the jail itself.)

> > There are two possible solutions for this problem:
> >
> > - configure listening on the jail IP address to avoid this
> >    implicit conversion;
> As above I'm not sure I follow you correctly as 127.0.0.1 is one of the 
> IP's available in the jail.

See above, looks like it's not, and it is implicitly converted to 
10.10.96.146 instead.

-- 
Maxim Dounin
http://nginx.org/