nginx upgrade fails due bind error on 127.0.0.1 in a FreeBSD jail

[Maxim Dounin]

Hello!

On Sun, Dec 04, 2016 at 09:39:59PM +0000, Steven Hartland wrote:

> We've used nginx for years and never had an issue with nginx upgrade 
> until today where the upgrade command ran but almost instantly after the 
> new process exited.
> 
> /usr/local/etc/rc.d/nginx upgrade
> Performing sanity check on nginx configuration:
> nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
> nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
> Upgrading nginx binary:
> Stopping old binary:
> 
> In the default nginx log we had:
> 2016/12/04 21:18:22 [emerg] 25435#0: bind() to 127.0.0.1:81 failed (48: 
> Address already in use)
> nginx: [emerg] bind() to 127.0.0.1:81 failed (48: Address already in use)
> 2016/12/04 21:18:22 [emerg] 25435#0: bind() to 127.0.0.1:81 failed (48: 
> Address already in use)
> nginx: [emerg] bind() to 127.0.0.1:81 failed (48: Address already in use)
> 2016/12/04 21:18:22 [emerg] 25435#0: bind() to 127.0.0.1:81 failed (48: 
> Address already in use)
> nginx: [emerg] bind() to 127.0.0.1:81 failed (48: Address already in use)
> 2016/12/04 21:18:22 [emerg] 25435#0: bind() to 127.0.0.1:81 failed (48: 
> Address already in use)
> nginx: [emerg] bind() to 127.0.0.1:81 failed (48: Address already in use)
> 2016/12/04 21:18:22 [emerg] 25435#0: bind() to 127.0.0.1:81 failed (48: 
> Address already in use)
> nginx: [emerg] bind() to 127.0.0.1:81 failed (48: Address already in use)
> 2016/12/04 21:18:22 [emerg] 25435#0: still could not bind()
> nginx: [emerg] still could not bind()
> 
> Running the start just after resulted in a running version but is 
> obviously unexpected to have upgrade result in a failure.
> 
> I believe the change to add a localhost bind to the server in question 
> was relatively recent so I suspect it has something to do with that.
> 
> The config for this is simply:
> server {
>      listen 127.0.0.1:81;
>      server_name localhost;
> 
>      location /status {
>          stub_status;
>      }
> }
> 
> The upgrade in this case was:
> nginx: 1.10.1_1,2 -> 1.10.2_2,2
> 
> Now this server is running under FreeBSD in a jail (10.2-RELEASE) and it 
> has 127.0.0.1 available yet it seems nginx has incorrectly bound the 
> address:
> netstat -na | grep LIST | grep 81
> tcp4       0      0 10.10.96.146.81        *.* LISTEN

In a FreeBSD jail with a single IP address any listening address 
is implicitly converted to the jail address.  As a result, if you 
write in config "127.0.0.1" - upgrade won't work, as it will see 
inherited socket listening on the jail address (10.10.96.146 in 
your case) and will try to create a new listening socket with the 
address from the configuration and this will fail.

There are two possible solutions for this problem:

- configure listening on the jail IP address to avoid this 
  implicit conversion;
- configure listening on "*" and use multiple addresses in the jail.

In both cases there will be no implicit conversion and as a result 
everything will work correctly.

-- 
Maxim Dounin
http://nginx.org/