nginx upgrade fails due bind error on 127.0.0.1 in a FreeBSD jail

[Steven Hartland]

On 05/12/2016 13:27, Maxim Dounin wrote:
> Hello!
>
> On Sun, Dec 04, 2016 at 09:39:59PM +0000, Steven Hartland wrote:
>
>> We've used nginx for years and never had an issue with nginx upgrade
>> until today where the upgrade command ran but almost instantly after the
>> new process exited.
>>
>> /usr/local/etc/rc.d/nginx upgrade
>> Performing sanity check on nginx configuration:
>> nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
>> nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
>> Upgrading nginx binary:
>> Stopping old binary:
>>
>> In the default nginx log we had:
>> 2016/12/04 21:18:22 [emerg] 25435#0: bind() to 127.0.0.1:81 failed (48:
>> Address already in use)
>> nginx: [emerg] bind() to 127.0.0.1:81 failed (48: Address already in use)
>> 2016/12/04 21:18:22 [emerg] 25435#0: bind() to 127.0.0.1:81 failed (48:
>> Address already in use)
>> nginx: [emerg] bind() to 127.0.0.1:81 failed (48: Address already in use)
>> 2016/12/04 21:18:22 [emerg] 25435#0: bind() to 127.0.0.1:81 failed (48:
>> Address already in use)
>> nginx: [emerg] bind() to 127.0.0.1:81 failed (48: Address already in use)
>> 2016/12/04 21:18:22 [emerg] 25435#0: bind() to 127.0.0.1:81 failed (48:
>> Address already in use)
>> nginx: [emerg] bind() to 127.0.0.1:81 failed (48: Address already in use)
>> 2016/12/04 21:18:22 [emerg] 25435#0: bind() to 127.0.0.1:81 failed (48:
>> Address already in use)
>> nginx: [emerg] bind() to 127.0.0.1:81 failed (48: Address already in use)
>> 2016/12/04 21:18:22 [emerg] 25435#0: still could not bind()
>> nginx: [emerg] still could not bind()
>>
>> Running the start just after resulted in a running version but is
>> obviously unexpected to have upgrade result in a failure.
>>
>> I believe the change to add a localhost bind to the server in question
>> was relatively recent so I suspect it has something to do with that.
>>
>> The config for this is simply:
>> server {
>>       listen 127.0.0.1:81;
>>       server_name localhost;
>>
>>       location /status {
>>           stub_status;
>>       }
>> }
>>
>> The upgrade in this case was:
>> nginx: 1.10.1_1,2 -> 1.10.2_2,2
>>
>> Now this server is running under FreeBSD in a jail (10.2-RELEASE) and it
>> has 127.0.0.1 available yet it seems nginx has incorrectly bound the
>> address:
>> netstat -na | grep LIST | grep 81
>> tcp4       0      0 10.10.96.146.81        *.* LISTEN
> In a FreeBSD jail with a single IP address any listening address
> is implicitly converted to the jail address.  As a result, if you
> write in config "127.0.0.1" - upgrade won't work, as it will see
> inherited socket listening on the jail address (10.10.96.146 in
> your case) and will try to create a new listening socket with the
> address from the configuration and this will fail.

Thanks for the response Maxim.

In our case we don't have a single IP in the jail we have 4 addresses:
1 x localhost address (127.0.0.1)
2 x external
1 x private address (10.10.96.146)

We have a number of binds the externals are just port binds the internal 
a localhost e.g.
listen 443 default_server accept_filter=httpready ssl;
listen 80 default_server accept_filter=httpready;
...
listen 80;
listen 443 ssl;
...
listen 127.0.0.1:81;

We're expecting the none IP specified listens to bind to * (this is what 
happens)

But the "listen 127.0.0.1:81" results in "10.10.96.146:81" instead.

Given your description I would only expect this 127.0.0.1 wasn't present 
in the jail and 10.10.96.146 was the only IP available.

Did I miss-understand your description?
> There are two possible solutions for this problem:
>
> - configure listening on the jail IP address to avoid this
>    implicit conversion;
As above I'm not sure I follow you correctly as 127.0.0.1 is one of the 
IP's available in the jail.
> - configure listening on "*" and use multiple addresses in the jail.
Unfortunately this is something we want to explicitly prevent with this 
bind, its an internal service only.
> In both cases there will be no implicit conversion and as a result
> everything will work correctly.
     Regards
     Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20161205/5c31a910/attachment.html>