another "bind() to 0.0.0.0:80 failed (98: Address already in use)" issue

[sum-it]

Hello All,

I have another "bind() to 0.0.0.0:80 failed (98: Address already in use)"
issue. 

I am working on a minimal system including nginx only. System startup time,
and readiness time are important points. Whilte testing I figured out
sometime system boots up within 500ms and sometimes it takes around 3
second. On further probing I find out nginx is taking different time to
start up which costs me extra 2.5 Seconds. So I tested and figured out that
error in those cases is "bind() to 0.0.0.0:80 failed (98: Address already in
use).

Few of my observation here are,
1. No other process is using that port, there is no other web server or
application running on the system.
2. The case is not only limited to nginx restart, where nginx might not be
shutdown correctly and itself might be using that port. Nginx even fails
during system start, in cases where it has caused longer boot time.
3. I use customized kernels, but that kernel shouldn't be culprit either
because sometimes it works on that kernel as well. Another point here is
failure in customized kernel is more often as compared to stock kernel. The
ratio of failure in stock kernel is around 30% and in customized is 70% but
system works on both and fails on both. 
4. Start/Stop scripts always exit with success status "0". 
5. I tested nginx in a restart loop, with a 1 second sleep before and after
start and stop. Failure is random.
6. Worse, nginx is actually running even though error said bind failed. I
can connect to it, access default web page, and it is listed in netstat as
listening as well.

Output of netstat -ntl is at: http://pastebin.com/26b6KNAZ

Error Log is at: http://pastebin.com/w0y8aa9p

This is one of the customized system, a derivative of debian, I am working
on. System wise, everything is consistent. I use same kernel, same system
image with same parameters and it works sometime and fails otherwise.

nginx -t gives 
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
So configuration shouldn't be a problem.

configuration file is default and available  at:
http://pastebin.com/iRFfW3UE

Process listing after nginx startup: http://pastebin.com/0vB19rLq
Process listing after nginx stop: http://pastebin.com/iQafxjiF

Any pointer to debug the issue would be very helpful.

Regards,
sum-it

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,272570,272570#msg-272570