After 1 minute, I get this error: "connect() to 127.0.0.1:8080 failed (99: Cannot assign requested address) while connecting to upstream"

[Maxim Dounin]

Hello!

On Wed, Feb 12, 2014 at 02:39:41PM -0500, Jack Andolini wrote:

> Hi,
> 
> First of all, my environment:
> 
> - About 1.6 GB RAM, which doesn't seem to be a bottleneck because actually
> I'm barely using it.
> - CPU fast enough (I guess)
> - Ubuntu 12.0.4 (32 bits, probably thats irrelevant here)
> - My users make requests using port 80 (actually not specifyng the port) to
> call a service I'm running on my server.
> - Nginx 1.4.3 receives the requests, and then derives them to Tomcat 7.0.33
> - Tomcat 7.0.33 is running on port 8080.
> 
> My website/service has been always running fine. I'm making stress tests in
> order to see if I can handle about 1000 queries per second, and I'm getting
> this error message in Nginx's log:
> 
> 
> 2014/02/12 09:59:42 [crit] 806#0: *595361 connect() to
> 127.0.0.1:8080failed (99: Cannot assign requested address) while
> connecting to upstream,
> client: 58.81.5.31, server: api.acme.com, request: "GET
> /iplocate/locate?key=UZ6FD8747F76VZ&ip=61.157.194.46 HTTP/1.1", upstream: "
> http://127.0.0.1:8080/iplocate/locate?key=UZ6FD8747F76VZ&ip=61.157.194.46",
> host: "services.acme.com"
> 
> My users are geting a "BAD GATEWAY" error status, which they are getting in
> their Java clients with an exception. My interpretation is that Nginx is
> suddenly unable to communicate with Tomcat, so it delivers an HTTP error
> status code "BAD GATEWAY".
> 
> It starts running fine, but after 1-2 minutes I start geting this error.
> Obviously I'm running out of some kind of resource (ports?). If I wait for
> a few minutes to let the system "rest", then it works again for a while (1
> minute maybe) and then it fails again.
> 
> If I run this command (which I suspect is relevant):
> 
>            sysctl net.ipv4.ip_local_port_range
> 
> I get this response, which I think its standard in Ubuntu (I haven't messed
> with it):
> 
>       root at ip-10-41-156-142:~# sysctl net.ipv4.ip_local_port_range
>       net.ipv4.ip_local_port_range = 32768    61000
> 
> I have read some postings about configuring ports in order to get rid of
> this error message, but I don't know if that is my problem.

You've run out of local ports due to sockets in TIME-WAIT state.  
There are more than one possible solution, including using lower 
MSL, using bigger local port range, using keepalive connections, 
using unix sockets to connect to backends and so on.

Simpliest solution would be to enable TIME-WAIT sockets reuse, 
with net.ipv4.tcp_tw_reuse (or net.ipv4.tcp_tw_recycle):

# sysctl net.ipv4.tcp_tw_reuse=1

-- 
Maxim Dounin
http://nginx.org/