TIME_WAITs and sustained connections was: Re: Memcached module -- unix domain socket support? (too man

[Tyler Kovacs]

> what is the operating system?
> what do you mean tweaked(details pls.)?

I am seeing the same behavior on Centos 4.5 with nginx 0.6.32 and
memcached 1.2.6.

After running load tests using apache bench, I see a high number
of TIME_WAIT connections from netstat.  The load tests hits nginx
which fetches static response from memcached.

$ uname -a
Linux geocode1.admin.zvents.com 2.6.9-55.ELsmp #1 SMP Wed May 2 14:04:42 EDT
2007 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release
CentOS release 4.5 (Final)

$ netstat -an |grep -c TIME_WAIT
6

$ ab -c 4 -n 20000 'http://localhost/geocode?address=94131'
<snip>

$ netstat -an |grep -c TIME_WAIT
34444

If I run the test again, I see a number of errors in error.log that look
like this:

2008/01/31 20:49:30 [crit] 24806#0: *70538 connect() to 127.0.0.1:11211
failed (99: Cannot assign requested address) while connecting to
upstream, client: 192.168.200.10, server: www.xxxxxxxxxx.com, request:
"GET /geocode?address=94131 HTTP/1.0", upstream: "memcached://127.0.0.1:11211",
host: "www.xxxxxxxxxx.com"

I believe this is a consequence of having too many sockets in TIME_WAIT.

Finally, I think the original poster was referring to the following kernel
parameter.

net.ipv4.ip_local_port_range = 1024  65000