Too many open files and unix

Axel ar at xlrs.de
Fri Oct 25 10:37:21 UTC 2013


Hello,

today I had too many open files and I need some advice to investigate in 
this issue.

I googled around and found a lot information how to solve this issue 
(e.g. http://forum.nginx.org/read.php?2,234191,234191)
I finally was able to get rid of the error.

But I wondered about nginx behaviour with unix sockets and open files on 
my machine, because every worker process on my machine opened 15892 
sockets which seems to be very much compared to other machines I took a 
look at. On other machines I only have as much open sockets as worker 
processes.

Perhaps someone can point me to the right direction to debug and solve 
this.

The system has 16 cores and 16 GB Ram (10 GB RAM used, Load average: 
0.3) Ulimits are set to

> su - www-data -s /bin/bash
> $ ulimit -Hn
> 32768
> $ ulimit -Sn
> 16384

In nginx.conf I set

> user www-data;
> worker_processes 12;
> worker_rlimit_nofile 65536; # raised from 16384
> worker_connections   65536; # raised from 16384

Checking the open files of the master process returns

> root at nginx:~ # lsof -p 31022 | wc -l
> 16066

> root at nginx:~ # lsof -p 31022 | grep socket | wc -l
> 15904

Checking the open sockets returns for every single worker process 15892
I restarted nginx several times, but this does not change anything.

This is what I get when I raise errorlog level to debug:

  2013/10/25 12:31:40 [debug] 45534#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:31:40 [debug] 45534#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:31:40 [debug] 45534#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:31:40 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: delete posted event 
00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign: 
00000000029B2B80:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113558 accept: 78.46.79.55 
fd:21
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign: 
000000000276D520:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113558 event timer add: 21: 
60000:1382697160518
2013/10/25 12:31:40 [debug] 45534#0: *257113558 reusable connection: 1
2013/10/25 12:31:40 [debug] 45534#0: *257113558 epoll add event: fd:21 
op:1 ev:80000001
2013/10/25 12:31:40 [debug] 45534#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113558 post event 
00007F7BB9FCEB70
2013/10/25 12:31:40 [debug] 45534#0: *257113558 delete posted event 
00007F7BB9FCEB70
2013/10/25 12:31:40 [debug] 45534#0: *257113558 http wait request 
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113558 malloc: 
0000000002BD1B50:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113558 recv: fd:21 39 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113558 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113558 posix_memalign: 
0000000002B2BE80:4096 @16
2013/10/25 12:31:40 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: delete posted event 
00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign: 
0000000002A78700:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113587 accept: 78.46.79.55 
fd:19
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign: 
0000000002716BF0:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113587 event timer add: 19: 
60000:1382697160776
2013/10/25 12:31:40 [debug] 45534#0: *257113587 reusable connection: 1
2013/10/25 12:31:40 [debug] 45534#0: *257113587 epoll add event: fd:19 
op:1 ev:80000001
2013/10/25 12:31:40 [debug] 45534#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113587 post event 
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113587 delete posted event 
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113587 http wait request 
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113587 malloc: 
0000000002B89FC0:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113587 recv: fd:19 39 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113587 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113587 posix_memalign: 
0000000002571760:4096 @16
2013/10/25 12:31:40 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: delete posted event 
00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign: 
00000000025F13C0:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113589 accept: 91.118.111.100 
fd:19
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign: 
0000000002658200:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113589 event timer add: 19: 
60000:1382697160795
2013/10/25 12:31:40 [debug] 45534#0: *257113589 reusable connection: 1
2013/10/25 12:31:40 [debug] 45534#0: *257113589 epoll add event: fd:19 
op:1 ev:80000001
2013/10/25 12:31:40 [debug] 45534#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113589 post event 
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 delete posted event 
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 http wait request 
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113589 malloc: 
0000000002927660:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 recv: fd:19 27 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 posix_memalign: 
0000000002571760:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event 
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign: 
0000000002BCE320:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 accept: 83.169.27.46 
fd:47
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign: 
0000000002B75DB0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 event timer add: 47: 
60000:1382697161017
2013/10/25 12:31:41 [debug] 45534#0: *257113643 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113643 epoll add event: fd:47 
op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113589 post event 
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 delete posted event 
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 http wait request 
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113589 malloc: 
0000000002927660:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 recv: fd:19 27 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 posix_memalign: 
0000000002571760:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event 
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign: 
0000000002BCE320:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 accept: 83.169.27.46 
fd:47
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign: 
0000000002B75DB0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 event timer add: 47: 
60000:1382697161017
2013/10/25 12:31:41 [debug] 45534#0: *257113643 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113643 epoll add event: fd:47 
op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45534#0: *257113643 post event 
00007F7BB9FD0028
2013/10/25 12:31:41 [debug] 45534#0: *257113643 delete posted event 
00007F7BB9FD0028
2013/10/25 12:31:41 [debug] 45534#0: *257113643 http wait request 
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113643 malloc: 
0000000002B39900:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113643 recv: fd:47 27 of 1024
2013/10/25 12:31:41 [debug] 45534#0: *257113643 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113643 posix_memalign: 
0000000003359E40:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event 
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign: 
0000000002A40C80:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113751 accept: 77.75.254.73 
fd:16014
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign: 
00000000029F4AE0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113751 event timer add: 16014: 
60000:1382697161443
2013/10/25 12:31:41 [debug] 45534#0: *257113751 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113751 epoll add event: 
fd:16014 op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45534#0: *257113751 post event 
00007F7BB9FD0C58
2013/10/25 12:31:41 [debug] 45534#0: *257113751 delete posted event 
00007F7BB9FD0C58
2013/10/25 12:31:41 [debug] 45534#0: *257113751 http wait request 
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113751 malloc: 
0000000002BB46F0:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113751 recv: fd:16014 27 of 
1024
2013/10/25 12:31:41 [debug] 45534#0: *257113751 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113751 posix_memalign: 
0000000002ADDDD0:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event 
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign: 
00000000028658D0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113758 accept: 78.46.79.55 
fd:15779
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign: 
00000000026528B0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113758 event timer add: 15779: 
60000:1382697161489
2013/10/25 12:31:41 [debug] 45534#0: *257113758 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113758 epoll add event: 
fd:15779 op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45534#0: *257113758 post event 
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113758 delete posted event 
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113758 http wait request 
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113758 malloc: 
0000000002BB46F0:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113758 recv: fd:15779 39 of 
1024
2013/10/25 12:31:41 [debug] 45534#0: *257113758 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113758 posix_memalign: 
0000000002ADDDD0:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event 
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign: 
000000000276C050:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113763 accept: 134.0.76.216 
fd:51
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign: 
00000000029EE590:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113763 event timer add: 51: 
60000:1382697161522
2013/10/25 12:31:41 [debug] 45534#0: *257113763 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113763 epoll add event: fd:51 
op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45535#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:31:41 [debug] 45534#0: *257113763 post event 
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113763 delete posted event 
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113763 http wait request 
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113763 malloc: 
00000000034A7F00:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113763 recv: fd:51 27 of 1024
2013/10/25 12:31:41 [debug] 45534#0: *257113763 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113763 posix_memalign: 
000000000322E2C0:4096 @16
...
...
...
2013/10/25 12:32:16 [debug] 45531#0: *257119454 epoll add event: 
fd:14481 op:1 ev:80000001
2013/10/25 12:32:16 [debug] 45531#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:32:16 [debug] 45531#0: *257119454 post event 
00007F7BB9FCFC18
2013/10/25 12:32:16 [debug] 45531#0: *257119454 delete posted event 
00007F7BB9FCFC18
2013/10/25 12:32:16 [debug] 45531#0: *257119454 http wait request 
handler
2013/10/25 12:32:16 [debug] 45531#0: *257119454 malloc: 
0000000002BB42E0:1024
2013/10/25 12:32:16 [debug] 45531#0: *257119454 recv: fd:14481 27 of 
1024
2013/10/25 12:32:16 [debug] 45535#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45531#0: *257119454 reusable connection: 0
2013/10/25 12:32:16 [debug] 45531#0: *257119454 posix_memalign: 
000000000286B6D0:4096 @16
2013/10/25 12:32:16 [debug] 45531#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45535#0: post event 00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45535#0: delete posted event 
00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45535#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:32:16 [debug] 45535#0: posix_memalign: 
0000000002AC72E0:256 @16
2013/10/25 12:32:16 [debug] 45535#0: *257119455 accept: 77.75.254.133 
fd:15745
2013/10/25 12:32:16 [debug] 45535#0: posix_memalign: 
000000000276A4D0:256 @16
2013/10/25 12:32:16 [debug] 45535#0: *257119455 event timer add: 15745: 
60000:1382697196597
2013/10/25 12:32:16 [debug] 45535#0: *257119455 reusable connection: 1
2013/10/25 12:32:16 [debug] 45535#0: *257119455 epoll add event: 
fd:15745 op:1 ev:80000001
2013/10/25 12:32:16 [debug] 45535#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:32:16 [debug] 45535#0: *257119455 post event 
00007F7BB9FD1000
2013/10/25 12:32:16 [debug] 45535#0: *257119455 delete posted event 
00007F7BB9FD1000
2013/10/25 12:32:16 [debug] 45535#0: *257119455 http wait request 
handler
2013/10/25 12:32:16 [debug] 45535#0: *257119455 malloc: 
00000000033F7A20:1024
2013/10/25 12:32:16 [debug] 45535#0: *257119455 recv: fd:15745 27 of 
1024
2013/10/25 12:32:16 [debug] 45535#0: *257119455 reusable connection: 0
2013/10/25 12:32:16 [debug] 45535#0: *257119455 posix_memalign: 
00000000026AF8C0:4096 @16
2013/10/25 12:32:16 [debug] 45534#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45535#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45534#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [info] 45879#0: Using 32768KiB of shared memory for 
push module in /etc/nginx/nginx.conf:100
2013/10/25 12:32:16 [debug] 45535#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45535#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45534#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45534#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: post event 00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45536#0: delete posted event 
00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45536#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:32:16 [debug] 45536#0: posix_memalign: 
000000000298FF90:256 @16
2013/10/25 12:32:16 [debug] 45536#0: *257119502 accept: 46.252.18.244 
fd:16232
2013/10/25 12:32:16 [debug] 45536#0: posix_memalign: 
0000000003CC5040:256 @16
2013/10/25 12:32:16 [debug] 45536#0: *257119502 event timer add: 16232: 
60000:1382697196773
2013/10/25 12:32:16 [debug] 45536#0: *257119502 reusable connection: 1
2013/10/25 12:32:16 [debug] 45536#0: *257119502 epoll add event: 
fd:16232 op:1 ev:80000001
2013/10/25 12:32:16 [debug] 45536#0: accept() not ready (11: Resource 
temporarily unavailable)
2013/10/25 12:32:16 [debug] 45536#0: *257119502 post event 
00007F7BB9FD3358
2013/10/25 12:32:16 [debug] 45536#0: *257119502 delete posted event 
00007F7BB9FD3358
2013/10/25 12:32:16 [debug] 45536#0: *257119502 http wait request 
handler
2013/10/25 12:32:16 [debug] 45536#0: *257119502 malloc: 
000000000263BD90:1024
2013/10/25 12:32:16 [debug] 45536#0: *257119502 recv: fd:16232 27 of 
1024
2013/10/25 12:32:16 [debug] 45536#0: *257119502 reusable connection: 0
2013/10/25 12:32:16 [debug] 45536#0: *257119502 posix_memalign: 
000000000323C850:4096 @16
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45532#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45532#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45535#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45535#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45534#0: epoll add event: fd:97 op:1 
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2 
ev:00000000
2013/10/25 12:32:16 [debug] 45534#0: epoll del event: fd:97 op:2 
ev:00000000


Thanks in advance,
Axel



More information about the nginx mailing list