<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Anoop,<br>
<br>
I don't see any troubles with your configuration.<br>
Also, if you have 120G of RAM and a single worker - the problem is
not in nginx.<br>
Do you have other software running on the host?<br>
<br>
Basically, you just run out of memory.<br>
<br>
You can optimize your reload though: use "service nginx reload"
(or "kill -SIGHUP") to reload nginx configuration.<br>
When you do nginx -s reload - you make nginx parse configuration
(and it requires memory) and then send a signal to the running
master. You can avoid this overhead with 'service' command as it
uses 'kill' documented in the manual page.<br>
<br>
On 06.08.2018 22:55, Anoop Alias wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAO6TEX0G-C5YKRQ4TQo_jpaNVr1S7GBpmsS1NPf87nFEKOjyuQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<div dir="ltr">Hi Igor,
<div><br>
</div>
<div>Config is reloaded using </div>
<div><br>
</div>
<div>/usr/sbin/nginx -s reload </div>
<div><br>
</div>
<div>this is invoked from a python/shell script ( Nginx is
installed on a web control panel )</div>
<div><br>
</div>
<div>The top-level Nginx config is in the gist below </div>
<div><br>
</div>
<div><a
href="https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a"
moz-do-not-send="true">https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a</a><br>
</div>
<div><br>
</div>
<div>It further includes ~8k server blocks or more in some
servers. Out of this 2/3 are server {} blocks with TLS config
and 1/3 non-TLS ones</div>
<div><br>
</div>
<div>
<div>]# pwd</div>
<div>/etc/nginx/sites-enabled</div>
</div>
<div>
<div># grep "server {" *|wc -l</div>
<div>7886</div>
</div>
<div><br>
</div>
<div>And yes most of them are very similar and mostly proxy to
upstream httpd</div>
<div><br>
</div>
<div>I have tried removing all the loadable modules and even
tried an older version of nginx and all produce the error </div>
<div><br>
</div>
<div><br>
</div>
<div>
<div># numastat -m</div>
<div><br>
</div>
<div>Per-node system memory usage (in MBs):</div>
<div> Node 0 Node 1
Total</div>
<div> --------------- ---------------
---------------</div>
<div>MemTotal 65430.84 65536.00
130966.84</div>
<div>MemFree 5491.26 40.89
5532.15</div>
<div>MemUsed 59939.58 65495.11
125434.69</div>
<div>Active 22295.61 21016.09
43311.70</div>
<div>Inactive 8742.76 4662.48
13405.24</div>
<div>Active(anon) 16717.10 16572.19
33289.29</div>
<div>Inactive(anon) 2931.94 1388.14
4320.08</div>
<div>Active(file) 5578.50 4443.91
10022.41</div>
<div>Inactive(file) 5810.82 3274.34
9085.16</div>
<div>Unevictable 0.00 0.00
0.00</div>
<div>Mlocked 0.00 0.00
0.00</div>
<div>Dirty 7.04 1.64
8.67</div>
<div>Writeback 0.00 0.00
0.00</div>
<div>FilePages 18458.93 10413.97
28872.90</div>
<div>Mapped 862.14 413.38
1275.52</div>
<div>AnonPages 12579.49 15264.37
27843.86</div>
<div>Shmem 7069.52 2695.71
9765.23</div>
<div>KernelStack 18.34 3.03
21.38</div>
<div>PageTables 153.14 107.77
260.90</div>
<div>NFS_Unstable 0.00 0.00
0.00</div>
<div>Bounce 0.00 0.00
0.00</div>
<div>WritebackTmp 0.00 0.00
0.00</div>
<div>Slab 4830.68 2254.55
7085.22</div>
<div>SReclaimable 2061.05 921.72
2982.77</div>
<div>SUnreclaim 2769.62 1332.83
4102.45</div>
<div>AnonHugePages 4.00 2.00
6.00</div>
<div>HugePages_Total 0.00 0.00
0.00</div>
<div>HugePages_Free 0.00 0.00
0.00</div>
<div>HugePages_Surp 0.00 0.00
0.00</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Mon, Aug 6, 2018 at 6:33 PM Igor A. Ippolitov
<<a href="mailto:iippolitov@nginx.com"
moz-do-not-send="true">iippolitov@nginx.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_367398162948788752moz-cite-prefix">Anoop,<br>
<br>
I suppose, most of your 10k servers are very similar,
right?<br>
Please, post top level configuration and a typical
server{}, please.<br>
<br>
Also, how do you reload configuration? With 'service nginx
reload' or may be other commands?<br>
<br>
It looks like you have a lot of fragmented memory and only
4gb free in the second numa node.<br>
So, I'd say this is OK that you are getting errors from
allocating a 16k stripes.<br>
<br>
Could you please post numastat -m output additionally.
Just to make sure you have half of the memory for the
second CPU.<br>
And we'll have a look if memory utilization may be
optimized based on your configuration.<br>
<br>
Regards,<br>
Igor.<br>
<br>
On 04.08.2018 07:54, Anoop Alias wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Igor,
<div><br>
</div>
<div>Setting vm.max_map_count to 20x the normal value
did not help</div>
<div><br>
</div>
<div>The issue happens on a group of servers and among
the group, it shows up only in servers which have
~10k server{} blocks</div>
<div><br>
</div>
<div>On servers that have lower number of server{}
blocks , the ENOMEM issue is not there</div>
<div><br>
</div>
<div>Also, I can find that the RAM usage of the Nginx
process is directly proportional to the number of
server {} blocks</div>
<div><br>
</div>
<div>For example on a server having the problem</div>
<div><br>
</div>
<div>
<div># ps_mem| head -1 && ps_mem |grep nginx</div>
<div> Private + Shared = RAM used Program </div>
<div> 1.0 GiB + 2.8 GiB = 3.8 GiB nginx (3)</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>That is for a single worker process with 4 threads
in thread_pool </div>
<div>
<div># pstree|grep nginx</div>
<div> |-nginx-+-nginx---4*[{nginx}]</div>
<div> | `-nginx</div>
</div>
<div><br>
</div>
<div>Whatever config change I try the memory usage seem
to mostly depend on the number of server contexts
defined</div>
<div><br>
</div>
<div>Now the issue mostly happen in nginx reload ,when
one more worker process will be active in shutting
down mode </div>
<div><br>
</div>
<div>I believe the memalign error is thrown by the
worker being shutdown, this is because the sites work
after the error and also the pid mentioned in the
error would have gone when I check ps</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div># pmap 948965|grep 16K</div>
<div>00007f2923ff2000 16K r-x--
ngx_http_redis2_module.so</div>
<div>00007f2924fd7000 16K r---- <a
href="http://libc-2.17.so" target="_blank"
moz-do-not-send="true">libc-2.17.so</a></div>
<div>00007f2925431000 16K rw--- [ anon ]</div>
<div>00007f292584a000 16K rw--- [ anon ]</div>
</div>
<div><br>
</div>
<div>
<div>Aug 4 05:50:00 b kernel: SysRq : Show Memory</div>
<div>Aug 4 05:50:00 b kernel: Mem-Info:</div>
<div>Aug 4 05:50:00 b kernel: active_anon:7757394
inactive_anon:1021319 isolated_anon:0#012 active_<a
class="m_367398162948788752moz-txt-link-freetext"
moz-do-not-send="true">file:3733324</a> inactive_<a
class="m_367398162948788752moz-txt-link-freetext"
moz-do-not-send="true">file:2136476</a> isolated_<a
class="m_367398162948788752moz-txt-link-freetext"
moz-do-not-send="true">file:0#012</a>
unevictable:0 dirty:1766 writeback:6 wbtmp:0
unstable:0#012 slab_reclaimable:2003687
slab_unreclaimable:901391#012 mapped:316734
shmem:2381810 pagetables:63163 bounce:0#012
free:4851283 free_pcp:11332 free_cma:0</div>
<div>Aug 4 05:50:00 bravo kernel: Node 0 DMA
free:15888kB min:8kB low:8kB high:12kB
active_anon:0kB inactive_anon:0kB active_<a
class="m_367398162948788752moz-txt-link-freetext"
moz-do-not-send="true">file:0kB</a> inactive_<a
class="m_367398162948788752moz-txt-link-freetext"
moz-do-not-send="true">file:0kB</a>
unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15972kB managed:15888kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB
shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:0kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes</div>
<div>Aug 4 05:50:00 b kernel: lowmem_reserve[]: 0
1679 64139 64139</div>
</div>
<div><br>
</div>
<div>
<div># cat /proc/buddyinfo</div>
<div>Node 0, zone DMA 0 0 1
0 2 1 1 0 1 1 3 </div>
<div>Node 0, zone DMA32 5284 6753 6677
1083 410 59 1 0 0 0
0 </div>
<div>Node 0, zone Normal 500327 638958 406737
14690 872 106 11 0 0 0
0 </div>
<div>Node 1, zone Normal 584840 291640 188
0 0 0 0 0 0 0 0 </div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>The only correlation I see in having the error is
the number of server {} blocks (close to 10k) which
then makes the nginx process consume ~ 4GB of mem with
a single worker process and then a reload is done</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Aug 2, 2018 at 6:02 PM Igor A.
Ippolitov <<a href="mailto:iippolitov@nginx.com"
target="_blank" moz-do-not-send="true">iippolitov@nginx.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div
class="m_367398162948788752m_-1786270149800573001moz-cite-prefix">Anoop,<br>
<br>
There are two guesses: either mmap allocations
limit is hit or memory is way too fragmented.<br>
Could you please track amount of mapped regions
for a worker with pmap and amount of 16k areas in
Normal zones (it is the third number)?<br>
<br>
You can also set vm.max_map_count to a higher
number (like 20 times higher than default) and
look if the error is gone.<br>
<br>
Please, let me know if increasing vm.max_map_count
helps you.<br>
<br>
On 02.08.2018 13:06, Anoop Alias wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Igor,
<div><br>
</div>
<div>The error happens randomly </div>
<div>
<div><br>
</div>
<div>
<div>2018/08/02 06:52:42 [emerg]
874514#874514: posix_memalign(16, 16384)
failed (12: Cannot allocate memory)</div>
<div>2018/08/02 09:42:53 [emerg]
872996#872996: posix_memalign(16, 16384)
failed (12: Cannot allocate memory)</div>
<div>2018/08/02 10:16:14 [emerg]
877611#877611: posix_memalign(16, 16384)
failed (12: Cannot allocate memory)</div>
<div>2018/08/02 10:16:48 [emerg]
879410#879410: posix_memalign(16, 16384)
failed (12: Cannot allocate memory)</div>
<div>2018/08/02 10:17:55 [emerg]
876563#876563: posix_memalign(16, 16384)
failed (12: Cannot allocate memory)</div>
<div>2018/08/02 10:20:21 [emerg]
879263#879263: posix_memalign(16, 16384)
failed (12: Cannot allocate memory)</div>
<div>2018/08/02 10:20:51 [emerg]
878991#878991: posix_memalign(16, 16384)
failed (12: Cannot allocate memory)</div>
</div>
<div><br>
</div>
<div>
<div># date</div>
<div>Thu Aug 2 10:58:48 BST 2018</div>
</div>
<div><br>
</div>
<div>------------------------------------------</div>
<div>
<div># cat /proc/buddyinfo</div>
<div>Node 0, zone DMA 0 0
1 0 2 1 1 0
1 1 3 </div>
<div>Node 0, zone DMA32 11722 11057
4663 1647 609 72 10 7
1 0 0 </div>
<div>Node 0, zone Normal 755026 710760
398136 21462 1114 18 1 0
0 0 0 </div>
<div>Node 1, zone Normal 341295 801810
179604 256 0 0 0 0
0 0 0 </div>
</div>
<div>-----------------------------------------</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>slabinfo - version: 2.1</div>
<div># name <active_objs>
<num_objs> <objsize>
<objperslab> <pagesperslab> :
tunables <limit> <batchcount>
<sharedfactor> : slabdata
<active_slabs> <num_slabs>
<sharedavail></div>
<div>SCTPv6 21 21 1536
21 8 : tunables 0 0 0 :
slabdata 1 1 0</div>
<div>SCTP 0 0 1408
23 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>kcopyd_job 0 0 3312
9 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dm_uevent 0 0 2608
12 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>nf_conntrack_ffffffff81acbb00 14054
14892 320 51 4 : tunables 0
0 0 : slabdata 292 292 0</div>
<div>lvp_cache 36 36 224
36 2 : tunables 0 0 0 :
slabdata 1 1 0</div>
<div>lve_struct 4140 4140 352
46 4 : tunables 0 0 0 :
slabdata 90 90 0</div>
<div>fat_inode_cache 0 0 744
44 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>fat_cache 0 0 40
102 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>isofs_inode_cache 0 0 664
49 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>ext4_inode_cache 30 30 1088
30 8 : tunables 0 0 0 :
slabdata 1 1 0</div>
<div>ext4_xattr 0 0 88
46 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>ext4_free_data 0 0 64
64 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>ext4_allocation_context 32 32
128 32 1 : tunables 0 0 0
: slabdata 1 1 0</div>
<div>ext4_io_end 0 0 72
56 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>ext4_extent_status 102 102
40 102 1 : tunables 0 0 0 :
slabdata 1 1 0</div>
<div>jbd2_journal_handle 0 0
48 85 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>jbd2_journal_head 0 0 112
36 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>jbd2_revoke_table_s 256 256
16 256 1 : tunables 0 0 0 :
slabdata 1 1 0</div>
<div>jbd2_revoke_record_s 0 0
32 128 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>kvm_async_pf 0 0 136
30 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>kvm_vcpu 0 0 18560
1 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>xfs_dqtrx 992 992 528
31 4 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>xfs_dquot 3264 3264 472
34 4 : tunables 0 0 0 :
slabdata 96 96 0</div>
<div>xfs_ili 4342175 4774399
152 53 2 : tunables 0 0 0 :
slabdata 90083 90083 0</div>
<div>xfs_inode 4915588 5486076
1088 30 8 : tunables 0 0 0
: slabdata 182871 182871 0</div>
<div>xfs_efd_item 2680 2760 400
40 4 : tunables 0 0 0 :
slabdata 69 69 0</div>
<div>xfs_da_state 1088 1088 480
34 4 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>xfs_btree_cur 1248 1248 208
39 2 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>xfs_log_ticket 14874 15048 184
44 2 : tunables 0 0 0 :
slabdata 342 342 0</div>
<div>xfs_ioend 12909 13104 104
39 1 : tunables 0 0 0 :
slabdata 336 336 0</div>
<div>scsi_cmd_cache 5400 5652 448
36 4 : tunables 0 0 0 :
slabdata 157 157 0</div>
<div>ve_struct 0 0 848
38 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>ip6_dst_cache 1152 1152 448
36 4 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>RAWv6 910 910 1216
26 8 : tunables 0 0 0 :
slabdata 35 35 0</div>
<div>UDPLITEv6 0 0 1216
26 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
</div>
<div>
<div>UDPv6 832 832 1216
26 8 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>tw_sock_TCPv6 1152 1376 256
32 2 : tunables 0 0 0 :
slabdata 43 43 0</div>
<div>TCPv6 510 510 2176
15 8 : tunables 0 0 0 :
slabdata 34 34 0</div>
<div>cfq_queue 3698 5145 232
35 2 : tunables 0 0 0 :
slabdata 147 147 0</div>
<div>bsg_cmd 0 0 312
52 4 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>mqueue_inode_cache 136 136
960 34 8 : tunables 0 0 0 :
slabdata 4 4 0</div>
<div>hugetlbfs_inode_cache 1632 1632
632 51 8 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>configfs_dir_cache 1472 1472
88 46 1 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>dquot 0 0 256
32 2 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>userfaultfd_ctx_cache 32 32
128 32 1 : tunables 0 0 0 :
slabdata 1 1 0</div>
<div>fanotify_event_info 2336 2336
56 73 1 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>dio 6171 6222 640
51 8 : tunables 0 0 0 :
slabdata 122 122 0</div>
<div>pid_namespace 42 42 2192
14 8 : tunables 0 0 0 :
slabdata 3 3 0</div>
<div>posix_timers_cache 1056 1056
248 33 2 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>UDP-Lite 0 0 1088
30 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>flow_cache 2268 2296 144
28 1 : tunables 0 0 0 :
slabdata 82 82 0</div>
<div>xfrm_dst_cache 896 896 576
28 4 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>ip_fib_alias 2720 2720 48
85 1 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>RAW 3977 4224 1024
32 8 : tunables 0 0 0 :
slabdata 132 132 0</div>
<div>UDP 4110 4110 1088
30 8 : tunables 0 0 0 :
slabdata 137 137 0</div>
<div>tw_sock_TCP 4756 5216 256
32 2 : tunables 0 0 0 :
slabdata 163 163 0</div>
<div>TCP 2705 2768 1984
16 8 : tunables 0 0 0 :
slabdata 173 173 0</div>
<div>scsi_data_buffer 5440 5440 24
170 1 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>blkdev_queue 154 154 2208
14 8 : tunables 0 0 0 :
slabdata 11 11 0</div>
<div>blkdev_requests 4397688 4405884
384 42 4 : tunables 0 0 0 :
slabdata 104902 104902 0</div>
<div>blkdev_ioc 11232 11232 112
36 1 : tunables 0 0 0 :
slabdata 312 312 0</div>
<div>user_namespace 0 0 1304
25 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>sock_inode_cache 12282 12282 704
46 8 : tunables 0 0 0 :
slabdata 267 267 0</div>
<div>file_lock_cache 20056 20960 200
40 2 : tunables 0 0 0 :
slabdata 524 524 0</div>
<div>net_namespace 6 6 5056
6 8 : tunables 0 0 0 :
slabdata 1 1 0</div>
<div>shmem_inode_cache 16970 18952 712
46 8 : tunables 0 0 0 :
slabdata 412 412 0</div>
<div>Acpi-ParseExt 39491 40432 72
56 1 : tunables 0 0 0 :
slabdata 722 722 0</div>
<div>Acpi-State 1683 1683 80
51 1 : tunables 0 0 0 :
slabdata 33 33 0</div>
<div>Acpi-Namespace 11424 11424 40
102 1 : tunables 0 0 0 :
slabdata 112 112 0</div>
<div>task_delay_info 15336 15336 112
36 1 : tunables 0 0 0 :
slabdata 426 426 0</div>
<div>taskstats 1568 1568 328
49 4 : tunables 0 0 0 :
slabdata 32 32 0</div>
<div>proc_inode_cache 169897 190608 680
48 8 : tunables 0 0 0 :
slabdata 3971 3971 0</div>
<div>sigqueue 2208 2208 168
48 2 : tunables 0 0 0 :
slabdata 46 46 0</div>
<div>bdev_cache 792 792 896
36 8 : tunables 0 0 0 :
slabdata 22 22 0</div>
</div>
<div>
<div>sysfs_dir_cache 74698 74698 120
34 1 : tunables 0 0 0 :
slabdata 2197 2197 0</div>
<div>mnt_cache 163197 163424 256
32 2 : tunables 0 0 0 :
slabdata 5107 5107 0</div>
<div>filp 64607 97257 320
51 4 : tunables 0 0 0 :
slabdata 1907 1907 0</div>
<div>inode_cache 370744 370947 616
53 8 : tunables 0 0 0 :
slabdata 6999 6999 0</div>
<div>dentry 1316262 2139228
192 42 2 : tunables 0 0 0 :
slabdata 50934 50934 0</div>
<div>iint_cache 0 0 80
51 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>buffer_head 1441470 2890290
104 39 1 : tunables 0 0 0 :
slabdata 74110 74110 0</div>
<div>vm_area_struct 194998 196840 216
37 2 : tunables 0 0 0 :
slabdata 5320 5320 0</div>
<div>mm_struct 2679 2760 1600
20 8 : tunables 0 0 0 :
slabdata 138 138 0</div>
<div>files_cache 8680 8925 640
51 8 : tunables 0 0 0 :
slabdata 175 175 0</div>
<div>signal_cache 3691 3780 1152
28 8 : tunables 0 0 0 :
slabdata 135 135 0</div>
<div>sighand_cache 1950 2160 2112
15 8 : tunables 0 0 0 :
slabdata 144 144 0</div>
<div>task_xstate 8070 8658 832
39 8 : tunables 0 0 0 :
slabdata 222 222 0</div>
<div>task_struct 1913 2088 4080
8 8 : tunables 0 0 0 :
slabdata 261 261 0</div>
<div>cred_jar 31699 33936 192
42 2 : tunables 0 0 0 :
slabdata 808 808 0</div>
<div>anon_vma_chain 164026 168704 64
64 1 : tunables 0 0 0 :
slabdata 2636 2636 0</div>
<div>anon_vma 84104 84594 88
46 1 : tunables 0 0 0 :
slabdata 1839 1839 0</div>
<div>pid 11127 12576 128
32 1 : tunables 0 0 0 :
slabdata 393 393 0</div>
<div>shared_policy_node 9350 9350
48 85 1 : tunables 0 0 0 :
slabdata 110 110 0</div>
<div>numa_policy 62 62 264
31 2 : tunables 0 0 0 :
slabdata 2 2 0</div>
<div>radix_tree_node 771778 1194312
584 28 4 : tunables 0 0 0 :
slabdata 42654 42654 0</div>
<div>idr_layer_cache 2538 2565 2112
15 8 : tunables 0 0 0 :
slabdata 171 171 0</div>
<div>dma-kmalloc-8192 0 0 8192
4 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-4096 0 0 4096
8 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-2048 0 0 2048
16 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-1024 0 0 1024
32 8 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-512 0 0 512
32 4 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-256 0 0 256
32 2 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-128 0 0 128
32 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-64 0 0 64
64 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-32 0 0 32
128 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-16 0 0 16
256 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-8 0 0 8
512 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-192 0 0 192
42 2 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>dma-kmalloc-96 0 0 96
42 1 : tunables 0 0 0 :
slabdata 0 0 0</div>
<div>kmalloc-8192 385 388 8192
4 8 : tunables 0 0 0 :
slabdata 97 97 0</div>
<div>kmalloc-4096 9296 10088 4096
8 8 : tunables 0 0 0 :
slabdata 1261 1261 0</div>
<div>kmalloc-2048 65061 133536 2048
16 8 : tunables 0 0 0 :
slabdata 8346 8346 0</div>
<div>kmalloc-1024 11987 21120 1024
32 8 : tunables 0 0 0 :
slabdata 660 660 0</div>
</div>
<div>
<div>kmalloc-512 107510 187072 512
32 4 : tunables 0 0 0 :
slabdata 5846 5846 0</div>
<div>kmalloc-256 160498 199104 256
32 2 : tunables 0 0 0 :
slabdata 6222 6222 0</div>
<div>kmalloc-192 144975 237426 192
42 2 : tunables 0 0 0 :
slabdata 5653 5653 0</div>
<div>kmalloc-128 36799 108096 128
32 1 : tunables 0 0 0 :
slabdata 3378 3378 0</div>
<div>kmalloc-96 99510 238896 96
42 1 : tunables 0 0 0 :
slabdata 5688 5688 0</div>
<div>kmalloc-64 7978152 8593280
64 64 1 : tunables 0 0 0 :
slabdata 134270 134270 0</div>
<div>kmalloc-32 2939882 3089664
32 128 1 : tunables 0 0 0 :
slabdata 24138 24138 0</div>
<div>kmalloc-16 172057 172288 16
256 1 : tunables 0 0 0 :
slabdata 673 673 0</div>
<div>kmalloc-8 109568 109568 8
512 1 : tunables 0 0 0 :
slabdata 214 214 0</div>
<div>kmem_cache_node 893 896 64
64 1 : tunables 0 0 0 :
slabdata 14 14 0</div>
<div>kmem_cache 612 612 320
51 4 : tunables 0 0 0 :
slabdata 12 12 0</div>
</div>
<div><br>
</div>
<div>-------------------------------------------------</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div># uname -r</div>
<div>3.10.0-714.10.2.lve1.5.17.1.el7.x86_64</div>
</div>
<div><br>
</div>
<div>--------------------------------------------------------</div>
<div><br>
</div>
<div>Core part of glances </div>
<div><a href="http://i.imgur.com/La5JbQn.png"
target="_blank" moz-do-not-send="true">http://i.imgur.com/La5JbQn.png</a><br>
</div>
<div>-----------------------------------------------------------</div>
<div><br>
</div>
<div>Thank you very much for looking into this</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Aug 2, 2018 at 12:37 PM
Igor A. Ippolitov <<a
href="mailto:iippolitov@nginx.com"
target="_blank" moz-do-not-send="true">iippolitov@nginx.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div
class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-cite-prefix">Anoop,<br>
<br>
I doubt this will be the solution, but may
we have a look at /proc/buddyinfo and
/proc/slabinfo the moment when nginx can't
allocate memory?<br>
<br>
On 02.08.2018 08:15, Anoop Alias wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Maxim,</div>
<div><br>
</div>
<div>I enabled debug and the memalign
call is happening on nginx reloads and
the ENOMEM happen sometimes on the
reload(not on all reloads)</div>
<div><br>
</div>
<div>
<div>2018/08/02 05:59:08 [notice]
872052#872052: signal process
started</div>
<div>2018/08/02 05:59:23 [notice]
871570#871570: signal 1 (SIGHUP)
received from 872052, reconfiguring</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: wake up, sigio 0</div>
<div>2018/08/02 05:59:23 [notice]
871570#871570: reconfiguring</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
0000000002B0DA00:16384 @16 ===
> the memalign call on reload</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: malloc:
00000000087924D0:4560</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
000000000E442E00:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: malloc:
0000000005650850:4096</div>
<div>20</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: bind() xxxx:443 #71</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: bind() xxxx:443 #72</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: bind() xxxx:443 #73</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: bind() xxxx:443 #74</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: add cleanup:
000000005340D728</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000024D3260:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000517BAF10:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
0000000053854FC0:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
0000000053855FD0:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
0000000053856FE0:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
0000000053857FF0:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: posix_memalign:
0000000053859000:16384 @16</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
000000005385D010:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
000000005385E020:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
000000005385F030:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536CD160:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536CE170:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536CF180:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536D0190:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536D11A0:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536D21B0:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536D31C0:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536D41D0:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536D51E0:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536D61F0:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536D7200:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536D8210:4096</div>
<div>2018/08/02 05:48:49 [debug]
871275#871275: malloc:
00000000536D9220:4096</div>
<div><br>
</div>
<div><br>
</div>
<div>Infact there are lot of such calls
during a reload</div>
<div><br>
</div>
<div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA17ED00:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA1B0FF0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA1E12C0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA211590:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA243880:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA271B30:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA2A3E20:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA2D20D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA3063E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA334690:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA366980:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA396C50:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA3C8F40:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA3F9210:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA4294E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA45B7D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA489A80:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA4BBD70:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA4EA020:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA51E330:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA54C5E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA57E8D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA5AEBA0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA5DEE70:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA611160:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA641430:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA671700:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA6A29E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA6D5CE0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA707FD0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA736280:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA768570:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA796820:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA7CAB30:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA7F8DE0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA82B0D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA85B3A0:16384 @16</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>What is perplexing is that the
system has enough free (available RAM)</div>
<div>#############</div>
<div>
<div># free -g</div>
<div> total used
free shared buff/cache
available</div>
<div>Mem: 125 54
24 8 46
58</div>
<div>Swap: 0 0
0</div>
</div>
<div>#############</div>
<div><br>
</div>
<div>
<div># ulimit -a</div>
<div>core file size (blocks,
-c) 0</div>
<div>data seg size (kbytes,
-d) unlimited</div>
<div>scheduling priority
(-e) 0</div>
<div>file size (blocks,
-f) unlimited</div>
<div>pending signals
(-i) 514579</div>
<div>max locked memory (kbytes,
-l) 64</div>
<div>max memory size (kbytes,
-m) unlimited</div>
<div>open files
(-n) 1024</div>
<div>pipe size (512 bytes,
-p) 8</div>
<div>POSIX message queues (bytes,
-q) 819200</div>
<div>real-time priority
(-r) 0</div>
<div>stack size (kbytes,
-s) 8192</div>
<div>cpu time (seconds,
-t) unlimited</div>
<div>max user processes
(-u) 514579</div>
<div>virtual memory (kbytes,
-v) unlimited</div>
<div>file locks
(-x) unlimited</div>
</div>
<div><br>
</div>
<div>#########################################</div>
<div><br>
</div>
<div>There is no other thing limiting
memory allocation </div>
<div><br>
</div>
<div>Any way to prevent this or probably
identify/prevent this</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Jul 31, 2018 at
7:08 PM Maxim Dounin <<a
href="mailto:mdounin@mdounin.ru"
target="_blank"
moz-do-not-send="true">mdounin@mdounin.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">Hello!<br>
<br>
On Tue, Jul 31, 2018 at 09:52:29AM
+0530, Anoop Alias wrote:<br>
<br>
> I am repeatedly seeing errors
like<br>
> <br>
> ######################<br>
> 2018/07/31 03:46:33 [emerg]
2854560#2854560: posix_memalign(16,
16384)<br>
> failed (12: Cannot allocate
memory)<br>
> 2018/07/31 03:54:09 [emerg]
2890190#2890190: posix_memalign(16,
16384)<br>
> failed (12: Cannot allocate
memory)<br>
> 2018/07/31 04:08:36 [emerg]
2939230#2939230: posix_memalign(16,
16384)<br>
> failed (12: Cannot allocate
memory)<br>
> 2018/07/31 04:24:48 [emerg]
2992650#2992650: posix_memalign(16,
16384)<br>
> failed (12: Cannot allocate
memory)<br>
> 2018/07/31 04:42:09 [emerg]
3053092#3053092: posix_memalign(16,
16384)<br>
> failed (12: Cannot allocate
memory)<br>
> 2018/07/31 04:42:17 [emerg]
3053335#3053335: posix_memalign(16,
16384)<br>
> failed (12: Cannot allocate
memory)<br>
> 2018/07/31 04:42:28 [emerg]
3053937#3053937: posix_memalign(16,
16384)<br>
> failed (12: Cannot allocate
memory)<br>
> 2018/07/31 04:47:54 [emerg]
3070638#3070638: posix_memalign(16,
16384)<br>
> failed (12: Cannot allocate
memory)<br>
> ####################<br>
> <br>
> on a few servers<br>
> <br>
> The servers have enough memory
free and the swap usage is 0, yet
somehow<br>
> the kernel denies the
posix_memalign with ENOMEM ( this is
what I think is<br>
> happening!)<br>
> <br>
> The numbers requested are always
16, 16k . This makes me suspicious<br>
> <br>
> I have no setting in nginx.conf
that reference a 16k<br>
> <br>
> Is there any chance of finding
out what requests this and why this is
not<br>
> fulfilled<br>
<br>
There are at least some buffers which
default to 16k - for <br>
example, ssl_buffer_size (<a
href="http://nginx.org/r/ssl_buffer_size"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://nginx.org/r/ssl_buffer_size</a>).<br>
<br>
You may try debugging log to futher
find out where the particular <br>
allocation happens, see here for
details:<br>
<br>
<a
href="http://nginx.org/en/docs/debugging_log.html"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://nginx.org/en/docs/debugging_log.html</a><br>
<br>
But I don't really think it worth the
effort. The error is pretty <br>
clear, and it's better to focus on why
these allocations are <br>
denied. Likely you are hitting some
limit.<br>
<br>
-- <br>
Maxim Dounin<br>
<a href="http://mdounin.ru/"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://mdounin.ru/</a><br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org"
target="_blank"
moz-do-not-send="true">nginx@nginx.org</a><br>
<a
href="http://mailman.nginx.org/mailman/listinfo/nginx"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div><b>Anoop P Alias</b>
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
nginx mailing list
<a class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank" moz-do-not-send="true">nginx@nginx.org</a>
<a class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank" moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
</blockquote>
<p><br>
</p>
</div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org"
target="_blank" moz-do-not-send="true">nginx@nginx.org</a><br>
<a
href="http://mailman.nginx.org/mailman/listinfo/nginx"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="m_367398162948788752m_-1786270149800573001gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div><b>Anoop P Alias</b>
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="m_367398162948788752m_-1786270149800573001mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
nginx mailing list
<a class="m_367398162948788752m_-1786270149800573001moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank" moz-do-not-send="true">nginx@nginx.org</a>
<a class="m_367398162948788752m_-1786270149800573001moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank" moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
</blockquote>
<p><br>
</p>
</div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank"
moz-do-not-send="true">nginx@nginx.org</a><br>
<a
href="http://mailman.nginx.org/mailman/listinfo/nginx"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="m_367398162948788752gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div><b>Anoop P Alias</b>
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset class="m_367398162948788752mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
nginx mailing list
<a class="m_367398162948788752moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank" moz-do-not-send="true">nginx@nginx.org</a>
<a class="m_367398162948788752moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank" moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
</blockquote>
<p><br>
</p>
</div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank"
moz-do-not-send="true">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div><b>Anoop P Alias</b>
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
nginx mailing list
<a class="moz-txt-link-abbreviated" href="mailto:nginx@nginx.org">nginx@nginx.org</a>
<a class="moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
</blockquote>
<p><br>
</p>
</body>
</html>