<div dir="ltr">Hi Igor,<div><br></div><div>Config is reloaded using </div><div><br></div><div>/usr/sbin/nginx -s reload </div><div><br></div><div>this is invoked from a python/shell script ( Nginx is installed on a web control panel )</div><div><br></div><div>The top-level Nginx config is in the gist below </div><div><br></div><div><a href="https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a">https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a</a><br></div><div><br></div><div>It further includes ~8k server blocks or more in some servers. Out of this 2/3 are server {} blocks with TLS config and 1/3 non-TLS ones</div><div><br></div><div><div>]# pwd</div><div>/etc/nginx/sites-enabled</div></div><div><div># grep "server {" *|wc -l</div><div>7886</div></div><div><br></div><div>And yes most of them are very similar and mostly proxy to upstream httpd</div><div><br></div><div>I have tried removing all the loadable modules and even tried an older version of nginx and all produce the error </div><div><br></div><div><br></div><div><div># numastat -m</div><div><br></div><div>Per-node system memory usage (in MBs):</div><div>             Node 0     Node 1      Total</div><div>         --------------- --------------- ---------------</div><div>MemTotal        65430.84    65536.00    130966.84</div><div>MemFree         5491.26      40.89     5532.15</div><div>MemUsed         59939.58    65495.11    125434.69</div><div>Active         22295.61    21016.09    43311.70</div><div>Inactive         8742.76     4662.48    13405.24</div><div>Active(anon)      16717.10    16572.19    33289.29</div><div>Inactive(anon)      2931.94     1388.14     4320.08</div><div>Active(file)       5578.50     4443.91    10022.41</div><div>Inactive(file)      5810.82     3274.34     9085.16</div><div>Unevictable         0.00      0.00      0.00</div><div>Mlocked           0.00      0.00      0.00</div><div>Dirty            7.04      1.64      8.67</div><div>Writeback          0.00      0.00      0.00</div><div>FilePages        18458.93    10413.97    28872.90</div><div>Mapped          862.14     413.38     1275.52</div><div>AnonPages        12579.49    15264.37    27843.86</div><div>Shmem          7069.52     2695.71     9765.23</div><div>KernelStack        18.34      3.03      21.38</div><div>PageTables        153.14     107.77     260.90</div><div>NFS_Unstable        0.00      0.00      0.00</div><div>Bounce           0.00      0.00      0.00</div><div>WritebackTmp        0.00      0.00      0.00</div><div>Slab           4830.68     2254.55     7085.22</div><div>SReclaimable       2061.05     921.72     2982.77</div><div>SUnreclaim        2769.62     1332.83     4102.45</div><div>AnonHugePages        4.00      2.00      6.00</div><div>HugePages_Total       0.00      0.00      0.00</div><div>HugePages_Free       0.00      0.00      0.00</div><div>HugePages_Surp       0.00      0.00      0.00</div></div><div><br></div><div><br></div><div>Thanks,</div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Aug 6, 2018 at 6:33 PM Igor A. Ippolitov <<a href="mailto:iippolitov@nginx.com">iippolitov@nginx.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_367398162948788752moz-cite-prefix">Anoop,<br>
<br>
I suppose, most of your 10k servers are very similar, right?<br>
Please, post top level configuration and a typical server{},
please.<br>
<br>
Also, how do you reload configuration? With 'service nginx reload'
or may be other commands?<br>
<br>
It looks like you have a lot of fragmented memory and only 4gb
free in the second numa node.<br>
So, I'd say this is OK that you are getting errors from allocating
a 16k stripes.<br>
<br>
Could you please post numastat -m output additionally. Just to
make sure you have half of the memory for the second CPU.<br>
And we'll have a look if memory utilization may be optimized based
on your configuration.<br>
<br>
Regards,<br>
Igor.<br>
<br>
On 04.08.2018 07:54, Anoop Alias wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Igor,
<div><br>
</div>
<div>Setting vm.max_map_count to 20x the normal value did not
help</div>
<div><br>
</div>
<div>The issue happens on a group of servers and among the
group, it shows up only in servers which have ~10k server{}
blocks</div>
<div><br>
</div>
<div>On servers that have lower number of server{} blocks , the
ENOMEM issue is not there</div>
<div><br>
</div>
<div>Also, I can find that the RAM usage of the Nginx process is
directly proportional to the number of server {} blocks</div>
<div><br>
</div>
<div>For example on a server having the problem</div>
<div><br>
</div>
<div>
<div># ps_mem| head -1 && ps_mem |grep nginx</div>
<div> Private +  Shared = RAM used    Program </div>
<div>Â 1.0 GiB +Â Â 2.8 GiB =Â Â 3.8 GiBÂ Â Â Â nginx (3)</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>That is for a single worker process with 4 threads in
thread_pool </div>
<div>
<div># pstree|grep nginx</div>
<div>Â Â Â Â |-nginx-+-nginx---4*[{nginx}]</div>
<div>Â Â Â Â |Â Â Â Â `-nginx</div>
</div>
<div><br>
</div>
<div>Whatever config change I try the memory usage seem to
mostly depend on the number of server contexts defined</div>
<div><br>
</div>
<div>Now the issue mostly happen in nginx reload ,when one more
worker process will be active in shutting down mode </div>
<div><br>
</div>
<div>I believe the memalign error is thrown by the worker being
shutdown, this is because the sites work after the error and
also the pid mentioned in the error would have gone when I
check ps</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div># pmap 948965|grep 16K</div>
<div>00007f2923ff2000Â Â Â 16K r-x-- ngx_http_redis2_module.so</div>
<div>00007f2924fd7000Â Â Â 16K r---- <a href="http://libc-2.17.so" target="_blank">libc-2.17.so</a></div>
<div>00007f2925431000Â Â Â 16K rw---Â Â [ anon ]</div>
<div>00007f292584a000Â Â Â 16K rw---Â Â [ anon ]</div>
</div>
<div><br>
</div>
<div>
<div>Aug 4 05:50:00 b kernel: SysRq : Show Memory</div>
<div>Aug 4 05:50:00 b kernel: Mem-Info:</div>
<div>Aug 4 05:50:00 b kernel: active_anon:7757394
inactive_anon:1021319 isolated_anon:0#012
active_<a class="m_367398162948788752moz-txt-link-freetext">file:3733324</a> inactive_<a class="m_367398162948788752moz-txt-link-freetext">file:2136476</a>
isolated_<a class="m_367398162948788752moz-txt-link-freetext">file:0#012</a> unevictable:0 dirty:1766 writeback:6
wbtmp:0 unstable:0#012 slab_reclaimable:2003687
slab_unreclaimable:901391#012 mapped:316734 shmem:2381810
pagetables:63163 bounce:0#012 free:4851283 free_pcp:11332
free_cma:0</div>
<div>Aug 4 05:50:00 bravo kernel: Node 0 DMA free:15888kB
min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB
active_<a class="m_367398162948788752moz-txt-link-freetext">file:0kB</a> inactive_<a class="m_367398162948788752moz-txt-link-freetext">file:0kB</a> unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:15972kB
managed:15888kB mlocked:0kB dirty:0kB writeback:0kB
mapped:0kB shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB
unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB
free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? yes</div>
<div>Aug 4 05:50:00 b kernel: lowmem_reserve[]: 0 1679 64139
64139</div>
</div>
<div><br>
</div>
<div>
<div># cat /proc/buddyinfo</div>
<div>Node 0, zone   DMA   0   0   1   0   2Â
  1   1   0   1   1   3 </div>
<div>Node 0, zone  DMA32  5284  6753  6677  1083  410Â
  59   1   0   0   0   0 </div>
<div>Node 0, zone  Normal 500327 638958 406737 14690  872Â
 106   11   0   0   0   0 </div>
<div>Node 1, zone  Normal 584840 291640  188   0   0Â
  0   0   0   0   0   0 </div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>The only correlation I see in having the error is the
number of server {} blocks (close to 10k) which then makes the
nginx process consume ~ 4GB of mem with a single worker
process and then a reload is done</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Aug 2, 2018 at 6:02 PM Igor A. Ippolitov
<<a href="mailto:iippolitov@nginx.com" target="_blank">iippolitov@nginx.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_367398162948788752m_-1786270149800573001moz-cite-prefix">Anoop,<br>
<br>
There are two guesses: either mmap allocations limit is
hit or memory is way too fragmented.<br>
Could you please track amount of mapped regions for a
worker with pmap and amount of 16k areas in Normal zones
(it is the third number)?<br>
<br>
You can also set vm.max_map_count to a higher number (like
20 times higher than default) and look if the error is
gone.<br>
<br>
Please, let me know if increasing vm.max_map_count helps
you.<br>
<br>
On 02.08.2018 13:06, Anoop Alias wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Igor,
<div><br>
</div>
<div>The error happens randomly </div>
<div>
<div><br>
</div>
<div>
<div>2018/08/02 06:52:42 [emerg] 874514#874514:
posix_memalign(16, 16384) failed (12: Cannot
allocate memory)</div>
<div>2018/08/02 09:42:53 [emerg] 872996#872996:
posix_memalign(16, 16384) failed (12: Cannot
allocate memory)</div>
<div>2018/08/02 10:16:14 [emerg] 877611#877611:
posix_memalign(16, 16384) failed (12: Cannot
allocate memory)</div>
<div>2018/08/02 10:16:48 [emerg] 879410#879410:
posix_memalign(16, 16384) failed (12: Cannot
allocate memory)</div>
<div>2018/08/02 10:17:55 [emerg] 876563#876563:
posix_memalign(16, 16384) failed (12: Cannot
allocate memory)</div>
<div>2018/08/02 10:20:21 [emerg] 879263#879263:
posix_memalign(16, 16384) failed (12: Cannot
allocate memory)</div>
<div>2018/08/02 10:20:51 [emerg] 878991#878991:
posix_memalign(16, 16384) failed (12: Cannot
allocate memory)</div>
</div>
<div><br>
</div>
<div>
<div># date</div>
<div>Thu Aug 2 10:58:48 BST 2018</div>
</div>
<div><br>
</div>
<div>------------------------------------------</div>
<div>
<div># cat /proc/buddyinfo</div>
<div>Node 0, zone   DMA   0   0   1  Â
0Â Â Â 2Â Â Â 1Â Â Â 1Â Â Â 0Â Â Â 1Â Â Â 1Â Â Â
3Â </div>
<div>Node 0, zone  DMA32 11722 11057  4663Â
 1647  609   72   10   7   1   0 Â
 0 </div>
<div>Node 0, zone  Normal 755026 710760 398136Â
21462Â Â 1114Â Â Â 18Â Â Â 1Â Â Â 0Â Â Â 0Â Â Â 0Â Â
 0 </div>
<div>Node 1, zone  Normal 341295 801810 179604 Â
256Â Â Â 0Â Â Â 0Â Â Â 0Â Â Â 0Â Â Â 0Â Â Â 0Â Â Â
0Â </div>
</div>
<div>-----------------------------------------</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>slabinfo - version: 2.1</div>
<div># name      <active_objs>
<num_objs> <objsize>
<objperslab> <pagesperslab> : tunables
<limit> <batchcount>
<sharedfactor> : slabdata
<active_slabs> <num_slabs>
<sharedavail></div>
<div>SCTPv6Â Â Â Â Â Â Â Â 21Â Â Â 21Â Â 1536Â Â 21Â Â
8 : tunables  0  0  0 : slabdata   1  Â
1Â Â Â 0</div>
<div>SCTPÂ Â Â Â Â Â Â Â Â Â 0Â Â Â 0Â Â 1408Â Â 23Â Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>kcopyd_job       0   0  3312  9 Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dm_uevent       0   0  2608  12 Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>nf_conntrack_ffffffff81acbb00Â 14054Â 14892Â Â
320  51  4 : tunables  0  0  0 :
slabdata  292  292   0</div>
<div>lvp_cache       36   36  224  36 Â
2 : tunables  0  0  0 : slabdata   1  Â
1Â Â Â 0</div>
<div>lve_struct     4140  4140  352  46 Â
4 : tunables  0  0  0 : slabdata   90 Â
 90   0</div>
<div>fat_inode_cache    0   0  744  44 Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>fat_cache       0   0   40 102 Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>isofs_inode_cache   0   0  664  49 Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>ext4_inode_cache   30   30  1088  30 Â
8 : tunables  0  0  0 : slabdata   1  Â
1Â Â Â 0</div>
<div>ext4_xattr       0   0   88  46 Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>ext4_free_data     0   0   64  64 Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>ext4_allocation_context   32   32  128Â
 32  1 : tunables  0  0  0 : slabdata  Â
1Â Â Â 1Â Â Â 0</div>
<div>ext4_io_end      0   0   72  56 Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>ext4_extent_status  102  102   40 102 Â
1 : tunables  0  0  0 : slabdata   1  Â
1Â Â Â 0</div>
<div>jbd2_journal_handle   0   0   48  85Â
 1 : tunables  0  0  0 : slabdata   0 Â
 0   0</div>
<div>jbd2_journal_head   0   0  112  36 Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>jbd2_revoke_table_s  256  256   16 256Â
 1 : tunables  0  0  0 : slabdata   1 Â
 1   0</div>
<div>jbd2_revoke_record_s   0   0   32 128Â
 1 : tunables  0  0  0 : slabdata   0 Â
 0   0</div>
<div>kvm_async_pf      0   0  136  30 Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>kvm_vcpu        0   0 18560  1 Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>xfs_dqtrx      992  992  528  31 Â
4 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>xfs_dquot      3264  3264  472  34 Â
4 : tunables  0  0  0 : slabdata   96 Â
 96   0</div>
<div>xfs_ili      4342175 4774399  152  53Â
 2 : tunables  0  0  0 : slabdata 90083Â
90083Â Â Â 0</div>
<div>xfs_inode     4915588 5486076  1088  30Â
 8 : tunables  0  0  0 : slabdata 182871
182871Â Â Â 0</div>
<div>xfs_efd_item    2680  2760  400  40 Â
4 : tunables  0  0  0 : slabdata   69 Â
 69   0</div>
<div>xfs_da_state    1088  1088  480  34 Â
4 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>xfs_btree_cur    1248  1248  208  39 Â
2 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>xfs_log_ticket   14874 15048  184  44 Â
2 : tunables  0  0  0 : slabdata  342 Â
342Â Â Â 0</div>
<div>xfs_ioend     12909 13104  104  39 Â
1 : tunables  0  0  0 : slabdata  336 Â
336Â Â Â 0</div>
<div>scsi_cmd_cache   5400  5652  448  36 Â
4 : tunables  0  0  0 : slabdata  157 Â
157Â Â Â 0</div>
<div>ve_struct       0   0  848  38 Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>ip6_dst_cache    1152  1152  448  36 Â
4 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>RAWv6Â Â Â Â Â Â Â Â 910Â Â 910Â Â 1216Â Â 26Â Â
8 : tunables  0  0  0 : slabdata   35 Â
 35   0</div>
<div>UDPLITEv6Â Â Â Â Â Â Â 0Â Â Â 0Â Â 1216Â Â 26Â Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
</div>
<div>
<div>UDPv6Â Â Â Â Â Â Â Â 832Â Â 832Â Â 1216Â Â 26Â Â
8 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>tw_sock_TCPv6Â Â Â Â 1152Â Â 1376Â Â 256Â Â 32Â Â
2 : tunables  0  0  0 : slabdata   43 Â
 43   0</div>
<div>TCPv6Â Â Â Â Â Â Â Â 510Â Â 510Â Â 2176Â Â 15Â Â
8 : tunables  0  0  0 : slabdata   34 Â
 34   0</div>
<div>cfq_queue      3698  5145  232  35 Â
2 : tunables  0  0  0 : slabdata  147 Â
147Â Â Â 0</div>
<div>bsg_cmd        0   0  312  52 Â
4 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>mqueue_inode_cache  136  136  960  34 Â
8 : tunables  0  0  0 : slabdata   4  Â
4Â Â Â 0</div>
<div>hugetlbfs_inode_cache  1632  1632  632Â
 51  8 : tunables  0  0  0 : slabdata Â
 32   32   0</div>
<div>configfs_dir_cache  1472  1472   88  46 Â
1 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>dquot         0   0  256  32 Â
2 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>userfaultfd_ctx_cache   32   32  128Â
 32  1 : tunables  0  0  0 : slabdata  Â
1Â Â Â 1Â Â Â 0</div>
<div>fanotify_event_info  2336  2336   56  73Â
 1 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>dio         6171  6222  640  51 Â
8 : tunables  0  0  0 : slabdata  122 Â
122Â Â Â 0</div>
<div>pid_namespace     42   42  2192  14 Â
8 : tunables  0  0  0 : slabdata   3  Â
3Â Â Â 0</div>
<div>posix_timers_cache  1056  1056  248  33 Â
2 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>UDP-Lite        0   0  1088  30 Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>flow_cache     2268  2296  144  28 Â
1 : tunables  0  0  0 : slabdata   82 Â
 82   0</div>
<div>xfrm_dst_cache    896  896  576  28 Â
4 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>ip_fib_alias    2720  2720   48  85 Â
1 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>RAWÂ Â Â Â Â Â Â Â Â 3977Â Â 4224Â Â 1024Â Â 32Â Â
8 : tunables  0  0  0 : slabdata  132 Â
132Â Â Â 0</div>
<div>UDPÂ Â Â Â Â Â Â Â Â 4110Â Â 4110Â Â 1088Â Â 30Â Â
8 : tunables  0  0  0 : slabdata  137 Â
137Â Â Â 0</div>
<div>tw_sock_TCPÂ Â Â Â Â 4756Â Â 5216Â Â 256Â Â 32Â Â
2 : tunables  0  0  0 : slabdata  163 Â
163Â Â Â 0</div>
<div>TCPÂ Â Â Â Â Â Â Â Â 2705Â Â 2768Â Â 1984Â Â 16Â Â
8 : tunables  0  0  0 : slabdata  173 Â
173Â Â Â 0</div>
<div>scsi_data_buffer  5440  5440   24 170 Â
1 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>blkdev_queue     154  154  2208  14 Â
8 : tunables  0  0  0 : slabdata   11 Â
 11   0</div>
<div>blkdev_requests  4397688 4405884  384  42Â
 4 : tunables  0  0  0 : slabdata 104902
104902Â Â Â 0</div>
<div>blkdev_ioc     11232 11232  112  36 Â
1 : tunables  0  0  0 : slabdata  312 Â
312Â Â Â 0</div>
<div>user_namespace     0   0  1304  25 Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>sock_inode_cache  12282 12282  704  46 Â
8 : tunables  0  0  0 : slabdata  267 Â
267Â Â Â 0</div>
<div>file_lock_cache  20056 20960  200  40 Â
2 : tunables  0  0  0 : slabdata  524 Â
524Â Â Â 0</div>
<div>net_namespace     6   6  5056  6 Â
8 : tunables  0  0  0 : slabdata   1  Â
1Â Â Â 0</div>
<div>shmem_inode_cache 16970 18952  712  46 Â
8 : tunables  0  0  0 : slabdata  412 Â
412Â Â Â 0</div>
<div>Acpi-ParseExt   39491 40432   72  56 Â
1 : tunables  0  0  0 : slabdata  722 Â
722Â Â Â 0</div>
<div>Acpi-State     1683  1683   80  51 Â
1 : tunables  0  0  0 : slabdata   33 Â
 33   0</div>
<div>Acpi-Namespace   11424 11424   40 102 Â
1 : tunables  0  0  0 : slabdata  112 Â
112Â Â Â 0</div>
<div>task_delay_info  15336 15336  112  36 Â
1 : tunables  0  0  0 : slabdata  426 Â
426Â Â Â 0</div>
<div>taskstats      1568  1568  328  49 Â
4 : tunables  0  0  0 : slabdata   32 Â
 32   0</div>
<div>proc_inode_cache 169897 190608  680  48 Â
8 : tunables  0  0  0 : slabdata  3971Â
 3971   0</div>
<div>sigqueue      2208  2208  168  48 Â
2 : tunables  0  0  0 : slabdata   46 Â
 46   0</div>
<div>bdev_cache      792  792  896  36 Â
8 : tunables  0  0  0 : slabdata   22 Â
 22   0</div>
</div>
<div>
<div>sysfs_dir_cache  74698 74698  120  34 Â
1 : tunables  0  0  0 : slabdata  2197Â
 2197   0</div>
<div>mnt_cache     163197 163424  256  32 Â
2 : tunables  0  0  0 : slabdata  5107Â
 5107   0</div>
<div>filp        64607 97257  320  51 Â
4 : tunables  0  0  0 : slabdata  1907Â
 1907   0</div>
<div>inode_cache    370744 370947  616  53 Â
8 : tunables  0  0  0 : slabdata  6999Â
 6999   0</div>
<div>dentry      1316262 2139228  192  42Â
 2 : tunables  0  0  0 : slabdata 50934Â
50934Â Â Â 0</div>
<div>iint_cache       0   0   80  51 Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>buffer_head    1441470 2890290  104  39Â
 1 : tunables  0  0  0 : slabdata 74110Â
74110Â Â Â 0</div>
<div>vm_area_struct  194998 196840  216  37 Â
2 : tunables  0  0  0 : slabdata  5320Â
 5320   0</div>
<div>mm_struct      2679  2760  1600  20 Â
8 : tunables  0  0  0 : slabdata  138 Â
138Â Â Â 0</div>
<div>files_cache     8680  8925  640  51 Â
8 : tunables  0  0  0 : slabdata  175 Â
175Â Â Â 0</div>
<div>signal_cache    3691  3780  1152  28 Â
8 : tunables  0  0  0 : slabdata  135 Â
135Â Â Â 0</div>
<div>sighand_cache    1950  2160  2112  15 Â
8 : tunables  0  0  0 : slabdata  144 Â
144Â Â Â 0</div>
<div>task_xstate     8070  8658  832  39 Â
8 : tunables  0  0  0 : slabdata  222 Â
222Â Â Â 0</div>
<div>task_struct     1913  2088  4080  8 Â
8 : tunables  0  0  0 : slabdata  261 Â
261Â Â Â 0</div>
<div>cred_jar      31699 33936  192  42 Â
2 : tunables  0  0  0 : slabdata  808 Â
808Â Â Â 0</div>
<div>anon_vma_chain  164026 168704   64  64 Â
1 : tunables  0  0  0 : slabdata  2636Â
 2636   0</div>
<div>anon_vma      84104 84594   88  46 Â
1 : tunables  0  0  0 : slabdata  1839Â
 1839   0</div>
<div>pid        11127 12576  128  32 Â
1 : tunables  0  0  0 : slabdata  393 Â
393Â Â Â 0</div>
<div>shared_policy_node  9350  9350   48  85 Â
1 : tunables  0  0  0 : slabdata  110 Â
110Â Â Â 0</div>
<div>numa_policy      62   62  264  31 Â
2 : tunables  0  0  0 : slabdata   2  Â
2Â Â Â 0</div>
<div>radix_tree_node  771778 1194312  584  28 Â
4 : tunables  0  0  0 : slabdata 42654Â
42654Â Â Â 0</div>
<div>idr_layer_cache   2538  2565  2112  15 Â
8 : tunables  0  0  0 : slabdata  171 Â
171Â Â Â 0</div>
<div>dma-kmalloc-8192Â Â Â Â 0Â Â Â 0Â Â 8192Â Â 4Â Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-4096Â Â Â Â 0Â Â Â 0Â Â 4096Â Â 8Â Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-2048Â Â Â Â 0Â Â Â 0Â Â 2048Â Â 16Â Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-1024Â Â Â Â 0Â Â Â 0Â Â 1024Â Â 32Â Â
8 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-512Â Â Â Â 0Â Â Â 0Â Â 512Â Â 32Â Â
4 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-256Â Â Â Â 0Â Â Â 0Â Â 256Â Â 32Â Â
2 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-128Â Â Â Â 0Â Â Â 0Â Â 128Â Â 32Â Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-64Â Â Â Â Â 0Â Â Â 0Â Â Â 64Â Â 64Â Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-32Â Â Â Â Â 0Â Â Â 0Â Â Â 32Â 128Â Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-16Â Â Â Â Â 0Â Â Â 0Â Â Â 16Â 256Â Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-8Â Â Â Â Â 0Â Â Â 0Â Â Â 8Â 512Â Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-192Â Â Â Â 0Â Â Â 0Â Â 192Â Â 42Â Â
2 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>dma-kmalloc-96Â Â Â Â Â 0Â Â Â 0Â Â Â 96Â Â 42Â Â
1 : tunables  0  0  0 : slabdata   0  Â
0Â Â Â 0</div>
<div>kmalloc-8192Â Â Â Â Â 385Â Â 388Â Â 8192Â Â 4Â Â
8 : tunables  0  0  0 : slabdata   97 Â
 97   0</div>
<div>kmalloc-4096Â Â Â Â 9296Â 10088Â Â 4096Â Â 8Â Â
8 : tunables  0  0  0 : slabdata  1261Â
 1261   0</div>
<div>kmalloc-2048Â Â Â Â 65061 133536Â Â 2048Â Â 16Â Â
8 : tunables  0  0  0 : slabdata  8346Â
 8346   0</div>
<div>kmalloc-1024Â Â Â Â 11987Â 21120Â Â 1024Â Â 32Â Â
8 : tunables  0  0  0 : slabdata  660 Â
660Â Â Â 0</div>
</div>
<div>
<div>kmalloc-512Â Â Â Â 107510 187072Â Â 512Â Â 32Â Â
4 : tunables  0  0  0 : slabdata  5846Â
 5846   0</div>
<div>kmalloc-256Â Â Â Â 160498 199104Â Â 256Â Â 32Â Â
2 : tunables  0  0  0 : slabdata  6222Â
 6222   0</div>
<div>kmalloc-192Â Â Â Â 144975 237426Â Â 192Â Â 42Â Â
2 : tunables  0  0  0 : slabdata  5653Â
 5653   0</div>
<div>kmalloc-128Â Â Â Â 36799 108096Â Â 128Â Â 32Â Â
1 : tunables  0  0  0 : slabdata  3378Â
 3378   0</div>
<div>kmalloc-96Â Â Â Â Â 99510 238896Â Â Â 96Â Â 42Â Â
1 : tunables  0  0  0 : slabdata  5688Â
 5688   0</div>
<div>kmalloc-64Â Â Â Â 7978152 8593280Â Â Â 64Â Â 64Â
 1 : tunables  0  0  0 : slabdata 134270
134270Â Â Â 0</div>
<div>kmalloc-32Â Â Â Â 2939882 3089664Â Â Â 32Â 128Â
 1 : tunables  0  0  0 : slabdata 24138Â
24138Â Â Â 0</div>
<div>kmalloc-16Â Â Â Â 172057 172288Â Â Â 16Â 256Â Â
1 : tunables  0  0  0 : slabdata  673 Â
673Â Â Â 0</div>
<div>kmalloc-8Â Â Â Â Â 109568 109568Â Â Â 8Â 512Â Â
1 : tunables  0  0  0 : slabdata  214 Â
214Â Â Â 0</div>
<div>kmem_cache_node   893  896   64  64 Â
1 : tunables  0  0  0 : slabdata   14 Â
 14   0</div>
<div>kmem_cache      612  612  320  51 Â
4 : tunables  0  0  0 : slabdata   12 Â
 12   0</div>
</div>
<div><br>
</div>
<div>-------------------------------------------------</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div># uname -r</div>
<div>3.10.0-714.10.2.lve1.5.17.1.el7.x86_64</div>
</div>
<div><br>
</div>
<div>--------------------------------------------------------</div>
<div><br>
</div>
<div>Core part of glances </div>
<div><a href="http://i.imgur.com/La5JbQn.png" target="_blank">http://i.imgur.com/La5JbQn.png</a><br>
</div>
<div>-----------------------------------------------------------</div>
<div><br>
</div>
<div>Thank you very much for looking into this</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Aug 2, 2018 at 12:37 PM Igor A.
Ippolitov <<a href="mailto:iippolitov@nginx.com" target="_blank">iippolitov@nginx.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-cite-prefix">Anoop,<br>
<br>
I doubt this will be the solution, but may we have
a look at /proc/buddyinfo and /proc/slabinfo the
moment when nginx can't allocate memory?<br>
<br>
On 02.08.2018 08:15, Anoop Alias wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Maxim,</div>
<div><br>
</div>
<div>I enabled debug and the memalign call is
happening on nginx reloads and the ENOMEM
happen sometimes on the reload(not on all
reloads)</div>
<div><br>
</div>
<div>
<div>2018/08/02 05:59:08 [notice]
872052#872052: signal process started</div>
<div>2018/08/02 05:59:23 [notice]
871570#871570: signal 1 (SIGHUP) received
from 872052, reconfiguring</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: wake up, sigio 0</div>
<div>2018/08/02 05:59:23 [notice]
871570#871570: reconfiguring</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
0000000002B0DA00:16384 @16Â Â Â === > the
memalign call on reload</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: malloc: 00000000087924D0:4560</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
000000000E442E00:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: malloc: 0000000005650850:4096</div>
<div>20</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
bind() xxxx:443 #71</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
bind() xxxx:443 #72</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
bind() xxxx:443 #73</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
bind() xxxx:443 #74</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
add cleanup: 000000005340D728</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000024D3260:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000517BAF10:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 0000000053854FC0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 0000000053855FD0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 0000000053856FE0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 0000000053857FF0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
posix_memalign: 0000000053859000:16384 @16</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 000000005385D010:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 000000005385E020:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 000000005385F030:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536CD160:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536CE170:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536CF180:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536D0190:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536D11A0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536D21B0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536D31C0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536D41D0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536D51E0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536D61F0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536D7200:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536D8210:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
malloc: 00000000536D9220:4096</div>
<div><br>
</div>
<div><br>
</div>
<div>Infact there are lot of such calls during a
reload</div>
<div><br>
</div>
<div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA17ED00:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA1B0FF0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA1E12C0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA211590:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA243880:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA271B30:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA2A3E20:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA2D20D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA3063E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA334690:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA366980:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA396C50:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA3C8F40:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA3F9210:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA4294E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA45B7D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA489A80:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA4BBD70:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA4EA020:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA51E330:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA54C5E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA57E8D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA5AEBA0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA5DEE70:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA611160:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA641430:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA671700:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA6A29E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA6D5CE0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA707FD0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA736280:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA768570:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA796820:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA7CAB30:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA7F8DE0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA82B0D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug]
871570#871570: posix_memalign:
00000000BA85B3A0:16384 @16</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>What is perplexing is that the system has
enough free (available RAM)</div>
<div>#############</div>
<div>
<div># free -g</div>
<div>       total    used   Â
free   shared buff/cache  available</div>
<div>Mem:Â Â Â Â Â Â 125Â Â Â Â Â 54Â Â Â Â Â
24Â Â Â Â Â Â 8Â Â Â Â Â 46Â Â Â Â Â 58</div>
<div>Swap:Â Â Â Â Â Â Â 0Â Â Â Â Â Â 0Â Â Â Â Â
 0</div>
</div>
<div>#############</div>
<div><br>
</div>
<div>
<div># ulimit -a</div>
<div>core file size     (blocks, -c) 0</div>
<div>data seg size      (kbytes, -d)
unlimited</div>
<div>scheduling priority       (-e) 0</div>
<div>file size        (blocks, -f)
unlimited</div>
<div>pending signals         (-i)
514579</div>
<div>max locked memory    (kbytes, -l) 64</div>
<div>max memory size     (kbytes, -m)
unlimited</div>
<div>open files           (-n) 1024</div>
<div>pipe size      (512 bytes, -p) 8</div>
<div>POSIX message queues   (bytes, -q)
819200</div>
<div>real-time priority       (-r) 0</div>
<div>stack size       (kbytes, -s) 8192</div>
<div>cpu time        (seconds, -t)
unlimited</div>
<div>max user processes       (-u)
514579</div>
<div>virtual memory     (kbytes, -v)
unlimited</div>
<div>file locks           (-x)
unlimited</div>
</div>
<div><br>
</div>
<div>#########################################</div>
<div><br>
</div>
<div>There is no other thing limiting memory
allocation </div>
<div><br>
</div>
<div>Any way to prevent this or probably
identify/prevent this</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Jul 31, 2018 at 7:08 PM
Maxim Dounin <<a href="mailto:mdounin@mdounin.ru" target="_blank">mdounin@mdounin.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello!<br>
<br>
On Tue, Jul 31, 2018 at 09:52:29AM +0530,
Anoop Alias wrote:<br>
<br>
> I am repeatedly seeing errors like<br>
> <br>
> ######################<br>
> 2018/07/31 03:46:33 [emerg]
2854560#2854560: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 03:54:09 [emerg]
2890190#2890190: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:08:36 [emerg]
2939230#2939230: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:24:48 [emerg]
2992650#2992650: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:09 [emerg]
3053092#3053092: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:17 [emerg]
3053335#3053335: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:28 [emerg]
3053937#3053937: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:47:54 [emerg]
3070638#3070638: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> ####################<br>
> <br>
> on a few servers<br>
> <br>
> The servers have enough memory free and
the swap usage is 0, yet somehow<br>
> the kernel denies the posix_memalign with
ENOMEM ( this is what I think is<br>
> happening!)<br>
> <br>
> The numbers requested are always 16, 16k
. This makes me suspicious<br>
> <br>
> I have no setting in nginx.conf that
reference a 16k<br>
> <br>
> Is there any chance of finding out what
requests this and why this is not<br>
> fulfilled<br>
<br>
There are at least some buffers which default
to 16k - for <br>
example, ssl_buffer_size (<a href="http://nginx.org/r/ssl_buffer_size" rel="noreferrer" target="_blank">http://nginx.org/r/ssl_buffer_size</a>).<br>
<br>
You may try debugging log to futher find out
where the particular <br>
allocation happens, see here for details:<br>
<br>
<a href="http://nginx.org/en/docs/debugging_log.html" rel="noreferrer" target="_blank">http://nginx.org/en/docs/debugging_log.html</a><br>
<br>
But I don't really think it worth the effort.Â
The error is pretty <br>
clear, and it's better to focus on why these
allocations are <br>
denied. Likely you are hitting some limit.<br>
<br>
-- <br>
Maxim Dounin<br>
<a href="http://mdounin.ru/" rel="noreferrer" target="_blank">http://mdounin.ru/</a><br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div><b>Anoop P Alias</b>Â
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
nginx mailing list
<a class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>
<a class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
</blockquote>
<p><br>
</p>
</div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="m_367398162948788752m_-1786270149800573001gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div><b>Anoop P Alias</b>Â
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset class="m_367398162948788752m_-1786270149800573001mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
nginx mailing list
<a class="m_367398162948788752m_-1786270149800573001moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>
<a class="m_367398162948788752m_-1786270149800573001moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
</blockquote>
<p><br>
</p>
</div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="m_367398162948788752gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div><b>Anoop P Alias</b>Â
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset class="m_367398162948788752mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
nginx mailing list
<a class="m_367398162948788752moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>
<a class="m_367398162948788752moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
</blockquote>
<p><br>
</p>
</div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><b>Anoop P Alias</b>Â <div><br></div></div></div></div>