<div dir="ltr">Hi Igor,<div><br></div><div>Yes the server runs other software including httpd with a similar number of vhost</div><div><br></div><div><div># grep "<VirtualHost" /etc/apache2/conf/httpd.conf|wc -l</div><div>5168</div></div><div><br></div><div>I haven't found issue with the other softwares in the logs relating to memory</div><div><br></div><div>Infact httpd (event mpm) use lesser memory to load similar config </div><div><br></div><div># ps_mem| head -1 && ps_mem |grep httpd</div><div> Private  +   Shared  =  RAM used       Program </div><div>585.6 MiB + 392.0 MiB = 977.6 MiB       httpd (63) </div><div><br></div><div><div># ps_mem| head -1 && ps_mem |grep nginx</div><div> Private  +   Shared  =  RAM used       Program </div><div>999.8 MiB +   2.5 GiB =   3.5 GiB       nginx (3)</div></div><div><br></div><div>The server is a shared hosting one and runs CloudLinux , but as far as I know ,CloudLinux applies limits to only user level process and not nginx </div><div><br></div><div>The nginx HUP is needed as this is triggered by changes in apache configuration and nginx need to reload the new config . For log file reload SIGUSR1 is used </div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Aug 7, 2018 at 5:50 PM Igor A. Ippolitov <<a href="mailto:iippolitov@nginx.com">iippolitov@nginx.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <div class="m_-6809339306932165473moz-cite-prefix">Anoop,<br>
      <br>
      I don't see any troubles with your configuration.<br>
      Also, if you have 120G of RAM and a single worker - the problem is
      not in nginx.<br>
      Do you have other software running on the host?<br>
      <br>
      Basically, you just run out of memory.<br>
      <br>
      You can optimize your reload though: use "service nginx reload"
      (or "kill -SIGHUP") to reload nginx configuration.<br>
      When you do nginx -s reload - you make nginx parse configuration
      (and it requires memory) and then send a signal to the running
      master. You can avoid this overhead with 'service' command as it
      uses 'kill' documented in the manual page.<br>
      <br>
      On 06.08.2018 22:55, Anoop Alias wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">Hi Igor,
        <div><br>
        </div>
        <div>Config is reloaded using </div>
        <div><br>
        </div>
        <div>/usr/sbin/nginx -s reload </div>
        <div><br>
        </div>
        <div>this is invoked from a python/shell script ( Nginx is
          installed on a web control panel )</div>
        <div><br>
        </div>
        <div>The top-level Nginx config is in the gist below </div>
        <div><br>
        </div>
        <div><a href="https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a" target="_blank">https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a</a><br>
        </div>
        <div><br>
        </div>
        <div>It further includes ~8k server blocks or more in some
          servers. Out of this 2/3 are server {} blocks with TLS config
          and 1/3 non-TLS ones</div>
        <div><br>
        </div>
        <div>
          <div>]# pwd</div>
          <div>/etc/nginx/sites-enabled</div>
        </div>
        <div>
          <div># grep "server {" *|wc -l</div>
          <div>7886</div>
        </div>
        <div><br>
        </div>
        <div>And yes most of them are very similar and mostly proxy to
          upstream httpd</div>
        <div><br>
        </div>
        <div>I have tried removing all the loadable modules and even
          tried an older version of nginx and all produce the error </div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div>
          <div># numastat -m</div>
          <div><br>
          </div>
          <div>Per-node system memory usage (in MBs):</div>
          <div>                          Node 0          Node 1         
             Total</div>
          <div>                 --------------- ---------------
            ---------------</div>
          <div>MemTotal                65430.84        65536.00     
             130966.84</div>
          <div>MemFree                  5491.26           40.89       
             5532.15</div>
          <div>MemUsed                 59939.58        65495.11     
             125434.69</div>
          <div>Active                  22295.61        21016.09       
            43311.70</div>
          <div>Inactive                 8742.76         4662.48       
            13405.24</div>
          <div>Active(anon)            16717.10        16572.19       
            33289.29</div>
          <div>Inactive(anon)           2931.94         1388.14       
             4320.08</div>
          <div>Active(file)             5578.50         4443.91       
            10022.41</div>
          <div>Inactive(file)           5810.82         3274.34       
             9085.16</div>
          <div>Unevictable                 0.00            0.00         
              0.00</div>
          <div>Mlocked                     0.00            0.00         
              0.00</div>
          <div>Dirty                       7.04            1.64         
              8.67</div>
          <div>Writeback                   0.00            0.00         
              0.00</div>
          <div>FilePages               18458.93        10413.97       
            28872.90</div>
          <div>Mapped                    862.14          413.38       
             1275.52</div>
          <div>AnonPages               12579.49        15264.37       
            27843.86</div>
          <div>Shmem                    7069.52         2695.71       
             9765.23</div>
          <div>KernelStack                18.34            3.03         
             21.38</div>
          <div>PageTables                153.14          107.77         
            260.90</div>
          <div>NFS_Unstable                0.00            0.00         
              0.00</div>
          <div>Bounce                      0.00            0.00         
              0.00</div>
          <div>WritebackTmp                0.00            0.00         
              0.00</div>
          <div>Slab                     4830.68         2254.55       
             7085.22</div>
          <div>SReclaimable             2061.05          921.72       
             2982.77</div>
          <div>SUnreclaim               2769.62         1332.83       
             4102.45</div>
          <div>AnonHugePages               4.00            2.00         
              6.00</div>
          <div>HugePages_Total             0.00            0.00         
              0.00</div>
          <div>HugePages_Free              0.00            0.00         
              0.00</div>
          <div>HugePages_Surp              0.00            0.00         
              0.00</div>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div>Thanks,</div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr">On Mon, Aug 6, 2018 at 6:33 PM Igor A. Ippolitov
          <<a href="mailto:iippolitov@nginx.com" target="_blank">iippolitov@nginx.com</a>> wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div text="#000000" bgcolor="#FFFFFF">
            <div class="m_-6809339306932165473m_367398162948788752moz-cite-prefix">Anoop,<br>
              <br>
              I suppose, most of your 10k servers are very similar,
              right?<br>
              Please, post top level configuration and a typical
              server{}, please.<br>
              <br>
              Also, how do you reload configuration? With 'service nginx
              reload' or may be other commands?<br>
              <br>
              It looks like you have a lot of fragmented memory and only
              4gb free in the second numa node.<br>
              So, I'd say this is OK that you are getting errors from
              allocating a 16k stripes.<br>
              <br>
              Could you please post numastat -m output additionally.
              Just to make sure you have half of the memory for the
              second CPU.<br>
              And we'll have a look if memory utilization may be
              optimized based on your configuration.<br>
              <br>
              Regards,<br>
              Igor.<br>
              <br>
              On 04.08.2018 07:54, Anoop Alias wrote:<br>
            </div>
            <blockquote type="cite">
              <div dir="ltr">Hi Igor,
                <div><br>
                </div>
                <div>Setting vm.max_map_count to 20x the normal value
                  did not help</div>
                <div><br>
                </div>
                <div>The issue happens on a group of servers and among
                  the group, it shows up only in servers which have
                  ~10k  server{} blocks</div>
                <div><br>
                </div>
                <div>On servers that have lower number of server{}
                  blocks , the ENOMEM issue is not there</div>
                <div><br>
                </div>
                <div>Also, I can find that the RAM usage of the Nginx
                  process is directly proportional to the number of
                  server {} blocks</div>
                <div><br>
                </div>
                <div>For example on a server having the problem</div>
                <div><br>
                </div>
                <div>
                  <div># ps_mem| head -1 && ps_mem |grep nginx</div>
                  <div> Private  +   Shared  =  RAM used       Program </div>
                  <div>  1.0 GiB +   2.8 GiB =   3.8 GiB       nginx (3)</div>
                </div>
                <div><br>
                </div>
                <div><br>
                </div>
                <div>That is for a single worker process with 4 threads
                  in thread_pool </div>
                <div>
                  <div># pstree|grep nginx</div>
                  <div>        |-nginx-+-nginx---4*[{nginx}]</div>
                  <div>        |       `-nginx</div>
                </div>
                <div><br>
                </div>
                <div>Whatever config change I try the memory usage seem
                  to mostly depend on the number of server contexts
                  defined</div>
                <div><br>
                </div>
                <div>Now the issue mostly happen in nginx reload ,when
                  one more worker process will be active in shutting
                  down mode </div>
                <div><br>
                </div>
                <div>I believe the memalign error is thrown by the
                  worker being shutdown, this is because the sites work
                  after the error and also the pid mentioned in the
                  error would have gone when I check ps</div>
                <div><br>
                </div>
                <div><br>
                </div>
                <div>
                  <div># pmap 948965|grep 16K</div>
                  <div>00007f2923ff2000     16K r-x--
                    ngx_http_redis2_module.so</div>
                  <div>00007f2924fd7000     16K r---- <a href="http://libc-2.17.so" target="_blank">libc-2.17.so</a></div>
                  <div>00007f2925431000     16K rw---   [ anon ]</div>
                  <div>00007f292584a000     16K rw---   [ anon ]</div>
                </div>
                <div><br>
                </div>
                <div>
                  <div>Aug  4 05:50:00 b kernel: SysRq : Show Memory</div>
                  <div>Aug  4 05:50:00 b kernel: Mem-Info:</div>
                  <div>Aug  4 05:50:00 b kernel: active_anon:7757394
                    inactive_anon:1021319 isolated_anon:0#012 active_<a class="m_-6809339306932165473m_367398162948788752moz-txt-link-freetext">file:3733324</a> inactive_<a class="m_-6809339306932165473m_367398162948788752moz-txt-link-freetext">file:2136476</a> isolated_<a class="m_-6809339306932165473m_367398162948788752moz-txt-link-freetext">file:0#012</a>
                    unevictable:0 dirty:1766 writeback:6 wbtmp:0
                    unstable:0#012 slab_reclaimable:2003687
                    slab_unreclaimable:901391#012 mapped:316734
                    shmem:2381810 pagetables:63163 bounce:0#012
                    free:4851283 free_pcp:11332 free_cma:0</div>
                  <div>Aug  4 05:50:00 bravo kernel: Node 0 DMA
                    free:15888kB min:8kB low:8kB high:12kB
                    active_anon:0kB inactive_anon:0kB active_<a class="m_-6809339306932165473m_367398162948788752moz-txt-link-freetext">file:0kB</a> inactive_<a class="m_-6809339306932165473m_367398162948788752moz-txt-link-freetext">file:0kB</a>
                    unevictable:0kB isolated(anon):0kB
                    isolated(file):0kB present:15972kB managed:15888kB
                    mlocked:0kB dirty:0kB writeback:0kB mapped:0kB
                    shmem:0kB slab_reclaimable:0kB
                    slab_unreclaimable:0kB kernel_stack:0kB
                    pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB
                    local_pcp:0kB free_cma:0kB writeback_tmp:0kB
                    pages_scanned:0 all_unreclaimable? yes</div>
                  <div>Aug  4 05:50:00 b kernel: lowmem_reserve[]: 0
                    1679 64139 64139</div>
                </div>
                <div><br>
                </div>
                <div>
                  <div># cat /proc/buddyinfo</div>
                  <div>Node 0, zone      DMA      0      0      1     
                    0      2      1      1      0      1      1      3 </div>
                  <div>Node 0, zone    DMA32   5284   6753   6677 
                     1083    410     59      1      0      0      0     
                    0 </div>
                  <div>Node 0, zone   Normal 500327 638958 406737 
                    14690    872    106     11      0      0      0     
                    0 </div>
                  <div>Node 1, zone   Normal 584840 291640    188     
                    0      0      0      0      0      0      0      0 </div>
                </div>
                <div><br>
                </div>
                <div><br>
                </div>
                <div>The only correlation I see in having the error is
                  the number of server {} blocks (close to 10k) which
                  then makes the nginx process consume ~ 4GB of mem with
                  a single worker process and then a reload is done</div>
                <div><br>
                </div>
                <div><br>
                </div>
                <div><br>
                </div>
              </div>
              <br>
              <div class="gmail_quote">
                <div dir="ltr">On Thu, Aug 2, 2018 at 6:02 PM Igor A.
                  Ippolitov <<a href="mailto:iippolitov@nginx.com" target="_blank">iippolitov@nginx.com</a>>
                  wrote:<br>
                </div>
                <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                  <div text="#000000" bgcolor="#FFFFFF">
                    <div class="m_-6809339306932165473m_367398162948788752m_-1786270149800573001moz-cite-prefix">Anoop,<br>
                      <br>
                      There are two guesses: either mmap allocations
                      limit is hit or memory is  way too fragmented.<br>
                      Could you please track amount of mapped regions
                      for a worker with pmap and amount of 16k areas in
                      Normal zones (it is the third number)?<br>
                      <br>
                      You can also set vm.max_map_count to a higher
                      number (like 20 times higher than default) and
                      look if the error is gone.<br>
                      <br>
                      Please, let me know if increasing vm.max_map_count
                      helps you.<br>
                      <br>
                      On 02.08.2018 13:06, Anoop Alias wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr">Hi Igor,
                        <div><br>
                        </div>
                        <div>The error happens randomly </div>
                        <div>
                          <div><br>
                          </div>
                          <div>
                            <div>2018/08/02 06:52:42 [emerg]
                              874514#874514: posix_memalign(16, 16384)
                              failed (12: Cannot allocate memory)</div>
                            <div>2018/08/02 09:42:53 [emerg]
                              872996#872996: posix_memalign(16, 16384)
                              failed (12: Cannot allocate memory)</div>
                            <div>2018/08/02 10:16:14 [emerg]
                              877611#877611: posix_memalign(16, 16384)
                              failed (12: Cannot allocate memory)</div>
                            <div>2018/08/02 10:16:48 [emerg]
                              879410#879410: posix_memalign(16, 16384)
                              failed (12: Cannot allocate memory)</div>
                            <div>2018/08/02 10:17:55 [emerg]
                              876563#876563: posix_memalign(16, 16384)
                              failed (12: Cannot allocate memory)</div>
                            <div>2018/08/02 10:20:21 [emerg]
                              879263#879263: posix_memalign(16, 16384)
                              failed (12: Cannot allocate memory)</div>
                            <div>2018/08/02 10:20:51 [emerg]
                              878991#878991: posix_memalign(16, 16384)
                              failed (12: Cannot allocate memory)</div>
                          </div>
                          <div><br>
                          </div>
                          <div>
                            <div># date</div>
                            <div>Thu Aug  2 10:58:48 BST 2018</div>
                          </div>
                          <div><br>
                          </div>
                          <div>------------------------------------------</div>
                          <div>
                            <div># cat /proc/buddyinfo</div>
                            <div>Node 0, zone      DMA      0      0   
                                1      0      2      1      1      0   
                                1      1      3 </div>
                            <div>Node 0, zone    DMA32  11722  11057 
                               4663   1647    609     72     10      7 
                                  1      0      0 </div>
                            <div>Node 0, zone   Normal 755026 710760
                              398136  21462   1114     18      1      0 
                                  0      0      0 </div>
                            <div>Node 1, zone   Normal 341295 801810
                              179604    256      0      0      0      0 
                                  0      0      0 </div>
                          </div>
                          <div>-----------------------------------------</div>
                          <div><br>
                          </div>
                          <div><br>
                          </div>
                          <div>
                            <div>slabinfo - version: 2.1</div>
                            <div># name            <active_objs>
                              <num_objs> <objsize>
                              <objperslab> <pagesperslab> :
                              tunables <limit> <batchcount>
                              <sharedfactor> : slabdata
                              <active_slabs> <num_slabs>
                              <sharedavail></div>
                            <div>SCTPv6                21     21   1536 
                               21    8 : tunables    0    0    0 :
                              slabdata      1      1      0</div>
                            <div>SCTP                   0      0   1408 
                               23    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>kcopyd_job             0      0   3312 
                                9    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dm_uevent              0      0   2608 
                               12    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>nf_conntrack_ffffffff81acbb00  14054 
                              14892    320   51    4 : tunables    0   
                              0    0 : slabdata    292    292      0</div>
                            <div>lvp_cache             36     36    224 
                               36    2 : tunables    0    0    0 :
                              slabdata      1      1      0</div>
                            <div>lve_struct          4140   4140    352 
                               46    4 : tunables    0    0    0 :
                              slabdata     90     90      0</div>
                            <div>fat_inode_cache        0      0    744 
                               44    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>fat_cache              0      0     40 
                              102    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>isofs_inode_cache      0      0    664 
                               49    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>ext4_inode_cache      30     30   1088 
                               30    8 : tunables    0    0    0 :
                              slabdata      1      1      0</div>
                            <div>ext4_xattr             0      0     88 
                               46    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>ext4_free_data         0      0     64 
                               64    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>ext4_allocation_context     32     32 
                                128   32    1 : tunables    0    0    0
                              : slabdata      1      1      0</div>
                            <div>ext4_io_end            0      0     72 
                               56    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>ext4_extent_status    102    102   
                               40  102    1 : tunables    0    0    0 :
                              slabdata      1      1      0</div>
                            <div>jbd2_journal_handle      0      0   
                               48   85    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>jbd2_journal_head      0      0    112 
                               36    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>jbd2_revoke_table_s    256    256   
                               16  256    1 : tunables    0    0    0 :
                              slabdata      1      1      0</div>
                            <div>jbd2_revoke_record_s      0      0   
                               32  128    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>kvm_async_pf           0      0    136 
                               30    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>kvm_vcpu               0      0  18560 
                                1    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>xfs_dqtrx            992    992    528 
                               31    4 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>xfs_dquot           3264   3264    472 
                               34    4 : tunables    0    0    0 :
                              slabdata     96     96      0</div>
                            <div>xfs_ili           4342175 4774399   
                              152   53    2 : tunables    0    0    0 :
                              slabdata  90083  90083      0</div>
                            <div>xfs_inode         4915588 5486076 
                               1088   30    8 : tunables    0    0    0
                              : slabdata 182871 182871      0</div>
                            <div>xfs_efd_item        2680   2760    400 
                               40    4 : tunables    0    0    0 :
                              slabdata     69     69      0</div>
                            <div>xfs_da_state        1088   1088    480 
                               34    4 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>xfs_btree_cur       1248   1248    208 
                               39    2 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>xfs_log_ticket     14874  15048    184 
                               44    2 : tunables    0    0    0 :
                              slabdata    342    342      0</div>
                            <div>xfs_ioend          12909  13104    104 
                               39    1 : tunables    0    0    0 :
                              slabdata    336    336      0</div>
                            <div>scsi_cmd_cache      5400   5652    448 
                               36    4 : tunables    0    0    0 :
                              slabdata    157    157      0</div>
                            <div>ve_struct              0      0    848 
                               38    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>ip6_dst_cache       1152   1152    448 
                               36    4 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>RAWv6                910    910   1216 
                               26    8 : tunables    0    0    0 :
                              slabdata     35     35      0</div>
                            <div>UDPLITEv6              0      0   1216 
                               26    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                          </div>
                          <div>
                            <div>UDPv6                832    832   1216 
                               26    8 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>tw_sock_TCPv6       1152   1376    256 
                               32    2 : tunables    0    0    0 :
                              slabdata     43     43      0</div>
                            <div>TCPv6                510    510   2176 
                               15    8 : tunables    0    0    0 :
                              slabdata     34     34      0</div>
                            <div>cfq_queue           3698   5145    232 
                               35    2 : tunables    0    0    0 :
                              slabdata    147    147      0</div>
                            <div>bsg_cmd                0      0    312 
                               52    4 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>mqueue_inode_cache    136    136   
                              960   34    8 : tunables    0    0    0 :
                              slabdata      4      4      0</div>
                            <div>hugetlbfs_inode_cache   1632   1632   
                              632   51    8 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>configfs_dir_cache   1472   1472   
                               88   46    1 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>dquot                  0      0    256 
                               32    2 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>userfaultfd_ctx_cache     32     32   
                              128   32    1 : tunables    0    0    0 :
                              slabdata      1      1      0</div>
                            <div>fanotify_event_info   2336   2336   
                               56   73    1 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>dio                 6171   6222    640 
                               51    8 : tunables    0    0    0 :
                              slabdata    122    122      0</div>
                            <div>pid_namespace         42     42   2192 
                               14    8 : tunables    0    0    0 :
                              slabdata      3      3      0</div>
                            <div>posix_timers_cache   1056   1056   
                              248   33    2 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>UDP-Lite               0      0   1088 
                               30    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>flow_cache          2268   2296    144 
                               28    1 : tunables    0    0    0 :
                              slabdata     82     82      0</div>
                            <div>xfrm_dst_cache       896    896    576 
                               28    4 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>ip_fib_alias        2720   2720     48 
                               85    1 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>RAW                 3977   4224   1024 
                               32    8 : tunables    0    0    0 :
                              slabdata    132    132      0</div>
                            <div>UDP                 4110   4110   1088 
                               30    8 : tunables    0    0    0 :
                              slabdata    137    137      0</div>
                            <div>tw_sock_TCP         4756   5216    256 
                               32    2 : tunables    0    0    0 :
                              slabdata    163    163      0</div>
                            <div>TCP                 2705   2768   1984 
                               16    8 : tunables    0    0    0 :
                              slabdata    173    173      0</div>
                            <div>scsi_data_buffer    5440   5440     24 
                              170    1 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>blkdev_queue         154    154   2208 
                               14    8 : tunables    0    0    0 :
                              slabdata     11     11      0</div>
                            <div>blkdev_requests   4397688 4405884   
                              384   42    4 : tunables    0    0    0 :
                              slabdata 104902 104902      0</div>
                            <div>blkdev_ioc         11232  11232    112 
                               36    1 : tunables    0    0    0 :
                              slabdata    312    312      0</div>
                            <div>user_namespace         0      0   1304 
                               25    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>sock_inode_cache   12282  12282    704 
                               46    8 : tunables    0    0    0 :
                              slabdata    267    267      0</div>
                            <div>file_lock_cache    20056  20960    200 
                               40    2 : tunables    0    0    0 :
                              slabdata    524    524      0</div>
                            <div>net_namespace          6      6   5056 
                                6    8 : tunables    0    0    0 :
                              slabdata      1      1      0</div>
                            <div>shmem_inode_cache  16970  18952    712 
                               46    8 : tunables    0    0    0 :
                              slabdata    412    412      0</div>
                            <div>Acpi-ParseExt      39491  40432     72 
                               56    1 : tunables    0    0    0 :
                              slabdata    722    722      0</div>
                            <div>Acpi-State          1683   1683     80 
                               51    1 : tunables    0    0    0 :
                              slabdata     33     33      0</div>
                            <div>Acpi-Namespace     11424  11424     40 
                              102    1 : tunables    0    0    0 :
                              slabdata    112    112      0</div>
                            <div>task_delay_info    15336  15336    112 
                               36    1 : tunables    0    0    0 :
                              slabdata    426    426      0</div>
                            <div>taskstats           1568   1568    328 
                               49    4 : tunables    0    0    0 :
                              slabdata     32     32      0</div>
                            <div>proc_inode_cache  169897 190608    680 
                               48    8 : tunables    0    0    0 :
                              slabdata   3971   3971      0</div>
                            <div>sigqueue            2208   2208    168 
                               48    2 : tunables    0    0    0 :
                              slabdata     46     46      0</div>
                            <div>bdev_cache           792    792    896 
                               36    8 : tunables    0    0    0 :
                              slabdata     22     22      0</div>
                          </div>
                          <div>
                            <div>sysfs_dir_cache    74698  74698    120 
                               34    1 : tunables    0    0    0 :
                              slabdata   2197   2197      0</div>
                            <div>mnt_cache         163197 163424    256 
                               32    2 : tunables    0    0    0 :
                              slabdata   5107   5107      0</div>
                            <div>filp               64607  97257    320 
                               51    4 : tunables    0    0    0 :
                              slabdata   1907   1907      0</div>
                            <div>inode_cache       370744 370947    616 
                               53    8 : tunables    0    0    0 :
                              slabdata   6999   6999      0</div>
                            <div>dentry            1316262 2139228   
                              192   42    2 : tunables    0    0    0 :
                              slabdata  50934  50934      0</div>
                            <div>iint_cache             0      0     80 
                               51    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>buffer_head       1441470 2890290   
                              104   39    1 : tunables    0    0    0 :
                              slabdata  74110  74110      0</div>
                            <div>vm_area_struct    194998 196840    216 
                               37    2 : tunables    0    0    0 :
                              slabdata   5320   5320      0</div>
                            <div>mm_struct           2679   2760   1600 
                               20    8 : tunables    0    0    0 :
                              slabdata    138    138      0</div>
                            <div>files_cache         8680   8925    640 
                               51    8 : tunables    0    0    0 :
                              slabdata    175    175      0</div>
                            <div>signal_cache        3691   3780   1152 
                               28    8 : tunables    0    0    0 :
                              slabdata    135    135      0</div>
                            <div>sighand_cache       1950   2160   2112 
                               15    8 : tunables    0    0    0 :
                              slabdata    144    144      0</div>
                            <div>task_xstate         8070   8658    832 
                               39    8 : tunables    0    0    0 :
                              slabdata    222    222      0</div>
                            <div>task_struct         1913   2088   4080 
                                8    8 : tunables    0    0    0 :
                              slabdata    261    261      0</div>
                            <div>cred_jar           31699  33936    192 
                               42    2 : tunables    0    0    0 :
                              slabdata    808    808      0</div>
                            <div>anon_vma_chain    164026 168704     64 
                               64    1 : tunables    0    0    0 :
                              slabdata   2636   2636      0</div>
                            <div>anon_vma           84104  84594     88 
                               46    1 : tunables    0    0    0 :
                              slabdata   1839   1839      0</div>
                            <div>pid                11127  12576    128 
                               32    1 : tunables    0    0    0 :
                              slabdata    393    393      0</div>
                            <div>shared_policy_node   9350   9350   
                               48   85    1 : tunables    0    0    0 :
                              slabdata    110    110      0</div>
                            <div>numa_policy           62     62    264 
                               31    2 : tunables    0    0    0 :
                              slabdata      2      2      0</div>
                            <div>radix_tree_node   771778 1194312   
                              584   28    4 : tunables    0    0    0 :
                              slabdata  42654  42654      0</div>
                            <div>idr_layer_cache     2538   2565   2112 
                               15    8 : tunables    0    0    0 :
                              slabdata    171    171      0</div>
                            <div>dma-kmalloc-8192       0      0   8192 
                                4    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-4096       0      0   4096 
                                8    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-2048       0      0   2048 
                               16    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-1024       0      0   1024 
                               32    8 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-512        0      0    512 
                               32    4 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-256        0      0    256 
                               32    2 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-128        0      0    128 
                               32    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-64         0      0     64 
                               64    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-32         0      0     32 
                              128    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-16         0      0     16 
                              256    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-8          0      0      8 
                              512    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-192        0      0    192 
                               42    2 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>dma-kmalloc-96         0      0     96 
                               42    1 : tunables    0    0    0 :
                              slabdata      0      0      0</div>
                            <div>kmalloc-8192         385    388   8192 
                                4    8 : tunables    0    0    0 :
                              slabdata     97     97      0</div>
                            <div>kmalloc-4096        9296  10088   4096 
                                8    8 : tunables    0    0    0 :
                              slabdata   1261   1261      0</div>
                            <div>kmalloc-2048       65061 133536   2048 
                               16    8 : tunables    0    0    0 :
                              slabdata   8346   8346      0</div>
                            <div>kmalloc-1024       11987  21120   1024 
                               32    8 : tunables    0    0    0 :
                              slabdata    660    660      0</div>
                          </div>
                          <div>
                            <div>kmalloc-512       107510 187072    512 
                               32    4 : tunables    0    0    0 :
                              slabdata   5846   5846      0</div>
                            <div>kmalloc-256       160498 199104    256 
                               32    2 : tunables    0    0    0 :
                              slabdata   6222   6222      0</div>
                            <div>kmalloc-192       144975 237426    192 
                               42    2 : tunables    0    0    0 :
                              slabdata   5653   5653      0</div>
                            <div>kmalloc-128        36799 108096    128 
                               32    1 : tunables    0    0    0 :
                              slabdata   3378   3378      0</div>
                            <div>kmalloc-96         99510 238896     96 
                               42    1 : tunables    0    0    0 :
                              slabdata   5688   5688      0</div>
                            <div>kmalloc-64        7978152 8593280   
                               64   64    1 : tunables    0    0    0 :
                              slabdata 134270 134270      0</div>
                            <div>kmalloc-32        2939882 3089664   
                               32  128    1 : tunables    0    0    0 :
                              slabdata  24138  24138      0</div>
                            <div>kmalloc-16        172057 172288     16 
                              256    1 : tunables    0    0    0 :
                              slabdata    673    673      0</div>
                            <div>kmalloc-8         109568 109568      8 
                              512    1 : tunables    0    0    0 :
                              slabdata    214    214      0</div>
                            <div>kmem_cache_node      893    896     64 
                               64    1 : tunables    0    0    0 :
                              slabdata     14     14      0</div>
                            <div>kmem_cache           612    612    320 
                               51    4 : tunables    0    0    0 :
                              slabdata     12     12      0</div>
                          </div>
                          <div><br>
                          </div>
                          <div>-------------------------------------------------</div>
                          <div><br>
                          </div>
                          <div><br>
                          </div>
                          <div>
                            <div># uname -r</div>
                            <div>3.10.0-714.10.2.lve1.5.17.1.el7.x86_64</div>
                          </div>
                          <div><br>
                          </div>
                          <div>--------------------------------------------------------</div>
                          <div><br>
                          </div>
                          <div>Core part of glances </div>
                          <div><a href="http://i.imgur.com/La5JbQn.png" target="_blank">http://i.imgur.com/La5JbQn.png</a><br>
                          </div>
                          <div>-----------------------------------------------------------</div>
                          <div><br>
                          </div>
                          <div>Thank you very much for looking into this</div>
                          <div><br>
                          </div>
                        </div>
                      </div>
                      <br>
                      <div class="gmail_quote">
                        <div dir="ltr">On Thu, Aug 2, 2018 at 12:37 PM
                          Igor A. Ippolitov <<a href="mailto:iippolitov@nginx.com" target="_blank">iippolitov@nginx.com</a>>
                          wrote:<br>
                        </div>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          <div text="#000000" bgcolor="#FFFFFF">
                            <div class="m_-6809339306932165473m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-cite-prefix">Anoop,<br>
                              <br>
                              I doubt this will be the solution, but may
                              we have a look at /proc/buddyinfo and
                              /proc/slabinfo the moment when nginx can't
                              allocate memory?<br>
                              <br>
                              On 02.08.2018 08:15, Anoop Alias wrote:<br>
                            </div>
                            <blockquote type="cite">
                              <div dir="ltr">
                                <div>Hi Maxim,</div>
                                <div><br>
                                </div>
                                <div>I enabled debug and the memalign
                                  call is happening on nginx reloads and
                                  the ENOMEM happen sometimes on the
                                  reload(not on all reloads)</div>
                                <div><br>
                                </div>
                                <div>
                                  <div>2018/08/02 05:59:08 [notice]
                                    872052#872052: signal process
                                    started</div>
                                  <div>2018/08/02 05:59:23 [notice]
                                    871570#871570: signal 1 (SIGHUP)
                                    received from 872052, reconfiguring</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: wake up, sigio 0</div>
                                  <div>2018/08/02 05:59:23 [notice]
                                    871570#871570: reconfiguring</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    0000000002B0DA00:16384 @16      ===
                                    > the memalign call on reload</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: malloc:
                                    00000000087924D0:4560</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    000000000E442E00:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: malloc:
                                    0000000005650850:4096</div>
                                  <div>20</div>
                                </div>
                                <div><br>
                                </div>
                                <div><br>
                                </div>
                                <div><br>
                                </div>
                                <div><br>
                                </div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: bind() xxxx:443 #71</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: bind() xxxx:443 #72</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: bind() xxxx:443 #73</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: bind() xxxx:443 #74</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: add cleanup:
                                  000000005340D728</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000024D3260:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000517BAF10:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  0000000053854FC0:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  0000000053855FD0:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  0000000053856FE0:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  0000000053857FF0:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: posix_memalign:
                                  0000000053859000:16384 @16</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  000000005385D010:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  000000005385E020:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  000000005385F030:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536CD160:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536CE170:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536CF180:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536D0190:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536D11A0:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536D21B0:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536D31C0:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536D41D0:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536D51E0:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536D61F0:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536D7200:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536D8210:4096</div>
                                <div>2018/08/02 05:48:49 [debug]
                                  871275#871275: malloc:
                                  00000000536D9220:4096</div>
                                <div><br>
                                </div>
                                <div><br>
                                </div>
                                <div>Infact there are lot of such calls
                                  during a reload</div>
                                <div><br>
                                </div>
                                <div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA17ED00:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA1B0FF0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA1E12C0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA211590:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA243880:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA271B30:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA2A3E20:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA2D20D0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA3063E0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA334690:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA366980:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA396C50:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA3C8F40:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA3F9210:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA4294E0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA45B7D0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA489A80:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA4BBD70:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA4EA020:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA51E330:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA54C5E0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA57E8D0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA5AEBA0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA5DEE70:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA611160:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA641430:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA671700:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA6A29E0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA6D5CE0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA707FD0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA736280:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA768570:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA796820:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA7CAB30:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA7F8DE0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA82B0D0:16384 @16</div>
                                  <div>2018/08/02 05:59:23 [debug]
                                    871570#871570: posix_memalign:
                                    00000000BA85B3A0:16384 @16</div>
                                </div>
                                <div><br>
                                </div>
                                <div><br>
                                </div>
                                <div><br>
                                </div>
                                <div>What is perplexing is that the
                                  system has enough free (available RAM)</div>
                                <div>#############</div>
                                <div>
                                  <div># free -g</div>
                                  <div>              total        used 
                                          free      shared  buff/cache 
                                     available</div>
                                  <div>Mem:            125          54 
                                            24           8          46 
                                            58</div>
                                  <div>Swap:             0           0 
                                             0</div>
                                </div>
                                <div>#############</div>
                                <div><br>
                                </div>
                                <div>
                                  <div># ulimit -a</div>
                                  <div>core file size          (blocks,
                                    -c) 0</div>
                                  <div>data seg size           (kbytes,
                                    -d) unlimited</div>
                                  <div>scheduling priority           
                                     (-e) 0</div>
                                  <div>file size               (blocks,
                                    -f) unlimited</div>
                                  <div>pending signals               
                                     (-i) 514579</div>
                                  <div>max locked memory       (kbytes,
                                    -l) 64</div>
                                  <div>max memory size         (kbytes,
                                    -m) unlimited</div>
                                  <div>open files                     
                                    (-n) 1024</div>
                                  <div>pipe size            (512 bytes,
                                    -p) 8</div>
                                  <div>POSIX message queues     (bytes,
                                    -q) 819200</div>
                                  <div>real-time priority             
                                    (-r) 0</div>
                                  <div>stack size              (kbytes,
                                    -s) 8192</div>
                                  <div>cpu time               (seconds,
                                    -t) unlimited</div>
                                  <div>max user processes             
                                    (-u) 514579</div>
                                  <div>virtual memory          (kbytes,
                                    -v) unlimited</div>
                                  <div>file locks                     
                                    (-x) unlimited</div>
                                </div>
                                <div><br>
                                </div>
                                <div>#########################################</div>
                                <div><br>
                                </div>
                                <div>There is no other thing limiting
                                  memory allocation </div>
                                <div><br>
                                </div>
                                <div>Any way to prevent this or probably
                                  identify/prevent this</div>
                                <div><br>
                                </div>
                              </div>
                              <br>
                              <div class="gmail_quote">
                                <div dir="ltr">On Tue, Jul 31, 2018 at
                                  7:08 PM Maxim Dounin <<a href="mailto:mdounin@mdounin.ru" target="_blank">mdounin@mdounin.ru</a>>
                                  wrote:<br>
                                </div>
                                <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello!<br>
                                  <br>
                                  On Tue, Jul 31, 2018 at 09:52:29AM
                                  +0530, Anoop Alias wrote:<br>
                                  <br>
                                  > I am repeatedly seeing errors
                                  like<br>
                                  > <br>
                                  > ######################<br>
                                  > 2018/07/31 03:46:33 [emerg]
                                  2854560#2854560: posix_memalign(16,
                                  16384)<br>
                                  > failed (12: Cannot allocate
                                  memory)<br>
                                  > 2018/07/31 03:54:09 [emerg]
                                  2890190#2890190: posix_memalign(16,
                                  16384)<br>
                                  > failed (12: Cannot allocate
                                  memory)<br>
                                  > 2018/07/31 04:08:36 [emerg]
                                  2939230#2939230: posix_memalign(16,
                                  16384)<br>
                                  > failed (12: Cannot allocate
                                  memory)<br>
                                  > 2018/07/31 04:24:48 [emerg]
                                  2992650#2992650: posix_memalign(16,
                                  16384)<br>
                                  > failed (12: Cannot allocate
                                  memory)<br>
                                  > 2018/07/31 04:42:09 [emerg]
                                  3053092#3053092: posix_memalign(16,
                                  16384)<br>
                                  > failed (12: Cannot allocate
                                  memory)<br>
                                  > 2018/07/31 04:42:17 [emerg]
                                  3053335#3053335: posix_memalign(16,
                                  16384)<br>
                                  > failed (12: Cannot allocate
                                  memory)<br>
                                  > 2018/07/31 04:42:28 [emerg]
                                  3053937#3053937: posix_memalign(16,
                                  16384)<br>
                                  > failed (12: Cannot allocate
                                  memory)<br>
                                  > 2018/07/31 04:47:54 [emerg]
                                  3070638#3070638: posix_memalign(16,
                                  16384)<br>
                                  > failed (12: Cannot allocate
                                  memory)<br>
                                  > ####################<br>
                                  > <br>
                                  > on a few servers<br>
                                  > <br>
                                  > The servers have enough memory
                                  free and the swap usage is 0, yet
                                  somehow<br>
                                  > the kernel denies the
                                  posix_memalign with ENOMEM ( this is
                                  what I think is<br>
                                  > happening!)<br>
                                  > <br>
                                  > The numbers requested are always
                                  16, 16k . This makes me suspicious<br>
                                  > <br>
                                  > I have no setting in nginx.conf
                                  that reference a 16k<br>
                                  > <br>
                                  > Is there any chance of finding
                                  out what requests this and why this is
                                  not<br>
                                  > fulfilled<br>
                                  <br>
                                  There are at least some buffers which
                                  default to 16k - for <br>
                                  example, ssl_buffer_size (<a href="http://nginx.org/r/ssl_buffer_size" rel="noreferrer" target="_blank">http://nginx.org/r/ssl_buffer_size</a>).<br>
                                  <br>
                                  You may try debugging log to futher
                                  find out where the particular <br>
                                  allocation happens, see here for
                                  details:<br>
                                  <br>
                                  <a href="http://nginx.org/en/docs/debugging_log.html" rel="noreferrer" target="_blank">http://nginx.org/en/docs/debugging_log.html</a><br>
                                  <br>
                                  But I don't really think it worth the
                                  effort.  The error is pretty <br>
                                  clear, and it's better to focus on why
                                  these allocations are <br>
                                  denied.  Likely you are hitting some
                                  limit.<br>
                                  <br>
                                  -- <br>
                                  Maxim Dounin<br>
                                  <a href="http://mdounin.ru/" rel="noreferrer" target="_blank">http://mdounin.ru/</a><br>
_______________________________________________<br>
                                  nginx mailing list<br>
                                  <a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
                                  <a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
                                </blockquote>
                              </div>
                              <br clear="all">
                              <div><br>
                              </div>
                              -- <br>
                              <div dir="ltr" class="m_-6809339306932165473m_367398162948788752m_-1786270149800573001m_-3921009750399893672gmail_signature" data-smartmail="gmail_signature">
                                <div dir="ltr">
                                  <div><b>Anoop P Alias</b> 
                                    <div><br>
                                    </div>
                                  </div>
                                </div>
                              </div>
                              <br>
                              <fieldset class="m_-6809339306932165473m_367398162948788752m_-1786270149800573001m_-3921009750399893672mimeAttachmentHeader"></fieldset>
                              <br>
                              <pre>_______________________________________________
nginx mailing list
<a class="m_-6809339306932165473m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>
<a class="m_-6809339306932165473m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
                            </blockquote>
                            <p><br>
                            </p>
                          </div>
_______________________________________________<br>
                          nginx mailing list<br>
                          <a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
                          <a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
                      </div>
                      <br clear="all">
                      <div><br>
                      </div>
                      -- <br>
                      <div dir="ltr" class="m_-6809339306932165473m_367398162948788752m_-1786270149800573001gmail_signature" data-smartmail="gmail_signature">
                        <div dir="ltr">
                          <div><b>Anoop P Alias</b> 
                            <div><br>
                            </div>
                          </div>
                        </div>
                      </div>
                      <br>
                      <fieldset class="m_-6809339306932165473m_367398162948788752m_-1786270149800573001mimeAttachmentHeader"></fieldset>
                      <br>
                      <pre>_______________________________________________
nginx mailing list
<a class="m_-6809339306932165473m_367398162948788752m_-1786270149800573001moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>
<a class="m_-6809339306932165473m_367398162948788752m_-1786270149800573001moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
                    </blockquote>
                    <p><br>
                    </p>
                  </div>
                  _______________________________________________<br>
                  nginx mailing list<br>
                  <a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
                  <a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
              </div>
              <br clear="all">
              <div><br>
              </div>
              -- <br>
              <div dir="ltr" class="m_-6809339306932165473m_367398162948788752gmail_signature" data-smartmail="gmail_signature">
                <div dir="ltr">
                  <div><b>Anoop P Alias</b> 
                    <div><br>
                    </div>
                  </div>
                </div>
              </div>
              <br>
              <fieldset class="m_-6809339306932165473m_367398162948788752mimeAttachmentHeader"></fieldset>
              <br>
              <pre>_______________________________________________
nginx mailing list
<a class="m_-6809339306932165473m_367398162948788752moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>
<a class="m_-6809339306932165473m_367398162948788752moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
            </blockquote>
            <p><br>
            </p>
          </div>
          _______________________________________________<br>
          nginx mailing list<br>
          <a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
          <a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
      </div>
      <br clear="all">
      <div><br>
      </div>
      -- <br>
      <div dir="ltr" class="m_-6809339306932165473gmail_signature" data-smartmail="gmail_signature">
        <div dir="ltr">
          <div><b>Anoop P Alias</b> 
            <div><br>
            </div>
          </div>
        </div>
      </div>
      <br>
      <fieldset class="m_-6809339306932165473mimeAttachmentHeader"></fieldset>
      <br>
      <pre>_______________________________________________
nginx mailing list
<a class="m_-6809339306932165473moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>
<a class="m_-6809339306932165473moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
    </blockquote>
    <p><br>
    </p>
  </div>

_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><b>Anoop P Alias</b> <div><br></div></div></div></div>