<div dir="ltr">Hi Igor,<div><br></div><div>Config is reloaded using </div><div><br></div><div>/usr/sbin/nginx -s reload </div><div><br></div><div>this is invoked from a python/shell script ( Nginx is installed on a web control panel )</div><div><br></div><div>The top-level Nginx config is in the gist below </div><div><br></div><div><a href="https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a">https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a</a><br></div><div><br></div><div>It further includes ~8k server blocks or more in some servers. Out of this 2/3 are server {} blocks with TLS config and 1/3 non-TLS ones</div><div><br></div><div><div>]# pwd</div><div>/etc/nginx/sites-enabled</div></div><div><div># grep "server {" *|wc -l</div><div>7886</div></div><div><br></div><div>And yes most of them are very similar and mostly proxy to upstream httpd</div><div><br></div><div>I have tried removing all the loadable modules and even tried an older version of nginx and all produce the error </div><div><br></div><div><br></div><div><div># numastat -m</div><div><br></div><div>Per-node system memory usage (in MBs):</div><div>  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Node 0  Â  Â  Â  Â  Node 1  Â  Â  Â  Â  Â Total</div><div>  Â  Â  Â  Â  Â  Â  Â  Â --------------- --------------- ---------------</div><div>MemTotal  Â  Â  Â  Â  Â  Â  Â  65430.84  Â  Â  Â  65536.00  Â  Â  Â 130966.84</div><div>MemFree  Â  Â  Â  Â  Â  Â  Â  Â  5491.26  Â  Â  Â  Â  Â 40.89  Â  Â  Â  Â 5532.15</div><div>MemUsed  Â  Â  Â  Â  Â  Â  Â  Â 59939.58  Â  Â  Â  65495.11  Â  Â  Â 125434.69</div><div>Active  Â  Â  Â  Â  Â  Â  Â  Â  22295.61  Â  Â  Â  21016.09  Â  Â  Â  43311.70</div><div>Inactive  Â  Â  Â  Â  Â  Â  Â  Â 8742.76  Â  Â  Â  Â 4662.48  Â  Â  Â  13405.24</div><div>Active(anon)  Â  Â  Â  Â  Â  16717.10  Â  Â  Â  16572.19  Â  Â  Â  33289.29</div><div>Inactive(anon)  Â  Â  Â  Â  Â 2931.94  Â  Â  Â  Â 1388.14  Â  Â  Â  Â 4320.08</div><div>Active(file)  Â  Â  Â  Â  Â  Â 5578.50  Â  Â  Â  Â 4443.91  Â  Â  Â  10022.41</div><div>Inactive(file)  Â  Â  Â  Â  Â 5810.82  Â  Â  Â  Â 3274.34  Â  Â  Â  Â 9085.16</div><div>Unevictable  Â  Â  Â  Â  Â  Â  Â  Â 0.00  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00</div><div>Mlocked  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â 0.00  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00</div><div>Dirty  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â 7.04  Â  Â  Â  Â  Â  1.64  Â  Â  Â  Â  Â  8.67</div><div>Writeback  Â  Â  Â  Â  Â  Â  Â  Â  Â 0.00  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00</div><div>FilePages  Â  Â  Â  Â  Â  Â  Â 18458.93  Â  Â  Â  10413.97  Â  Â  Â  28872.90</div><div>Mapped  Â  Â  Â  Â  Â  Â  Â  Â  Â  862.14  Â  Â  Â  Â  413.38  Â  Â  Â  Â 1275.52</div><div>AnonPages  Â  Â  Â  Â  Â  Â  Â 12579.49  Â  Â  Â  15264.37  Â  Â  Â  27843.86</div><div>Shmem  Â  Â  Â  Â  Â  Â  Â  Â  Â  7069.52  Â  Â  Â  Â 2695.71  Â  Â  Â  Â 9765.23</div><div>KernelStack  Â  Â  Â  Â  Â  Â  Â  18.34  Â  Â  Â  Â  Â  3.03  Â  Â  Â  Â  Â 21.38</div><div>PageTables  Â  Â  Â  Â  Â  Â  Â  153.14  Â  Â  Â  Â  107.77  Â  Â  Â  Â  260.90</div><div>NFS_Unstable  Â  Â  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00</div><div>Bounce  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00</div><div>WritebackTmp  Â  Â  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00</div><div>Slab  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â 4830.68  Â  Â  Â  Â 2254.55  Â  Â  Â  Â 7085.22</div><div>SReclaimable  Â  Â  Â  Â  Â  Â 2061.05  Â  Â  Â  Â  921.72  Â  Â  Â  Â 2982.77</div><div>SUnreclaim  Â  Â  Â  Â  Â  Â  Â 2769.62  Â  Â  Â  Â 1332.83  Â  Â  Â  Â 4102.45</div><div>AnonHugePages  Â  Â  Â  Â  Â  Â  Â 4.00  Â  Â  Â  Â  Â  2.00  Â  Â  Â  Â  Â  6.00</div><div>HugePages_Total  Â  Â  Â  Â  Â  Â 0.00  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00</div><div>HugePages_Free  Â  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00</div><div>HugePages_Surp  Â  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00  Â  Â  Â  Â  Â  0.00</div></div><div><br></div><div><br></div><div>Thanks,</div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Aug 6, 2018 at 6:33 PM Igor A. Ippolitov <<a href="mailto:iippolitov@nginx.com">iippolitov@nginx.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <div class="m_367398162948788752moz-cite-prefix">Anoop,<br>
      <br>
      I suppose, most of your 10k servers are very similar, right?<br>
      Please, post top level configuration and a typical server{},
      please.<br>
      <br>
      Also, how do you reload configuration? With 'service nginx reload'
      or may be other commands?<br>
      <br>
      It looks like you have a lot of fragmented memory and only 4gb
      free in the second numa node.<br>
      So, I'd say this is OK that you are getting errors from allocating
      a 16k stripes.<br>
      <br>
      Could you please post numastat -m output additionally. Just to
      make sure you have half of the memory for the second CPU.<br>
      And we'll have a look if memory utilization may be optimized based
      on your configuration.<br>
      <br>
      Regards,<br>
      Igor.<br>
      <br>
      On 04.08.2018 07:54, Anoop Alias wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">Hi Igor,
        <div><br>
        </div>
        <div>Setting vm.max_map_count to 20x the normal value did not
          help</div>
        <div><br>
        </div>
        <div>The issue happens on a group of servers and among the
          group, it shows up only in servers which have ~10k  server{}
          blocks</div>
        <div><br>
        </div>
        <div>On servers that have lower number of server{} blocks , the
          ENOMEM issue is not there</div>
        <div><br>
        </div>
        <div>Also, I can find that the RAM usage of the Nginx process is
          directly proportional to the number of server {} blocks</div>
        <div><br>
        </div>
        <div>For example on a server having the problem</div>
        <div><br>
        </div>
        <div>
          <div># ps_mem| head -1 && ps_mem |grep nginx</div>
          <div> Private  +  Â Shared  =  RAM used  Â  Â  Â Program </div>
          <div>  1.0 GiB +  Â 2.8 GiB =  Â 3.8 GiB  Â  Â  Â nginx (3)</div>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div>That is for a single worker process with 4 threads in
          thread_pool </div>
        <div>
          <div># pstree|grep nginx</div>
          <div>  Â  Â  Â  |-nginx-+-nginx---4*[{nginx}]</div>
          <div>  Â  Â  Â  |  Â  Â  Â `-nginx</div>
        </div>
        <div><br>
        </div>
        <div>Whatever config change I try the memory usage seem to
          mostly depend on the number of server contexts defined</div>
        <div><br>
        </div>
        <div>Now the issue mostly happen in nginx reload ,when one more
          worker process will be active in shutting down mode </div>
        <div><br>
        </div>
        <div>I believe the memalign error is thrown by the worker being
          shutdown, this is because the sites work after the error and
          also the pid mentioned in the error would have gone when I
          check ps</div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div>
          <div># pmap 948965|grep 16K</div>
          <div>00007f2923ff2000  Â  Â 16K r-x-- ngx_http_redis2_module.so</div>
          <div>00007f2924fd7000  Â  Â 16K r---- <a href="http://libc-2.17.so" target="_blank">libc-2.17.so</a></div>
          <div>00007f2925431000  Â  Â 16K rw---  Â [ anon ]</div>
          <div>00007f292584a000  Â  Â 16K rw---  Â [ anon ]</div>
        </div>
        <div><br>
        </div>
        <div>
          <div>Aug  4 05:50:00 b kernel: SysRq : Show Memory</div>
          <div>Aug  4 05:50:00 b kernel: Mem-Info:</div>
          <div>Aug  4 05:50:00 b kernel: active_anon:7757394
            inactive_anon:1021319 isolated_anon:0#012
            active_<a class="m_367398162948788752moz-txt-link-freetext">file:3733324</a> inactive_<a class="m_367398162948788752moz-txt-link-freetext">file:2136476</a>
            isolated_<a class="m_367398162948788752moz-txt-link-freetext">file:0#012</a> unevictable:0 dirty:1766 writeback:6
            wbtmp:0 unstable:0#012 slab_reclaimable:2003687
            slab_unreclaimable:901391#012 mapped:316734 shmem:2381810
            pagetables:63163 bounce:0#012 free:4851283 free_pcp:11332
            free_cma:0</div>
          <div>Aug  4 05:50:00 bravo kernel: Node 0 DMA free:15888kB
            min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB
            active_<a class="m_367398162948788752moz-txt-link-freetext">file:0kB</a> inactive_<a class="m_367398162948788752moz-txt-link-freetext">file:0kB</a> unevictable:0kB
            isolated(anon):0kB isolated(file):0kB present:15972kB
            managed:15888kB mlocked:0kB dirty:0kB writeback:0kB
            mapped:0kB shmem:0kB slab_reclaimable:0kB
            slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB
            unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB
            free_cma:0kB writeback_tmp:0kB pages_scanned:0
            all_unreclaimable? yes</div>
          <div>Aug  4 05:50:00 b kernel: lowmem_reserve[]: 0 1679 64139
            64139</div>
        </div>
        <div><br>
        </div>
        <div>
          <div># cat /proc/buddyinfo</div>
          <div>Node 0, zone  Â  Â  DMA  Â  Â  0  Â  Â  0  Â  Â  1  Â  Â  0  Â  Â  2 
            Â  Â  1  Â  Â  1  Â  Â  0  Â  Â  1  Â  Â  1  Â  Â  3 </div>
          <div>Node 0, zone  Â  DMA32  Â 5284  Â 6753  Â 6677  Â 1083  Â  410 
            Â  Â 59  Â  Â  1  Â  Â  0  Â  Â  0  Â  Â  0  Â  Â  0 </div>
          <div>Node 0, zone  Â Normal 500327 638958 406737  14690  Â  872 
            Â  106  Â  Â 11  Â  Â  0  Â  Â  0  Â  Â  0  Â  Â  0 </div>
          <div>Node 1, zone  Â Normal 584840 291640  Â  188  Â  Â  0  Â  Â  0 
            Â  Â  0  Â  Â  0  Â  Â  0  Â  Â  0  Â  Â  0  Â  Â  0 </div>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div>The only correlation I see in having the error is the
          number of server {} blocks (close to 10k) which then makes the
          nginx process consume ~ 4GB of mem with a single worker
          process and then a reload is done</div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr">On Thu, Aug 2, 2018 at 6:02 PM Igor A. Ippolitov
          <<a href="mailto:iippolitov@nginx.com" target="_blank">iippolitov@nginx.com</a>> wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div text="#000000" bgcolor="#FFFFFF">
            <div class="m_367398162948788752m_-1786270149800573001moz-cite-prefix">Anoop,<br>
              <br>
              There are two guesses: either mmap allocations limit is
              hit or memory is  way too fragmented.<br>
              Could you please track amount of mapped regions for a
              worker with pmap and amount of 16k areas in Normal zones
              (it is the third number)?<br>
              <br>
              You can also set vm.max_map_count to a higher number (like
              20 times higher than default) and look if the error is
              gone.<br>
              <br>
              Please, let me know if increasing vm.max_map_count helps
              you.<br>
              <br>
              On 02.08.2018 13:06, Anoop Alias wrote:<br>
            </div>
            <blockquote type="cite">
              <div dir="ltr">Hi Igor,
                <div><br>
                </div>
                <div>The error happens randomly </div>
                <div>
                  <div><br>
                  </div>
                  <div>
                    <div>2018/08/02 06:52:42 [emerg] 874514#874514:
                      posix_memalign(16, 16384) failed (12: Cannot
                      allocate memory)</div>
                    <div>2018/08/02 09:42:53 [emerg] 872996#872996:
                      posix_memalign(16, 16384) failed (12: Cannot
                      allocate memory)</div>
                    <div>2018/08/02 10:16:14 [emerg] 877611#877611:
                      posix_memalign(16, 16384) failed (12: Cannot
                      allocate memory)</div>
                    <div>2018/08/02 10:16:48 [emerg] 879410#879410:
                      posix_memalign(16, 16384) failed (12: Cannot
                      allocate memory)</div>
                    <div>2018/08/02 10:17:55 [emerg] 876563#876563:
                      posix_memalign(16, 16384) failed (12: Cannot
                      allocate memory)</div>
                    <div>2018/08/02 10:20:21 [emerg] 879263#879263:
                      posix_memalign(16, 16384) failed (12: Cannot
                      allocate memory)</div>
                    <div>2018/08/02 10:20:51 [emerg] 878991#878991:
                      posix_memalign(16, 16384) failed (12: Cannot
                      allocate memory)</div>
                  </div>
                  <div><br>
                  </div>
                  <div>
                    <div># date</div>
                    <div>Thu Aug  2 10:58:48 BST 2018</div>
                  </div>
                  <div><br>
                  </div>
                  <div>------------------------------------------</div>
                  <div>
                    <div># cat /proc/buddyinfo</div>
                    <div>Node 0, zone  Â  Â  DMA  Â  Â  0  Â  Â  0  Â  Â  1  Â  Â 
                      0  Â  Â  2  Â  Â  1  Â  Â  1  Â  Â  0  Â  Â  1  Â  Â  1  Â  Â 
                      3 </div>
                    <div>Node 0, zone  Â  DMA32  11722  11057  Â 4663 
                      Â 1647  Â  609  Â  Â 72  Â  Â 10  Â  Â  7  Â  Â  1  Â  Â  0  Â 
                      Â  0 </div>
                    <div>Node 0, zone  Â Normal 755026 710760 398136 
                      21462  Â 1114  Â  Â 18  Â  Â  1  Â  Â  0  Â  Â  0  Â  Â  0  Â 
                      Â  0 </div>
                    <div>Node 1, zone  Â Normal 341295 801810 179604  Â 
                      256  Â  Â  0  Â  Â  0  Â  Â  0  Â  Â  0  Â  Â  0  Â  Â  0  Â  Â 
                      0 </div>
                  </div>
                  <div>-----------------------------------------</div>
                  <div><br>
                  </div>
                  <div><br>
                  </div>
                  <div>
                    <div>slabinfo - version: 2.1</div>
                    <div># name  Â  Â  Â  Â  Â  <active_objs>
                      <num_objs> <objsize>
                      <objperslab> <pagesperslab> : tunables
                      <limit> <batchcount>
                      <sharedfactor> : slabdata
                      <active_slabs> <num_slabs>
                      <sharedavail></div>
                    <div>SCTPv6  Â  Â  Â  Â  Â  Â  Â  21  Â  Â 21  Â 1536  Â 21  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  1  Â  Â 
                      1  Â  Â  0</div>
                    <div>SCTP  Â  Â  Â  Â  Â  Â  Â  Â  Â 0  Â  Â  0  Â 1408  Â 23  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>kcopyd_job  Â  Â  Â  Â  Â  Â 0  Â  Â  0  Â 3312  Â  9  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dm_uevent  Â  Â  Â  Â  Â  Â  0  Â  Â  0  Â 2608  Â 12  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>nf_conntrack_ffffffff81acbb00  14054  14892  Â 
                      320  Â 51  Â  4 : tunables  Â  0  Â  0  Â  0 :
                      slabdata  Â  292  Â  292  Â  Â  0</div>
                    <div>lvp_cache  Â  Â  Â  Â  Â  Â 36  Â  Â 36  Â  224  Â 36  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  1  Â  Â 
                      1  Â  Â  0</div>
                    <div>lve_struct  Â  Â  Â  Â  4140  Â 4140  Â  352  Â 46  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 90  Â 
                      Â 90  Â  Â  0</div>
                    <div>fat_inode_cache  Â  Â  Â  0  Â  Â  0  Â  744  Â 44  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>fat_cache  Â  Â  Â  Â  Â  Â  0  Â  Â  0  Â  Â 40  102  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>isofs_inode_cache  Â  Â  0  Â  Â  0  Â  664  Â 49  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>ext4_inode_cache  Â  Â  30  Â  Â 30  Â 1088  Â 30  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  1  Â  Â 
                      1  Â  Â  0</div>
                    <div>ext4_xattr  Â  Â  Â  Â  Â  Â 0  Â  Â  0  Â  Â 88  Â 46  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>ext4_free_data  Â  Â  Â  Â 0  Â  Â  0  Â  Â 64  Â 64  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>ext4_allocation_context  Â  Â 32  Â  Â 32  Â  128 
                      Â 32  Â  1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 
                      1  Â  Â  1  Â  Â  0</div>
                    <div>ext4_io_end  Â  Â  Â  Â  Â  0  Â  Â  0  Â  Â 72  Â 56  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>ext4_extent_status  Â  102  Â  102  Â  Â 40  102  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  1  Â  Â 
                      1  Â  Â  0</div>
                    <div>jbd2_journal_handle  Â  Â  0  Â  Â  0  Â  Â 48  Â 85 
                      Â  1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â 
                      Â  0  Â  Â  0</div>
                    <div>jbd2_journal_head  Â  Â  0  Â  Â  0  Â  112  Â 36  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>jbd2_revoke_table_s  Â  256  Â  256  Â  Â 16  256 
                      Â  1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  1  Â 
                      Â  1  Â  Â  0</div>
                    <div>jbd2_revoke_record_s  Â  Â  0  Â  Â  0  Â  Â 32  128 
                      Â  1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â 
                      Â  0  Â  Â  0</div>
                    <div>kvm_async_pf  Â  Â  Â  Â  Â 0  Â  Â  0  Â  136  Â 30  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>kvm_vcpu  Â  Â  Â  Â  Â  Â  Â 0  Â  Â  0  18560  Â  1  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>xfs_dqtrx  Â  Â  Â  Â  Â  992  Â  992  Â  528  Â 31  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>xfs_dquot  Â  Â  Â  Â  Â 3264  Â 3264  Â  472  Â 34  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 96  Â 
                      Â 96  Â  Â  0</div>
                    <div>xfs_ili  Â  Â  Â  Â  Â 4342175 4774399  Â  152  Â 53 
                      Â  2 : tunables  Â  0  Â  0  Â  0 : slabdata  90083 
                      90083  Â  Â  0</div>
                    <div>xfs_inode  Â  Â  Â  Â 4915588 5486076  Â 1088  Â 30 
                      Â  8 : tunables  Â  0  Â  0  Â  0 : slabdata 182871
                      182871  Â  Â  0</div>
                    <div>xfs_efd_item  Â  Â  Â  2680  Â 2760  Â  400  Â 40  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 69  Â 
                      Â 69  Â  Â  0</div>
                    <div>xfs_da_state  Â  Â  Â  1088  Â 1088  Â  480  Â 34  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>xfs_btree_cur  Â  Â  Â 1248  Â 1248  Â  208  Â 39  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>xfs_log_ticket  Â  Â 14874  15048  Â  184  Â 44  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  342  Â 
                      342  Â  Â  0</div>
                    <div>xfs_ioend  Â  Â  Â  Â  12909  13104  Â  104  Â 39  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  336  Â 
                      336  Â  Â  0</div>
                    <div>scsi_cmd_cache  Â  Â  5400  Â 5652  Â  448  Â 36  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  157  Â 
                      157  Â  Â  0</div>
                    <div>ve_struct  Â  Â  Â  Â  Â  Â  0  Â  Â  0  Â  848  Â 38  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>ip6_dst_cache  Â  Â  Â 1152  Â 1152  Â  448  Â 36  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>RAWv6  Â  Â  Â  Â  Â  Â  Â  910  Â  910  Â 1216  Â 26  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 35  Â 
                      Â 35  Â  Â  0</div>
                    <div>UDPLITEv6  Â  Â  Â  Â  Â  Â  0  Â  Â  0  Â 1216  Â 26  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                  </div>
                  <div>
                    <div>UDPv6  Â  Â  Â  Â  Â  Â  Â  832  Â  832  Â 1216  Â 26  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>tw_sock_TCPv6  Â  Â  Â 1152  Â 1376  Â  256  Â 32  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 43  Â 
                      Â 43  Â  Â  0</div>
                    <div>TCPv6  Â  Â  Â  Â  Â  Â  Â  510  Â  510  Â 2176  Â 15  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 34  Â 
                      Â 34  Â  Â  0</div>
                    <div>cfq_queue  Â  Â  Â  Â  Â 3698  Â 5145  Â  232  Â 35  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  147  Â 
                      147  Â  Â  0</div>
                    <div>bsg_cmd  Â  Â  Â  Â  Â  Â  Â  0  Â  Â  0  Â  312  Â 52  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>mqueue_inode_cache  Â  136  Â  136  Â  960  Â 34  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  4  Â  Â 
                      4  Â  Â  0</div>
                    <div>hugetlbfs_inode_cache  Â 1632  Â 1632  Â  632 
                      Â 51  Â  8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 
                      Â 32  Â  Â 32  Â  Â  0</div>
                    <div>configfs_dir_cache  Â 1472  Â 1472  Â  Â 88  Â 46  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>dquot  Â  Â  Â  Â  Â  Â  Â  Â  0  Â  Â  0  Â  256  Â 32  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>userfaultfd_ctx_cache  Â  Â 32  Â  Â 32  Â  128 
                      Â 32  Â  1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 
                      1  Â  Â  1  Â  Â  0</div>
                    <div>fanotify_event_info  Â 2336  Â 2336  Â  Â 56  Â 73 
                      Â  1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>dio  Â  Â  Â  Â  Â  Â  Â  Â 6171  Â 6222  Â  640  Â 51  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  122  Â 
                      122  Â  Â  0</div>
                    <div>pid_namespace  Â  Â  Â  Â 42  Â  Â 42  Â 2192  Â 14  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  3  Â  Â 
                      3  Â  Â  0</div>
                    <div>posix_timers_cache  Â 1056  Â 1056  Â  248  Â 33  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>UDP-Lite  Â  Â  Â  Â  Â  Â  Â 0  Â  Â  0  Â 1088  Â 30  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>flow_cache  Â  Â  Â  Â  2268  Â 2296  Â  144  Â 28  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 82  Â 
                      Â 82  Â  Â  0</div>
                    <div>xfrm_dst_cache  Â  Â  Â 896  Â  896  Â  576  Â 28  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>ip_fib_alias  Â  Â  Â  2720  Â 2720  Â  Â 48  Â 85  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>RAW  Â  Â  Â  Â  Â  Â  Â  Â 3977  Â 4224  Â 1024  Â 32  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  132  Â 
                      132  Â  Â  0</div>
                    <div>UDP  Â  Â  Â  Â  Â  Â  Â  Â 4110  Â 4110  Â 1088  Â 30  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  137  Â 
                      137  Â  Â  0</div>
                    <div>tw_sock_TCP  Â  Â  Â  Â 4756  Â 5216  Â  256  Â 32  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  163  Â 
                      163  Â  Â  0</div>
                    <div>TCP  Â  Â  Â  Â  Â  Â  Â  Â 2705  Â 2768  Â 1984  Â 16  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  173  Â 
                      173  Â  Â  0</div>
                    <div>scsi_data_buffer  Â  5440  Â 5440  Â  Â 24  170  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>blkdev_queue  Â  Â  Â  Â 154  Â  154  Â 2208  Â 14  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 11  Â 
                      Â 11  Â  Â  0</div>
                    <div>blkdev_requests  Â 4397688 4405884  Â  384  Â 42 
                      Â  4 : tunables  Â  0  Â  0  Â  0 : slabdata 104902
                      104902  Â  Â  0</div>
                    <div>blkdev_ioc  Â  Â  Â  Â 11232  11232  Â  112  Â 36  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  312  Â 
                      312  Â  Â  0</div>
                    <div>user_namespace  Â  Â  Â  Â 0  Â  Â  0  Â 1304  Â 25  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>sock_inode_cache  Â 12282  12282  Â  704  Â 46  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  267  Â 
                      267  Â  Â  0</div>
                    <div>file_lock_cache  Â  20056  20960  Â  200  Â 40  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  524  Â 
                      524  Â  Â  0</div>
                    <div>net_namespace  Â  Â  Â  Â  6  Â  Â  6  Â 5056  Â  6  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  1  Â  Â 
                      1  Â  Â  0</div>
                    <div>shmem_inode_cache  16970  18952  Â  712  Â 46  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  412  Â 
                      412  Â  Â  0</div>
                    <div>Acpi-ParseExt  Â  Â  39491  40432  Â  Â 72  Â 56  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  722  Â 
                      722  Â  Â  0</div>
                    <div>Acpi-State  Â  Â  Â  Â  1683  Â 1683  Â  Â 80  Â 51  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 33  Â 
                      Â 33  Â  Â  0</div>
                    <div>Acpi-Namespace  Â  Â 11424  11424  Â  Â 40  102  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  112  Â 
                      112  Â  Â  0</div>
                    <div>task_delay_info  Â  15336  15336  Â  112  Â 36  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  426  Â 
                      426  Â  Â  0</div>
                    <div>taskstats  Â  Â  Â  Â  Â 1568  Â 1568  Â  328  Â 49  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 32  Â 
                      Â 32  Â  Â  0</div>
                    <div>proc_inode_cache  169897 190608  Â  680  Â 48  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 3971 
                      Â 3971  Â  Â  0</div>
                    <div>sigqueue  Â  Â  Â  Â  Â  2208  Â 2208  Â  168  Â 48  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 46  Â 
                      Â 46  Â  Â  0</div>
                    <div>bdev_cache  Â  Â  Â  Â  Â 792  Â  792  Â  896  Â 36  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 22  Â 
                      Â 22  Â  Â  0</div>
                  </div>
                  <div>
                    <div>sysfs_dir_cache  Â  74698  74698  Â  120  Â 34  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 2197 
                      Â 2197  Â  Â  0</div>
                    <div>mnt_cache  Â  Â  Â  Â 163197 163424  Â  256  Â 32  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 5107 
                      Â 5107  Â  Â  0</div>
                    <div>filp  Â  Â  Â  Â  Â  Â  Â 64607  97257  Â  320  Â 51  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 1907 
                      Â 1907  Â  Â  0</div>
                    <div>inode_cache  Â  Â  Â 370744 370947  Â  616  Â 53  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 6999 
                      Â 6999  Â  Â  0</div>
                    <div>dentry  Â  Â  Â  Â  Â  1316262 2139228  Â  192  Â 42 
                      Â  2 : tunables  Â  0  Â  0  Â  0 : slabdata  50934 
                      50934  Â  Â  0</div>
                    <div>iint_cache  Â  Â  Â  Â  Â  Â 0  Â  Â  0  Â  Â 80  Â 51  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>buffer_head  Â  Â  Â 1441470 2890290  Â  104  Â 39 
                      Â  1 : tunables  Â  0  Â  0  Â  0 : slabdata  74110 
                      74110  Â  Â  0</div>
                    <div>vm_area_struct  Â  194998 196840  Â  216  Â 37  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 5320 
                      Â 5320  Â  Â  0</div>
                    <div>mm_struct  Â  Â  Â  Â  Â 2679  Â 2760  Â 1600  Â 20  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  138  Â 
                      138  Â  Â  0</div>
                    <div>files_cache  Â  Â  Â  Â 8680  Â 8925  Â  640  Â 51  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  175  Â 
                      175  Â  Â  0</div>
                    <div>signal_cache  Â  Â  Â  3691  Â 3780  Â 1152  Â 28  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  135  Â 
                      135  Â  Â  0</div>
                    <div>sighand_cache  Â  Â  Â 1950  Â 2160  Â 2112  Â 15  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  144  Â 
                      144  Â  Â  0</div>
                    <div>task_xstate  Â  Â  Â  Â 8070  Â 8658  Â  832  Â 39  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  222  Â 
                      222  Â  Â  0</div>
                    <div>task_struct  Â  Â  Â  Â 1913  Â 2088  Â 4080  Â  8  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  261  Â 
                      261  Â  Â  0</div>
                    <div>cred_jar  Â  Â  Â  Â  Â 31699  33936  Â  192  Â 42  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  808  Â 
                      808  Â  Â  0</div>
                    <div>anon_vma_chain  Â  164026 168704  Â  Â 64  Â 64  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 2636 
                      Â 2636  Â  Â  0</div>
                    <div>anon_vma  Â  Â  Â  Â  Â 84104  84594  Â  Â 88  Â 46  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 1839 
                      Â 1839  Â  Â  0</div>
                    <div>pid  Â  Â  Â  Â  Â  Â  Â  11127  12576  Â  128  Â 32  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  393  Â 
                      393  Â  Â  0</div>
                    <div>shared_policy_node  Â 9350  Â 9350  Â  Â 48  Â 85  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  110  Â 
                      110  Â  Â  0</div>
                    <div>numa_policy  Â  Â  Â  Â  Â 62  Â  Â 62  Â  264  Â 31  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  2  Â  Â 
                      2  Â  Â  0</div>
                    <div>radix_tree_node  Â 771778 1194312  Â  584  Â 28  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  42654 
                      42654  Â  Â  0</div>
                    <div>idr_layer_cache  Â  Â 2538  Â 2565  Â 2112  Â 15  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  171  Â 
                      171  Â  Â  0</div>
                    <div>dma-kmalloc-8192  Â  Â  Â 0  Â  Â  0  Â 8192  Â  4  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-4096  Â  Â  Â 0  Â  Â  0  Â 4096  Â  8  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-2048  Â  Â  Â 0  Â  Â  0  Â 2048  Â 16  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-1024  Â  Â  Â 0  Â  Â  0  Â 1024  Â 32  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-512  Â  Â  Â  0  Â  Â  0  Â  512  Â 32  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-256  Â  Â  Â  0  Â  Â  0  Â  256  Â 32  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-128  Â  Â  Â  0  Â  Â  0  Â  128  Â 32  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-64  Â  Â  Â  Â 0  Â  Â  0  Â  Â 64  Â 64  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-32  Â  Â  Â  Â 0  Â  Â  0  Â  Â 32  128  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-16  Â  Â  Â  Â 0  Â  Â  0  Â  Â 16  256  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-8  Â  Â  Â  Â  0  Â  Â  0  Â  Â  8  512  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-192  Â  Â  Â  0  Â  Â  0  Â  192  Â 42  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>dma-kmalloc-96  Â  Â  Â  Â 0  Â  Â  0  Â  Â 96  Â 42  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â  0  Â  Â 
                      0  Â  Â  0</div>
                    <div>kmalloc-8192  Â  Â  Â  Â 385  Â  388  Â 8192  Â  4  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 97  Â 
                      Â 97  Â  Â  0</div>
                    <div>kmalloc-4096  Â  Â  Â  9296  10088  Â 4096  Â  8  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 1261 
                      Â 1261  Â  Â  0</div>
                    <div>kmalloc-2048  Â  Â  Â 65061 133536  Â 2048  Â 16  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 8346 
                      Â 8346  Â  Â  0</div>
                    <div>kmalloc-1024  Â  Â  Â 11987  21120  Â 1024  Â 32  Â 
                      8 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  660  Â 
                      660  Â  Â  0</div>
                  </div>
                  <div>
                    <div>kmalloc-512  Â  Â  Â 107510 187072  Â  512  Â 32  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 5846 
                      Â 5846  Â  Â  0</div>
                    <div>kmalloc-256  Â  Â  Â 160498 199104  Â  256  Â 32  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 6222 
                      Â 6222  Â  Â  0</div>
                    <div>kmalloc-192  Â  Â  Â 144975 237426  Â  192  Â 42  Â 
                      2 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 5653 
                      Â 5653  Â  Â  0</div>
                    <div>kmalloc-128  Â  Â  Â  36799 108096  Â  128  Â 32  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 3378 
                      Â 3378  Â  Â  0</div>
                    <div>kmalloc-96  Â  Â  Â  Â 99510 238896  Â  Â 96  Â 42  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â 5688 
                      Â 5688  Â  Â  0</div>
                    <div>kmalloc-64  Â  Â  Â  7978152 8593280  Â  Â 64  Â 64 
                      Â  1 : tunables  Â  0  Â  0  Â  0 : slabdata 134270
                      134270  Â  Â  0</div>
                    <div>kmalloc-32  Â  Â  Â  2939882 3089664  Â  Â 32  128 
                      Â  1 : tunables  Â  0  Â  0  Â  0 : slabdata  24138 
                      24138  Â  Â  0</div>
                    <div>kmalloc-16  Â  Â  Â  172057 172288  Â  Â 16  256  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  673  Â 
                      673  Â  Â  0</div>
                    <div>kmalloc-8  Â  Â  Â  Â 109568 109568  Â  Â  8  512  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  214  Â 
                      214  Â  Â  0</div>
                    <div>kmem_cache_node  Â  Â  893  Â  896  Â  Â 64  Â 64  Â 
                      1 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 14  Â 
                      Â 14  Â  Â  0</div>
                    <div>kmem_cache  Â  Â  Â  Â  Â 612  Â  612  Â  320  Â 51  Â 
                      4 : tunables  Â  0  Â  0  Â  0 : slabdata  Â  Â 12  Â 
                      Â 12  Â  Â  0</div>
                  </div>
                  <div><br>
                  </div>
                  <div>-------------------------------------------------</div>
                  <div><br>
                  </div>
                  <div><br>
                  </div>
                  <div>
                    <div># uname -r</div>
                    <div>3.10.0-714.10.2.lve1.5.17.1.el7.x86_64</div>
                  </div>
                  <div><br>
                  </div>
                  <div>--------------------------------------------------------</div>
                  <div><br>
                  </div>
                  <div>Core part of glances </div>
                  <div><a href="http://i.imgur.com/La5JbQn.png" target="_blank">http://i.imgur.com/La5JbQn.png</a><br>
                  </div>
                  <div>-----------------------------------------------------------</div>
                  <div><br>
                  </div>
                  <div>Thank you very much for looking into this</div>
                  <div><br>
                  </div>
                </div>
              </div>
              <br>
              <div class="gmail_quote">
                <div dir="ltr">On Thu, Aug 2, 2018 at 12:37 PM Igor A.
                  Ippolitov <<a href="mailto:iippolitov@nginx.com" target="_blank">iippolitov@nginx.com</a>>
                  wrote:<br>
                </div>
                <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                  <div text="#000000" bgcolor="#FFFFFF">
                    <div class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-cite-prefix">Anoop,<br>
                      <br>
                      I doubt this will be the solution, but may we have
                      a look at /proc/buddyinfo and /proc/slabinfo the
                      moment when nginx can't allocate memory?<br>
                      <br>
                      On 02.08.2018 08:15, Anoop Alias wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr">
                        <div>Hi Maxim,</div>
                        <div><br>
                        </div>
                        <div>I enabled debug and the memalign call is
                          happening on nginx reloads and the ENOMEM
                          happen sometimes on the reload(not on all
                          reloads)</div>
                        <div><br>
                        </div>
                        <div>
                          <div>2018/08/02 05:59:08 [notice]
                            872052#872052: signal process started</div>
                          <div>2018/08/02 05:59:23 [notice]
                            871570#871570: signal 1 (SIGHUP) received
                            from 872052, reconfiguring</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: wake up, sigio 0</div>
                          <div>2018/08/02 05:59:23 [notice]
                            871570#871570: reconfiguring</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            0000000002B0DA00:16384 @16  Â  Â  === > the
                            memalign call on reload</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: malloc: 00000000087924D0:4560</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            000000000E442E00:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: malloc: 0000000005650850:4096</div>
                          <div>20</div>
                        </div>
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          bind() xxxx:443 #71</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          bind() xxxx:443 #72</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          bind() xxxx:443 #73</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          bind() xxxx:443 #74</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          add cleanup: 000000005340D728</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000024D3260:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000517BAF10:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 0000000053854FC0:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 0000000053855FD0:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 0000000053856FE0:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 0000000053857FF0:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          posix_memalign: 0000000053859000:16384 @16</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 000000005385D010:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 000000005385E020:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 000000005385F030:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536CD160:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536CE170:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536CF180:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536D0190:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536D11A0:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536D21B0:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536D31C0:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536D41D0:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536D51E0:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536D61F0:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536D7200:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536D8210:4096</div>
                        <div>2018/08/02 05:48:49 [debug] 871275#871275:
                          malloc: 00000000536D9220:4096</div>
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                        <div>Infact there are lot of such calls during a
                          reload</div>
                        <div><br>
                        </div>
                        <div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA17ED00:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA1B0FF0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA1E12C0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA211590:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA243880:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA271B30:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA2A3E20:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA2D20D0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA3063E0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA334690:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA366980:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA396C50:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA3C8F40:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA3F9210:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA4294E0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA45B7D0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA489A80:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA4BBD70:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA4EA020:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA51E330:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA54C5E0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA57E8D0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA5AEBA0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA5DEE70:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA611160:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA641430:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA671700:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA6A29E0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA6D5CE0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA707FD0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA736280:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA768570:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA796820:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA7CAB30:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA7F8DE0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA82B0D0:16384 @16</div>
                          <div>2018/08/02 05:59:23 [debug]
                            871570#871570: posix_memalign:
                            00000000BA85B3A0:16384 @16</div>
                        </div>
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                        <div>What is perplexing is that the system has
                          enough free (available RAM)</div>
                        <div>#############</div>
                        <div>
                          <div># free -g</div>
                          <div>  Â  Â  Â  Â  Â  Â  total  Â  Â  Â  used  Â  Â  Â 
                            free  Â  Â  shared  buff/cache  Â available</div>
                          <div>Mem:  Â  Â  Â  Â  Â  125  Â  Â  Â  Â  54  Â  Â  Â  Â 
                            24  Â  Â  Â  Â  Â 8  Â  Â  Â  Â  46  Â  Â  Â  Â  58</div>
                          <div>Swap:  Â  Â  Â  Â  Â  Â 0  Â  Â  Â  Â  Â 0  Â  Â  Â  Â 
                            Â 0</div>
                        </div>
                        <div>#############</div>
                        <div><br>
                        </div>
                        <div>
                          <div># ulimit -a</div>
                          <div>core file size  Â  Â  Â  Â  (blocks, -c) 0</div>
                          <div>data seg size  Â  Â  Â  Â  Â (kbytes, -d)
                            unlimited</div>
                          <div>scheduling priority  Â  Â  Â  Â  Â  Â (-e) 0</div>
                          <div>file size  Â  Â  Â  Â  Â  Â  Â (blocks, -f)
                            unlimited</div>
                          <div>pending signals  Â  Â  Â  Â  Â  Â  Â  Â (-i)
                            514579</div>
                          <div>max locked memory  Â  Â  Â (kbytes, -l) 64</div>
                          <div>max memory size  Â  Â  Â  Â (kbytes, -m)
                            unlimited</div>
                          <div>open files  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  (-n) 1024</div>
                          <div>pipe size  Â  Â  Â  Â  Â  (512 bytes, -p) 8</div>
                          <div>POSIX message queues  Â  Â (bytes, -q)
                            819200</div>
                          <div>real-time priority  Â  Â  Â  Â  Â  Â  (-r) 0</div>
                          <div>stack size  Â  Â  Â  Â  Â  Â  (kbytes, -s) 8192</div>
                          <div>cpu time  Â  Â  Â  Â  Â  Â  Â (seconds, -t)
                            unlimited</div>
                          <div>max user processes  Â  Â  Â  Â  Â  Â  (-u)
                            514579</div>
                          <div>virtual memory  Â  Â  Â  Â  (kbytes, -v)
                            unlimited</div>
                          <div>file locks  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  (-x)
                            unlimited</div>
                        </div>
                        <div><br>
                        </div>
                        <div>#########################################</div>
                        <div><br>
                        </div>
                        <div>There is no other thing limiting memory
                          allocation </div>
                        <div><br>
                        </div>
                        <div>Any way to prevent this or probably
                          identify/prevent this</div>
                        <div><br>
                        </div>
                      </div>
                      <br>
                      <div class="gmail_quote">
                        <div dir="ltr">On Tue, Jul 31, 2018 at 7:08 PM
                          Maxim Dounin <<a href="mailto:mdounin@mdounin.ru" target="_blank">mdounin@mdounin.ru</a>>
                          wrote:<br>
                        </div>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello!<br>
                          <br>
                          On Tue, Jul 31, 2018 at 09:52:29AM +0530,
                          Anoop Alias wrote:<br>
                          <br>
                          > I am repeatedly seeing errors like<br>
                          > <br>
                          > ######################<br>
                          > 2018/07/31 03:46:33 [emerg]
                          2854560#2854560: posix_memalign(16, 16384)<br>
                          > failed (12: Cannot allocate memory)<br>
                          > 2018/07/31 03:54:09 [emerg]
                          2890190#2890190: posix_memalign(16, 16384)<br>
                          > failed (12: Cannot allocate memory)<br>
                          > 2018/07/31 04:08:36 [emerg]
                          2939230#2939230: posix_memalign(16, 16384)<br>
                          > failed (12: Cannot allocate memory)<br>
                          > 2018/07/31 04:24:48 [emerg]
                          2992650#2992650: posix_memalign(16, 16384)<br>
                          > failed (12: Cannot allocate memory)<br>
                          > 2018/07/31 04:42:09 [emerg]
                          3053092#3053092: posix_memalign(16, 16384)<br>
                          > failed (12: Cannot allocate memory)<br>
                          > 2018/07/31 04:42:17 [emerg]
                          3053335#3053335: posix_memalign(16, 16384)<br>
                          > failed (12: Cannot allocate memory)<br>
                          > 2018/07/31 04:42:28 [emerg]
                          3053937#3053937: posix_memalign(16, 16384)<br>
                          > failed (12: Cannot allocate memory)<br>
                          > 2018/07/31 04:47:54 [emerg]
                          3070638#3070638: posix_memalign(16, 16384)<br>
                          > failed (12: Cannot allocate memory)<br>
                          > ####################<br>
                          > <br>
                          > on a few servers<br>
                          > <br>
                          > The servers have enough memory free and
                          the swap usage is 0, yet somehow<br>
                          > the kernel denies the posix_memalign with
                          ENOMEM ( this is what I think is<br>
                          > happening!)<br>
                          > <br>
                          > The numbers requested are always 16, 16k
                          . This makes me suspicious<br>
                          > <br>
                          > I have no setting in nginx.conf that
                          reference a 16k<br>
                          > <br>
                          > Is there any chance of finding out what
                          requests this and why this is not<br>
                          > fulfilled<br>
                          <br>
                          There are at least some buffers which default
                          to 16k - for <br>
                          example, ssl_buffer_size (<a href="http://nginx.org/r/ssl_buffer_size" rel="noreferrer" target="_blank">http://nginx.org/r/ssl_buffer_size</a>).<br>
                          <br>
                          You may try debugging log to futher find out
                          where the particular <br>
                          allocation happens, see here for details:<br>
                          <br>
                          <a href="http://nginx.org/en/docs/debugging_log.html" rel="noreferrer" target="_blank">http://nginx.org/en/docs/debugging_log.html</a><br>
                          <br>
                          But I don't really think it worth the effort. 
                          The error is pretty <br>
                          clear, and it's better to focus on why these
                          allocations are <br>
                          denied.  Likely you are hitting some limit.<br>
                          <br>
                          -- <br>
                          Maxim Dounin<br>
                          <a href="http://mdounin.ru/" rel="noreferrer" target="_blank">http://mdounin.ru/</a><br>
_______________________________________________<br>
                          nginx mailing list<br>
                          <a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
                          <a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
                        </blockquote>
                      </div>
                      <br clear="all">
                      <div><br>
                      </div>
                      -- <br>
                      <div dir="ltr" class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672gmail_signature" data-smartmail="gmail_signature">
                        <div dir="ltr">
                          <div><b>Anoop P Alias</b> 
                            <div><br>
                            </div>
                          </div>
                        </div>
                      </div>
                      <br>
                      <fieldset class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672mimeAttachmentHeader"></fieldset>
                      <br>
                      <pre>_______________________________________________
nginx mailing list
<a class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>
<a class="m_367398162948788752m_-1786270149800573001m_-3921009750399893672moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
                    </blockquote>
                    <p><br>
                    </p>
                  </div>
                  _______________________________________________<br>
                  nginx mailing list<br>
                  <a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
                  <a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
              </div>
              <br clear="all">
              <div><br>
              </div>
              -- <br>
              <div dir="ltr" class="m_367398162948788752m_-1786270149800573001gmail_signature" data-smartmail="gmail_signature">
                <div dir="ltr">
                  <div><b>Anoop P Alias</b> 
                    <div><br>
                    </div>
                  </div>
                </div>
              </div>
              <br>
              <fieldset class="m_367398162948788752m_-1786270149800573001mimeAttachmentHeader"></fieldset>
              <br>
              <pre>_______________________________________________
nginx mailing list
<a class="m_367398162948788752m_-1786270149800573001moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>
<a class="m_367398162948788752m_-1786270149800573001moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
            </blockquote>
            <p><br>
            </p>
          </div>
          _______________________________________________<br>
          nginx mailing list<br>
          <a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
          <a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
      </div>
      <br clear="all">
      <div><br>
      </div>
      -- <br>
      <div dir="ltr" class="m_367398162948788752gmail_signature" data-smartmail="gmail_signature">
        <div dir="ltr">
          <div><b>Anoop P Alias</b> 
            <div><br>
            </div>
          </div>
        </div>
      </div>
      <br>
      <fieldset class="m_367398162948788752mimeAttachmentHeader"></fieldset>
      <br>
      <pre>_______________________________________________
nginx mailing list
<a class="m_367398162948788752moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>
<a class="m_367398162948788752moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
    </blockquote>
    <p><br>
    </p>
  </div>

_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><b>Anoop P Alias</b> <div><br></div></div></div></div>