<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Anoop,<br>
<br>
There are two guesses: either mmap allocations limit is hit or
memory is way too fragmented.<br>
Could you please track amount of mapped regions for a worker with
pmap and amount of 16k areas in Normal zones (it is the third
number)?<br>
<br>
You can also set vm.max_map_count to a higher number (like 20
times higher than default) and look if the error is gone.<br>
<br>
Please, let me know if increasing vm.max_map_count helps you.<br>
<br>
On 02.08.2018 13:06, Anoop Alias wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAO6TEX3e524Qq-uabw1KA=fu0AQV0CGrAtXJsMdSkPs1F4sdYw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<div dir="ltr">Hi Igor,
<div><br>
</div>
<div>The error happens randomly </div>
<div>
<div><br>
</div>
<div>
<div>2018/08/02 06:52:42 [emerg] 874514#874514:
posix_memalign(16, 16384) failed (12: Cannot allocate
memory)</div>
<div>2018/08/02 09:42:53 [emerg] 872996#872996:
posix_memalign(16, 16384) failed (12: Cannot allocate
memory)</div>
<div>2018/08/02 10:16:14 [emerg] 877611#877611:
posix_memalign(16, 16384) failed (12: Cannot allocate
memory)</div>
<div>2018/08/02 10:16:48 [emerg] 879410#879410:
posix_memalign(16, 16384) failed (12: Cannot allocate
memory)</div>
<div>2018/08/02 10:17:55 [emerg] 876563#876563:
posix_memalign(16, 16384) failed (12: Cannot allocate
memory)</div>
<div>2018/08/02 10:20:21 [emerg] 879263#879263:
posix_memalign(16, 16384) failed (12: Cannot allocate
memory)</div>
<div>2018/08/02 10:20:51 [emerg] 878991#878991:
posix_memalign(16, 16384) failed (12: Cannot allocate
memory)</div>
</div>
<div><br>
</div>
<div>
<div># date</div>
<div>Thu Aug 2 10:58:48 BST 2018</div>
</div>
<div><br>
</div>
<div>------------------------------------------</div>
<div>
<div># cat /proc/buddyinfo</div>
<div>Node 0, zone DMA 0 0 1 0
2 1 1 0 1 1 3 </div>
<div>Node 0, zone DMA32 11722 11057 4663 1647
609 72 10 7 1 0 0 </div>
<div>Node 0, zone Normal 755026 710760 398136 21462
1114 18 1 0 0 0 0 </div>
<div>Node 1, zone Normal 341295 801810 179604 256
0 0 0 0 0 0 0 </div>
</div>
<div>-----------------------------------------</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>slabinfo - version: 2.1</div>
<div># name <active_objs> <num_objs>
<objsize> <objperslab> <pagesperslab> :
tunables <limit> <batchcount>
<sharedfactor> : slabdata <active_slabs>
<num_slabs> <sharedavail></div>
<div>SCTPv6 21 21 1536 21 8 :
tunables 0 0 0 : slabdata 1 1 0</div>
<div>SCTP 0 0 1408 23 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>kcopyd_job 0 0 3312 9 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dm_uevent 0 0 2608 12 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>nf_conntrack_ffffffff81acbb00 14054 14892 320
51 4 : tunables 0 0 0 : slabdata 292
292 0</div>
<div>lvp_cache 36 36 224 36 2 :
tunables 0 0 0 : slabdata 1 1 0</div>
<div>lve_struct 4140 4140 352 46 4 :
tunables 0 0 0 : slabdata 90 90 0</div>
<div>fat_inode_cache 0 0 744 44 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>fat_cache 0 0 40 102 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>isofs_inode_cache 0 0 664 49 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>ext4_inode_cache 30 30 1088 30 8 :
tunables 0 0 0 : slabdata 1 1 0</div>
<div>ext4_xattr 0 0 88 46 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>ext4_free_data 0 0 64 64 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>ext4_allocation_context 32 32 128 32 1
: tunables 0 0 0 : slabdata 1 1 0</div>
<div>ext4_io_end 0 0 72 56 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>ext4_extent_status 102 102 40 102 1 :
tunables 0 0 0 : slabdata 1 1 0</div>
<div>jbd2_journal_handle 0 0 48 85 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>jbd2_journal_head 0 0 112 36 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>jbd2_revoke_table_s 256 256 16 256 1 :
tunables 0 0 0 : slabdata 1 1 0</div>
<div>jbd2_revoke_record_s 0 0 32 128 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>kvm_async_pf 0 0 136 30 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>kvm_vcpu 0 0 18560 1 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>xfs_dqtrx 992 992 528 31 4 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>xfs_dquot 3264 3264 472 34 4 :
tunables 0 0 0 : slabdata 96 96 0</div>
<div>xfs_ili 4342175 4774399 152 53 2 :
tunables 0 0 0 : slabdata 90083 90083 0</div>
<div>xfs_inode 4915588 5486076 1088 30 8 :
tunables 0 0 0 : slabdata 182871 182871 0</div>
<div>xfs_efd_item 2680 2760 400 40 4 :
tunables 0 0 0 : slabdata 69 69 0</div>
<div>xfs_da_state 1088 1088 480 34 4 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>xfs_btree_cur 1248 1248 208 39 2 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>xfs_log_ticket 14874 15048 184 44 2 :
tunables 0 0 0 : slabdata 342 342 0</div>
<div>xfs_ioend 12909 13104 104 39 1 :
tunables 0 0 0 : slabdata 336 336 0</div>
<div>scsi_cmd_cache 5400 5652 448 36 4 :
tunables 0 0 0 : slabdata 157 157 0</div>
<div>ve_struct 0 0 848 38 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>ip6_dst_cache 1152 1152 448 36 4 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>RAWv6 910 910 1216 26 8 :
tunables 0 0 0 : slabdata 35 35 0</div>
<div>UDPLITEv6 0 0 1216 26 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
</div>
<div>
<div>UDPv6 832 832 1216 26 8 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>tw_sock_TCPv6 1152 1376 256 32 2 :
tunables 0 0 0 : slabdata 43 43 0</div>
<div>TCPv6 510 510 2176 15 8 :
tunables 0 0 0 : slabdata 34 34 0</div>
<div>cfq_queue 3698 5145 232 35 2 :
tunables 0 0 0 : slabdata 147 147 0</div>
<div>bsg_cmd 0 0 312 52 4 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>mqueue_inode_cache 136 136 960 34 8 :
tunables 0 0 0 : slabdata 4 4 0</div>
<div>hugetlbfs_inode_cache 1632 1632 632 51 8 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>configfs_dir_cache 1472 1472 88 46 1 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>dquot 0 0 256 32 2 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>userfaultfd_ctx_cache 32 32 128 32 1 :
tunables 0 0 0 : slabdata 1 1 0</div>
<div>fanotify_event_info 2336 2336 56 73 1 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>dio 6171 6222 640 51 8 :
tunables 0 0 0 : slabdata 122 122 0</div>
<div>pid_namespace 42 42 2192 14 8 :
tunables 0 0 0 : slabdata 3 3 0</div>
<div>posix_timers_cache 1056 1056 248 33 2 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>UDP-Lite 0 0 1088 30 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>flow_cache 2268 2296 144 28 1 :
tunables 0 0 0 : slabdata 82 82 0</div>
<div>xfrm_dst_cache 896 896 576 28 4 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>ip_fib_alias 2720 2720 48 85 1 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>RAW 3977 4224 1024 32 8 :
tunables 0 0 0 : slabdata 132 132 0</div>
<div>UDP 4110 4110 1088 30 8 :
tunables 0 0 0 : slabdata 137 137 0</div>
<div>tw_sock_TCP 4756 5216 256 32 2 :
tunables 0 0 0 : slabdata 163 163 0</div>
<div>TCP 2705 2768 1984 16 8 :
tunables 0 0 0 : slabdata 173 173 0</div>
<div>scsi_data_buffer 5440 5440 24 170 1 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>blkdev_queue 154 154 2208 14 8 :
tunables 0 0 0 : slabdata 11 11 0</div>
<div>blkdev_requests 4397688 4405884 384 42 4 :
tunables 0 0 0 : slabdata 104902 104902 0</div>
<div>blkdev_ioc 11232 11232 112 36 1 :
tunables 0 0 0 : slabdata 312 312 0</div>
<div>user_namespace 0 0 1304 25 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>sock_inode_cache 12282 12282 704 46 8 :
tunables 0 0 0 : slabdata 267 267 0</div>
<div>file_lock_cache 20056 20960 200 40 2 :
tunables 0 0 0 : slabdata 524 524 0</div>
<div>net_namespace 6 6 5056 6 8 :
tunables 0 0 0 : slabdata 1 1 0</div>
<div>shmem_inode_cache 16970 18952 712 46 8 :
tunables 0 0 0 : slabdata 412 412 0</div>
<div>Acpi-ParseExt 39491 40432 72 56 1 :
tunables 0 0 0 : slabdata 722 722 0</div>
<div>Acpi-State 1683 1683 80 51 1 :
tunables 0 0 0 : slabdata 33 33 0</div>
<div>Acpi-Namespace 11424 11424 40 102 1 :
tunables 0 0 0 : slabdata 112 112 0</div>
<div>task_delay_info 15336 15336 112 36 1 :
tunables 0 0 0 : slabdata 426 426 0</div>
<div>taskstats 1568 1568 328 49 4 :
tunables 0 0 0 : slabdata 32 32 0</div>
<div>proc_inode_cache 169897 190608 680 48 8 :
tunables 0 0 0 : slabdata 3971 3971 0</div>
<div>sigqueue 2208 2208 168 48 2 :
tunables 0 0 0 : slabdata 46 46 0</div>
<div>bdev_cache 792 792 896 36 8 :
tunables 0 0 0 : slabdata 22 22 0</div>
</div>
<div>
<div>sysfs_dir_cache 74698 74698 120 34 1 :
tunables 0 0 0 : slabdata 2197 2197 0</div>
<div>mnt_cache 163197 163424 256 32 2 :
tunables 0 0 0 : slabdata 5107 5107 0</div>
<div>filp 64607 97257 320 51 4 :
tunables 0 0 0 : slabdata 1907 1907 0</div>
<div>inode_cache 370744 370947 616 53 8 :
tunables 0 0 0 : slabdata 6999 6999 0</div>
<div>dentry 1316262 2139228 192 42 2 :
tunables 0 0 0 : slabdata 50934 50934 0</div>
<div>iint_cache 0 0 80 51 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>buffer_head 1441470 2890290 104 39 1 :
tunables 0 0 0 : slabdata 74110 74110 0</div>
<div>vm_area_struct 194998 196840 216 37 2 :
tunables 0 0 0 : slabdata 5320 5320 0</div>
<div>mm_struct 2679 2760 1600 20 8 :
tunables 0 0 0 : slabdata 138 138 0</div>
<div>files_cache 8680 8925 640 51 8 :
tunables 0 0 0 : slabdata 175 175 0</div>
<div>signal_cache 3691 3780 1152 28 8 :
tunables 0 0 0 : slabdata 135 135 0</div>
<div>sighand_cache 1950 2160 2112 15 8 :
tunables 0 0 0 : slabdata 144 144 0</div>
<div>task_xstate 8070 8658 832 39 8 :
tunables 0 0 0 : slabdata 222 222 0</div>
<div>task_struct 1913 2088 4080 8 8 :
tunables 0 0 0 : slabdata 261 261 0</div>
<div>cred_jar 31699 33936 192 42 2 :
tunables 0 0 0 : slabdata 808 808 0</div>
<div>anon_vma_chain 164026 168704 64 64 1 :
tunables 0 0 0 : slabdata 2636 2636 0</div>
<div>anon_vma 84104 84594 88 46 1 :
tunables 0 0 0 : slabdata 1839 1839 0</div>
<div>pid 11127 12576 128 32 1 :
tunables 0 0 0 : slabdata 393 393 0</div>
<div>shared_policy_node 9350 9350 48 85 1 :
tunables 0 0 0 : slabdata 110 110 0</div>
<div>numa_policy 62 62 264 31 2 :
tunables 0 0 0 : slabdata 2 2 0</div>
<div>radix_tree_node 771778 1194312 584 28 4 :
tunables 0 0 0 : slabdata 42654 42654 0</div>
<div>idr_layer_cache 2538 2565 2112 15 8 :
tunables 0 0 0 : slabdata 171 171 0</div>
<div>dma-kmalloc-8192 0 0 8192 4 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-4096 0 0 4096 8 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-2048 0 0 2048 16 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-1024 0 0 1024 32 8 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-512 0 0 512 32 4 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-256 0 0 256 32 2 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-128 0 0 128 32 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-64 0 0 64 64 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-32 0 0 32 128 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-16 0 0 16 256 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-8 0 0 8 512 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-192 0 0 192 42 2 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>dma-kmalloc-96 0 0 96 42 1 :
tunables 0 0 0 : slabdata 0 0 0</div>
<div>kmalloc-8192 385 388 8192 4 8 :
tunables 0 0 0 : slabdata 97 97 0</div>
<div>kmalloc-4096 9296 10088 4096 8 8 :
tunables 0 0 0 : slabdata 1261 1261 0</div>
<div>kmalloc-2048 65061 133536 2048 16 8 :
tunables 0 0 0 : slabdata 8346 8346 0</div>
<div>kmalloc-1024 11987 21120 1024 32 8 :
tunables 0 0 0 : slabdata 660 660 0</div>
</div>
<div>
<div>kmalloc-512 107510 187072 512 32 4 :
tunables 0 0 0 : slabdata 5846 5846 0</div>
<div>kmalloc-256 160498 199104 256 32 2 :
tunables 0 0 0 : slabdata 6222 6222 0</div>
<div>kmalloc-192 144975 237426 192 42 2 :
tunables 0 0 0 : slabdata 5653 5653 0</div>
<div>kmalloc-128 36799 108096 128 32 1 :
tunables 0 0 0 : slabdata 3378 3378 0</div>
<div>kmalloc-96 99510 238896 96 42 1 :
tunables 0 0 0 : slabdata 5688 5688 0</div>
<div>kmalloc-64 7978152 8593280 64 64 1 :
tunables 0 0 0 : slabdata 134270 134270 0</div>
<div>kmalloc-32 2939882 3089664 32 128 1 :
tunables 0 0 0 : slabdata 24138 24138 0</div>
<div>kmalloc-16 172057 172288 16 256 1 :
tunables 0 0 0 : slabdata 673 673 0</div>
<div>kmalloc-8 109568 109568 8 512 1 :
tunables 0 0 0 : slabdata 214 214 0</div>
<div>kmem_cache_node 893 896 64 64 1 :
tunables 0 0 0 : slabdata 14 14 0</div>
<div>kmem_cache 612 612 320 51 4 :
tunables 0 0 0 : slabdata 12 12 0</div>
</div>
<div><br>
</div>
<div>-------------------------------------------------</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div># uname -r</div>
<div>3.10.0-714.10.2.lve1.5.17.1.el7.x86_64</div>
</div>
<div><br>
</div>
<div>--------------------------------------------------------</div>
<div><br>
</div>
<div>Core part of glances </div>
<div><a href="http://i.imgur.com/La5JbQn.png"
moz-do-not-send="true">http://i.imgur.com/La5JbQn.png</a><br>
</div>
<div>-----------------------------------------------------------</div>
<div><br>
</div>
<div>Thank you very much for looking into this</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Aug 2, 2018 at 12:37 PM Igor A. Ippolitov
<<a href="mailto:iippolitov@nginx.com"
moz-do-not-send="true">iippolitov@nginx.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_-3921009750399893672moz-cite-prefix">Anoop,<br>
<br>
I doubt this will be the solution, but may we have a look
at /proc/buddyinfo and /proc/slabinfo the moment when
nginx can't allocate memory?<br>
<br>
On 02.08.2018 08:15, Anoop Alias wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Maxim,</div>
<div><br>
</div>
<div>I enabled debug and the memalign call is happening
on nginx reloads and the ENOMEM happen sometimes on
the reload(not on all reloads)</div>
<div><br>
</div>
<div>
<div>2018/08/02 05:59:08 [notice] 872052#872052:
signal process started</div>
<div>2018/08/02 05:59:23 [notice] 871570#871570:
signal 1 (SIGHUP) received from 872052,
reconfiguring</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570: wake
up, sigio 0</div>
<div>2018/08/02 05:59:23 [notice] 871570#871570:
reconfiguring</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 0000000002B0DA00:16384 @16 ===
> the memalign call on reload</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
malloc: 00000000087924D0:4560</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 000000000E442E00:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
malloc: 0000000005650850:4096</div>
<div>20</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: bind()
xxxx:443 #71</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: bind()
xxxx:443 #72</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: bind()
xxxx:443 #73</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: bind()
xxxx:443 #74</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: add
cleanup: 000000005340D728</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000024D3260:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000517BAF10:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
0000000053854FC0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
0000000053855FD0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
0000000053856FE0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
0000000053857FF0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275:
posix_memalign: 0000000053859000:16384 @16</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
000000005385D010:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
000000005385E020:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
000000005385F030:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536CD160:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536CE170:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536CF180:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D0190:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D11A0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D21B0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D31C0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D41D0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D51E0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D61F0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D7200:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D8210:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D9220:4096</div>
<div><br>
</div>
<div><br>
</div>
<div>Infact there are lot of such calls during a reload</div>
<div><br>
</div>
<div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA17ED00:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA1B0FF0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA1E12C0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA211590:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA243880:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA271B30:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA2A3E20:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA2D20D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA3063E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA334690:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA366980:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA396C50:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA3C8F40:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA3F9210:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA4294E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA45B7D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA489A80:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA4BBD70:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA4EA020:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA51E330:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA54C5E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA57E8D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA5AEBA0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA5DEE70:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA611160:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA641430:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA671700:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA6A29E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA6D5CE0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA707FD0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA736280:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA768570:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA796820:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA7CAB30:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA7F8DE0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA82B0D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA85B3A0:16384 @16</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>What is perplexing is that the system has enough
free (available RAM)</div>
<div>#############</div>
<div>
<div># free -g</div>
<div> total used free
shared buff/cache available</div>
<div>Mem: 125 54 24
8 46 58</div>
<div>Swap: 0 0 0</div>
</div>
<div>#############</div>
<div><br>
</div>
<div>
<div># ulimit -a</div>
<div>core file size (blocks, -c) 0</div>
<div>data seg size (kbytes, -d) unlimited</div>
<div>scheduling priority (-e) 0</div>
<div>file size (blocks, -f) unlimited</div>
<div>pending signals (-i) 514579</div>
<div>max locked memory (kbytes, -l) 64</div>
<div>max memory size (kbytes, -m) unlimited</div>
<div>open files (-n) 1024</div>
<div>pipe size (512 bytes, -p) 8</div>
<div>POSIX message queues (bytes, -q) 819200</div>
<div>real-time priority (-r) 0</div>
<div>stack size (kbytes, -s) 8192</div>
<div>cpu time (seconds, -t) unlimited</div>
<div>max user processes (-u) 514579</div>
<div>virtual memory (kbytes, -v) unlimited</div>
<div>file locks (-x) unlimited</div>
</div>
<div><br>
</div>
<div>#########################################</div>
<div><br>
</div>
<div>There is no other thing limiting memory allocation </div>
<div><br>
</div>
<div>Any way to prevent this or probably
identify/prevent this</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Jul 31, 2018 at 7:08 PM Maxim
Dounin <<a href="mailto:mdounin@mdounin.ru"
target="_blank" moz-do-not-send="true">mdounin@mdounin.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hello!<br>
<br>
On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias
wrote:<br>
<br>
> I am repeatedly seeing errors like<br>
> <br>
> ######################<br>
> 2018/07/31 03:46:33 [emerg] 2854560#2854560:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 03:54:09 [emerg] 2890190#2890190:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:08:36 [emerg] 2939230#2939230:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:24:48 [emerg] 2992650#2992650:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:09 [emerg] 3053092#3053092:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:17 [emerg] 3053335#3053335:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:28 [emerg] 3053937#3053937:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:47:54 [emerg] 3070638#3070638:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> ####################<br>
> <br>
> on a few servers<br>
> <br>
> The servers have enough memory free and the swap
usage is 0, yet somehow<br>
> the kernel denies the posix_memalign with ENOMEM
( this is what I think is<br>
> happening!)<br>
> <br>
> The numbers requested are always 16, 16k . This
makes me suspicious<br>
> <br>
> I have no setting in nginx.conf that reference a
16k<br>
> <br>
> Is there any chance of finding out what requests
this and why this is not<br>
> fulfilled<br>
<br>
There are at least some buffers which default to 16k -
for <br>
example, ssl_buffer_size (<a
href="http://nginx.org/r/ssl_buffer_size"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://nginx.org/r/ssl_buffer_size</a>).<br>
<br>
You may try debugging log to futher find out where the
particular <br>
allocation happens, see here for details:<br>
<br>
<a href="http://nginx.org/en/docs/debugging_log.html"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://nginx.org/en/docs/debugging_log.html</a><br>
<br>
But I don't really think it worth the effort. The
error is pretty <br>
clear, and it's better to focus on why these
allocations are <br>
denied. Likely you are hitting some limit.<br>
<br>
-- <br>
Maxim Dounin<br>
<a href="http://mdounin.ru/" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://mdounin.ru/</a><br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank"
moz-do-not-send="true">nginx@nginx.org</a><br>
<a
href="http://mailman.nginx.org/mailman/listinfo/nginx"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="m_-3921009750399893672gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div><b>Anoop P Alias</b>
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="m_-3921009750399893672mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
nginx mailing list
<a class="m_-3921009750399893672moz-txt-link-abbreviated" href="mailto:nginx@nginx.org" target="_blank" moz-do-not-send="true">nginx@nginx.org</a>
<a class="m_-3921009750399893672moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank" moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
</blockquote>
<p><br>
</p>
</div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank"
moz-do-not-send="true">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div><b>Anoop P Alias</b>
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
nginx mailing list
<a class="moz-txt-link-abbreviated" href="mailto:nginx@nginx.org">nginx@nginx.org</a>
<a class="moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
</blockquote>
<p><br>
</p>
</body>
</html>