<div dir="ltr"><div>Hi Maxim,</div><div><br></div><div>I enabled debug and the memalign call is happening on nginx reloads and the ENOMEM happen sometimes on the reload(not on all reloads)</div><div><br></div><div><div>2018/08/02 05:59:08 [notice] 872052#872052: signal process started</div><div>2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) received from 872052, reconfiguring</div><div>2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0</div><div>2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 0000000002B0DA00:16384 @16 === > the memalign call on reload</div><div>2018/08/02 05:59:23 [debug] 871570#871570: malloc: 00000000087924D0:4560</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 000000000E442E00:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: malloc: 0000000005650850:4096</div><div>20</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div>2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71</div><div>2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72</div><div>2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73</div><div>2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74</div><div>2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: 000000005340D728</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000024D3260:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000517BAF10:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053854FC0:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053855FD0:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053856FE0:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053857FF0:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign: 0000000053859000:16384 @16</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385D010:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385E020:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385F030:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CD160:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CE170:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CF180:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D0190:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D11A0:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D21B0:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D31C0:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D41D0:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D51E0:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D61F0:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D7200:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D8210:4096</div><div>2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D9220:4096</div><div><br></div><div><br></div><div>Infact there are lot of such calls during a reload</div><div><br></div><div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA17ED00:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA1B0FF0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA1E12C0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA211590:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA243880:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA271B30:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA2A3E20:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA2D20D0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA3063E0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA334690:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA366980:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA396C50:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA3C8F40:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA3F9210:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA4294E0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA45B7D0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA489A80:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA4BBD70:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA4EA020:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA51E330:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA54C5E0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA57E8D0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA5AEBA0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA5DEE70:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA611160:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA641430:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA671700:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA6A29E0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA6D5CE0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA707FD0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA736280:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA768570:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA796820:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA7CAB30:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA7F8DE0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA82B0D0:16384 @16</div><div>2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA85B3A0:16384 @16</div></div><div><br></div><div><br></div><div><br></div><div>What is perplexing is that the system has enough free (available RAM)</div><div>#############</div><div><div># free -g</div><div> total used free shared buff/cache available</div><div>Mem: 125 54 24 8 46 58</div><div>Swap: 0 0 0</div></div><div>#############</div><div><br></div><div><div># ulimit -a</div><div>core file size (blocks, -c) 0</div><div>data seg size (kbytes, -d) unlimited</div><div>scheduling priority (-e) 0</div><div>file size (blocks, -f) unlimited</div><div>pending signals (-i) 514579</div><div>max locked memory (kbytes, -l) 64</div><div>max memory size (kbytes, -m) unlimited</div><div>open files (-n) 1024</div><div>pipe size (512 bytes, -p) 8</div><div>POSIX message queues (bytes, -q) 819200</div><div>real-time priority (-r) 0</div><div>stack size (kbytes, -s) 8192</div><div>cpu time (seconds, -t) unlimited</div><div>max user processes (-u) 514579</div><div>virtual memory (kbytes, -v) unlimited</div><div>file locks (-x) unlimited</div></div><div><br></div><div>#########################################</div><div><br></div><div>There is no other thing limiting memory allocation </div><div><br></div><div>Any way to prevent this or probably identify/prevent this</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin <<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello!<br>
<br>
On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote:<br>
<br>
> I am repeatedly seeing errors like<br>
> <br>
> ######################<br>
> 2018/07/31 03:46:33 [emerg] 2854560#2854560: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 03:54:09 [emerg] 2890190#2890190: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:08:36 [emerg] 2939230#2939230: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:24:48 [emerg] 2992650#2992650: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:09 [emerg] 3053092#3053092: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:17 [emerg] 3053335#3053335: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:28 [emerg] 3053937#3053937: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:47:54 [emerg] 3070638#3070638: posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> ####################<br>
> <br>
> on a few servers<br>
> <br>
> The servers have enough memory free and the swap usage is 0, yet somehow<br>
> the kernel denies the posix_memalign with ENOMEM ( this is what I think is<br>
> happening!)<br>
> <br>
> The numbers requested are always 16, 16k . This makes me suspicious<br>
> <br>
> I have no setting in nginx.conf that reference a 16k<br>
> <br>
> Is there any chance of finding out what requests this and why this is not<br>
> fulfilled<br>
<br>
There are at least some buffers which default to 16k - for <br>
example, ssl_buffer_size (<a href="http://nginx.org/r/ssl_buffer_size" rel="noreferrer" target="_blank">http://nginx.org/r/ssl_buffer_size</a>).<br>
<br>
You may try debugging log to futher find out where the particular <br>
allocation happens, see here for details:<br>
<br>
<a href="http://nginx.org/en/docs/debugging_log.html" rel="noreferrer" target="_blank">http://nginx.org/en/docs/debugging_log.html</a><br>
<br>
But I don't really think it worth the effort. The error is pretty <br>
clear, and it's better to focus on why these allocations are <br>
denied. Likely you are hitting some limit.<br>
<br>
-- <br>
Maxim Dounin<br>
<a href="http://mdounin.ru/" rel="noreferrer" target="_blank">http://mdounin.ru/</a><br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><b>Anoop P Alias</b> <div><br></div></div></div></div>