<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Anoop,<br>
<br>
I doubt this will be the solution, but may we have a look at
/proc/buddyinfo and /proc/slabinfo the moment when nginx can't
allocate memory?<br>
<br>
On 02.08.2018 08:15, Anoop Alias wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAO6TEX1upk68RYdeDynh_05+bfnT_DB77WYbGnBv2rsh8Q73VA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<div dir="ltr">
<div>Hi Maxim,</div>
<div><br>
</div>
<div>I enabled debug and the memalign call is happening on nginx
reloads and the ENOMEM happen sometimes on the reload(not on
all reloads)</div>
<div><br>
</div>
<div>
<div>2018/08/02 05:59:08 [notice] 872052#872052: signal
process started</div>
<div>2018/08/02 05:59:23 [notice] 871570#871570: signal 1
(SIGHUP) received from 872052, reconfiguring</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio
0</div>
<div>2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 0000000002B0DA00:16384 @16 === > the
memalign call on reload</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570: malloc:
00000000087924D0:4560</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 000000000E442E00:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570: malloc:
0000000005650850:4096</div>
<div>20</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443
#71</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443
#72</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443
#73</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443
#74</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: add cleanup:
000000005340D728</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000024D3260:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000517BAF10:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
0000000053854FC0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
0000000053855FD0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
0000000053856FE0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
0000000053857FF0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign:
0000000053859000:16384 @16</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
000000005385D010:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
000000005385E020:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
000000005385F030:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536CD160:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536CE170:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536CF180:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D0190:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D11A0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D21B0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D31C0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D41D0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D51E0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D61F0:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D7200:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D8210:4096</div>
<div>2018/08/02 05:48:49 [debug] 871275#871275: malloc:
00000000536D9220:4096</div>
<div><br>
</div>
<div><br>
</div>
<div>Infact there are lot of such calls during a reload</div>
<div><br>
</div>
<div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA17ED00:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA1B0FF0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA1E12C0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA211590:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA243880:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA271B30:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA2A3E20:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA2D20D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA3063E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA334690:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA366980:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA396C50:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA3C8F40:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA3F9210:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA4294E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA45B7D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA489A80:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA4BBD70:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA4EA020:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA51E330:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA54C5E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA57E8D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA5AEBA0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA5DEE70:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA611160:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA641430:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA671700:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA6A29E0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA6D5CE0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA707FD0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA736280:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA768570:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA796820:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA7CAB30:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA7F8DE0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA82B0D0:16384 @16</div>
<div>2018/08/02 05:59:23 [debug] 871570#871570:
posix_memalign: 00000000BA85B3A0:16384 @16</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>What is perplexing is that the system has enough free
(available RAM)</div>
<div>#############</div>
<div>
<div># free -g</div>
<div> total used free shared
buff/cache available</div>
<div>Mem: 125 54 24 8
46 58</div>
<div>Swap: 0 0 0</div>
</div>
<div>#############</div>
<div><br>
</div>
<div>
<div># ulimit -a</div>
<div>core file size (blocks, -c) 0</div>
<div>data seg size (kbytes, -d) unlimited</div>
<div>scheduling priority (-e) 0</div>
<div>file size (blocks, -f) unlimited</div>
<div>pending signals (-i) 514579</div>
<div>max locked memory (kbytes, -l) 64</div>
<div>max memory size (kbytes, -m) unlimited</div>
<div>open files (-n) 1024</div>
<div>pipe size (512 bytes, -p) 8</div>
<div>POSIX message queues (bytes, -q) 819200</div>
<div>real-time priority (-r) 0</div>
<div>stack size (kbytes, -s) 8192</div>
<div>cpu time (seconds, -t) unlimited</div>
<div>max user processes (-u) 514579</div>
<div>virtual memory (kbytes, -v) unlimited</div>
<div>file locks (-x) unlimited</div>
</div>
<div><br>
</div>
<div>#########################################</div>
<div><br>
</div>
<div>There is no other thing limiting memory allocation </div>
<div><br>
</div>
<div>Any way to prevent this or probably identify/prevent this</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin <<a
href="mailto:mdounin@mdounin.ru" moz-do-not-send="true">mdounin@mdounin.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hello!<br>
<br>
On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote:<br>
<br>
> I am repeatedly seeing errors like<br>
> <br>
> ######################<br>
> 2018/07/31 03:46:33 [emerg] 2854560#2854560:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 03:54:09 [emerg] 2890190#2890190:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:08:36 [emerg] 2939230#2939230:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:24:48 [emerg] 2992650#2992650:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:09 [emerg] 3053092#3053092:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:17 [emerg] 3053335#3053335:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:42:28 [emerg] 3053937#3053937:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> 2018/07/31 04:47:54 [emerg] 3070638#3070638:
posix_memalign(16, 16384)<br>
> failed (12: Cannot allocate memory)<br>
> ####################<br>
> <br>
> on a few servers<br>
> <br>
> The servers have enough memory free and the swap usage is
0, yet somehow<br>
> the kernel denies the posix_memalign with ENOMEM ( this
is what I think is<br>
> happening!)<br>
> <br>
> The numbers requested are always 16, 16k . This makes me
suspicious<br>
> <br>
> I have no setting in nginx.conf that reference a 16k<br>
> <br>
> Is there any chance of finding out what requests this and
why this is not<br>
> fulfilled<br>
<br>
There are at least some buffers which default to 16k - for <br>
example, ssl_buffer_size (<a
href="http://nginx.org/r/ssl_buffer_size" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://nginx.org/r/ssl_buffer_size</a>).<br>
<br>
You may try debugging log to futher find out where the
particular <br>
allocation happens, see here for details:<br>
<br>
<a href="http://nginx.org/en/docs/debugging_log.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://nginx.org/en/docs/debugging_log.html</a><br>
<br>
But I don't really think it worth the effort. The error is
pretty <br>
clear, and it's better to focus on why these allocations are <br>
denied. Likely you are hitting some limit.<br>
<br>
-- <br>
Maxim Dounin<br>
<a href="http://mdounin.ru/" rel="noreferrer" target="_blank"
moz-do-not-send="true">http://mdounin.ru/</a><br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank"
moz-do-not-send="true">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div><b>Anoop P Alias</b>
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
nginx mailing list
<a class="moz-txt-link-abbreviated" href="mailto:nginx@nginx.org">nginx@nginx.org</a>
<a class="moz-txt-link-freetext" href="http://mailman.nginx.org/mailman/listinfo/nginx">http://mailman.nginx.org/mailman/listinfo/nginx</a></pre>
</blockquote>
<p><br>
</p>
</body>
</html>