posix_memalign(16, 16384) failed (12: Cannot allocate memory)

[Maxim Dounin]

Hello!

On Fri, Sep 23, 2016 at 05:20:56AM -0400, JohnCarne wrote:

> Hello,
> 
> [root at web1 ~]# nginx -v
> nginx version: nginx/1.11.4
> 
> We are now after 13 days we observer suddenly in nginx logs this in an
> intempestive manner, and causing nginx to reload, causing slow down on
> server : posix_memalign(16, 16384) failed (12: Cannot allocate memory)
> 
> This happens after our upgrade to last nginx version through nDeploy.
> 
> I called in nginx sysadmin, Ndeploy sysadmin too, and finally cloudlinux
> support which made an incredible job investigating the issue over 7 days by
> enabling multiple kernel debug tools to find out what is going on.
> 
> All nginx/linux settings has been tweaked/verified. Issue can't be solved,
> and about 5 guys has broken their head on the issue, without being able to
> solve. We know all the basic, even advanced, and experts were in.

Just a basic hint, in case you haven't tried it yet: re-compile 
nginx without any 3rd party modules, and check if it helps.

> Cloudlinux support says this is the cause, and you need nginx expert to find
> out why nginx beheave likes this :
> 
> From the information we collected it appears that nginx is really changing
> his ulimits:
> # grep nginx /home/abackupnomem3.log | tail
> nginx-792752 [009] 5438179.898678: setrlimit: (sys_setrlimit+0x63/0x70
> 
> Conclusion is that nginx manage those rlimits. This is not a solution, but a
> way for you where to dig more.

The setrlimit() call is used by nginx to manage some limits it 
knows about and configured to manage.  In particular, it is used 
for the

    worker_rlimit_nofile 50000;

directive as seen in your config, and for the worker_rlimit_core 
directive.  Details about the directives can be found here:

http://nginx.org/r/worker_rlimit_core
http://nginx.org/r/worker_rlimit_nofile

They set RLIMIT_CORE and RLIMIT_NOFILE limits, nothing more, and 
have nothing to do with the memory allocation errors you see.

-- 
Maxim Dounin
http://nginx.org/