ngx_slab_alloc() failed: no memory (not push, not key space)

[Maxwell Lamb]

Hi Folks,

As ever, thanks for the mindblowingly great webserver.

Unfortunately, we've run into an odd little issue.

We've recently started auto-scaling at AWS, and we add and remove upstream hosts from the config via a dinky shell script which checks for hosts in the autoscaling group via the AWS API, and just adds them to/removes them from the config appropriately.

After modifying our upstream conf, we do

service nginx configtest 

which if successful is followed by

service nginx reload

so far, so good.

Our config is pretty straightforward, and looks a bit like

upstream apache_v5 {
	server 10.0.1.81:80;
	server 10.0.1.78:80;
	server 10.0.1.218:80;
	server 10.0.1.96:80;
	server 10.0.1.71:80;
	server 10.0.1.97:80;
	server 10.0.1.237:80;
	server 10.0.1.224:80;
	server 10.0.1.66:80;
	server 10.0.1.51:80;
	server 10.0.1.21:80;

	fair;
}

This all works beautifully for a while, but after a number of reloads and config modifications (no particular number we've been able to establish), each reload results in

ngx_slab_alloc() failed: no memory

and no reload of the config, preventing us from adding or removing further upstream hosts without completely restarting nginx. There are no errors preceding this, and on each subsequent reload, it reports the same error.

We thought this may have been related to proxy buffers, so we decreased both their size and number, but this has had no impact.

Additionally, while there's no particular pattern in terms of an absolute number of reloads that we've been able to determine, the two nginx entry servers we're currently using both exhibit the same behaviour at the same time. We're running 1.0.14. We haven't tried with a more up-to-date version just yet, in the hope that this may be an obscure known issue.

Any help/insight/suggestions would be much appreciated!

Thanks,

Max