Nginx runs out of memory with large value for 'keepalive_requests'

[Maxim Dounin]

Hello!

On Thu, Jul 12, 2018 at 05:48:55AM -0400, prajos wrote:

> Hi all,
> I'm using nginx as a Revers proxy to a service (A). nginx receives a large
> number of persistent connections from a single client service(B). 
> Service B sends a lot of requests (2K rps) over these persistent
> connections.
> 
> The amount of memory nginx uses seems to increase as a function of 
> 'keepalive_requests 2147483647' . The memory used keeps raising until the
> machine runs out of memory (4GB, aws instance).  While a smaller
> ''keepalive_requests 8192' doesn't create the exact problem.  
> 
> Some additional observations:
> When I reload nginx the memory usage comes down and then slowly starts
> building up.
> when I test nginx with a gatling test tool as a client, this behaviour is
> not observed.
> When I use the actual service(B), this behaviour seems to reappear. 
> 
> I curious to know what exactly is happening and how can I fix this issue of
> high memory usage ?

There may be allocations from a connection memory pool, and these 
allocations are freed only on connection close.  Trying to use 
"keepalive_requests 2147483647" is expected to result in memory 
usage growth as long as connections are never closed.

Tuning various settings might help to eliminate connection-related 
allocations.  In particular, if you've already tuned some settings 
from their default values, switching back to defaults might be a 
good starting point.  Though in general it is a bad idea to never 
close keepalive connections, the number of requests is limited for 
a reason.

-- 
Maxim Dounin
http://mdounin.ru/