Resident memory not released

[fredr]

Hi all,

We are using the kubernetes nginx-ingress for websocket connections in front
of one of our applications. We have added automatic scaling based on the
resident memory, as that seems to be a good scaling metric when dealing with
persistent connections. But we noticed that the memory seems to never be
released, and thus it only scales up and never down.

I've done some testing locally using this dockerfile and nginx.conf with
vanilla nginx:
https://gist.github.com/fredr/d58f8221b813e4fdcf7bbfc08df30afa and it seems
to have the same behaviour.

Starting the docker image, nginx has allocated 211Mb in the resident memory
column in htop. 
Then I connected 20K websocket connections, memory raised to about 540Mb,
and then I disconnecting all the connections, the used memory stays at
540Mb.

If I reconnect all the 20K websocket connections, it seems to reuse the
already allocated memory. 
So I wounder, is this by design? is it somehow configurable, so that nginx
would release this memory? or is it the wrong value to base scaling on?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285025,285025#msg-285025