High memory usage

[bvek1]

Hello,

I need your help, I have a problem with my Nginx server : all the Nginx
processes are consuming a huge amount of memory and it sets off the OOM
Killer.
I don't understand why Nginx uses this much memory, it seems odd, I expected
the CPU to be the first problem instead of memory.

My Nginx server is just a reverse proxy, to all kind of backend. SSL can be
offloaded or not, it depends on the backend.

I have a huge amount of vhosts with SSL on this Nginx (~15K) and the server
have a lot of trafic (for me at least !) . 
The /stub_status tell me the following thing at this moment : 

Active connections: 3867 
server accepts handled requests
 2350039 2350039 5489648 
Reading: 0 Writing: 332 Waiting: 3605

Netdata (https://github.com/netdata/netdata) tell me that it handle between
300 and 600 req/s. 

[root at nginxsrv nginx]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 

[root at nginxsrv nginx]# nginx -V
nginx version: nginx/1.17.3
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) 
built with OpenSSL 1.1.0k  28 May 2019
TLS SNI support enabled
configure arguments: --add-module=/usr/local/src/ngx_brotli
--add-module=/usr/local/src/kyprizel-testcookie-nginx-module 
--with-threads --with-openssl=/usr/local/src/openssl --with-http_v2_module
--with-http_flv_module --with-ipv6 --with-http_mp4_module 
--sbin-path=/usr/local/sbin --conf-path=/etc/nginx/nginx.conf
--pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log 
--http-log-path=/var/log/nginx/access.log --with-http_realip_module
--with-http_ssl_module --http-client-body-temp-path=/eacc/nginx_client 
--http-proxy-temp-path=/eacc/nginx_proxy
--http-fastcgi-temp-path=/eacc/nginx_fastcgi --with-http_stub_status_module

The server have a lot of ram, 120G and the CPU is an Intel Xeon E5-2697 CPU.
The CPU is not used that much.

A typical vhost file is quite simple, it's just a proxy_pass to a backend,
with some hack to do a retry is we first get and error message (based on
proxy_intercept_errors). http2 is on. I don't use any caches. 

Things I have tested or I suspect to be an issue : 
- I reduced the worker_processes to 6. I had issue with the frequent
reloading of Nginx with an upper value. I suspect that it's more difficult
for Nginx to reload properly if a lot of processes are spawned (correct me
if i'm wrong)
- I have gzip on and gzip_proxied to any. I suspect the compression to be an
issue, I reduced the buffers value to gzip_buffers  32 4k; and
gzip_comp_level to 1
- I reduced the ssl_session_cache from 10m to 1m
- I reduced the keep_alive timeout too (30)
- Reloading Nginx often seems to be a problem too, the memory usage seems
higher during the reloading task. The reloading is quite slow too, probably
because of the number of vhosts file to load (if someone know how to speed
up this, it interest me !)
- I have a lot of limit_conn configuration
- ssl_stapling is off, I had issue with it with the DNS resolution.

Do you have idea on what can cause a high memory usage on Nginx ?

Thanks,
Alexis

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285979,285979#msg-285979