Normal memory usage for SSL terminating, reverse proxy nginx?

[Maxim Dounin]

Hello!

On Fri, May 27, 2016 at 03:15:25AM -0400, onecrazymonkey wrote:

> It has been a difficult topic to research. The nginx instance is doing
> nothing more than what the subject stated. It reverse proxies to a backend,
> load balanced set of web app instances and terminates SSL for a large number
> of unique domains each with their own SSL cert. Here's a `ps aux` of nginx
> running after a clean start and zero (out of rotation) traffic.
> 
> root      20   0  676052 598224   1848 S   0.0 16.5   0:00.06 nginx         
>                                   
> nginx     20   0  675552 597204   1228 S   0.0 16.5   0:00.44 nginx         
>                                   
> nginx     20   0  675552 596612    636 S   0.0 16.5   0:00.36 nginx
> 
> Looking at that process list, nginx is using about 676mb of RAM for ~400
> vhosts each with their own unique SSL cert for a unique domain. Here's an
> example of a vhost server config. They're all generated based on the same
> base template:
> 
> server {
>   listen              443 ssl proxy_protocol;
>   server_name         <uniquehostname> www.<uniquehostname>;
>   access_log          /var/log/nginx/access_vhost_443.log accesslog;
>   error_log           /var/log/nginx/error_vhost_443.log warn;
>   real_ip_header      proxy_protocol;
> 
>   ssl                 on;
>   ssl_certificate     /etc/nginx/ssl/<uniquehostname>/<uniquehostname>.crt;
>   ssl_certificate_key /etc/nginx/ssl/<uniquehostname>/<uniquehostname>.key;
> 
>   ssl_stapling        on;
>   ssl_stapling_verify on;
> 
>   resolver        internal-dns.vpc  valid=60s;

Note: by using using own resolver in each server{} you add more 
flexibility, but waste memory and other resources.  Consider using 
single resolver defined at http{} level instead.

>   set             $internal "upstream-load-balancer.vpc";
>   location / {
>     if ($denied) {
>       return 444;
>     }
>     proxy_pass          http://$internal;
>   }
> }
> 
> Now, this wouldn't be all that bad. 1.69mb of memory per vhost isn't
> horrible, high, but not unsustainable. However, if I do `nginx -s reload` or
> restart via systemd service... 
> 
> root      20   0 1370188 1.176g   3240 S   0.0 33.4   0:14.98 nginx         
>                                   
> nginx     20   0 1370192 1.175g   2584 S   0.3 33.4   2:39.95 nginx         
>                                   
> nginx     20   0 1370192 1.175g   2584 S   1.7 33.4   2:28.42 nginx
> 
> It doubles the memory consumption! It never goes up or down drastically
> again. It's as if it duplicates and never frees or releases unless you do a
> restart.

As nginx allocates a new configuration before releasing the old 
one, and this doubles memory consumption of the master process for a 
while.  The old configuration is then freed, but in many cases 
this memory isn't really returned to kernel by the allocator, and 
hence seen in various stats.

> This was tested on a handful of AWS EC2 instance types using vanilla Centos7
> and both nginx 1.6.3 (stable in centos repos) and nginx 1.10.0 (nginx.org
> repo).
> 
> In summary, my questions are thus:
> - Is it normal for nginx to use ~1.7mb per SSL vhost?

Not really.  This is likely due to a particular configuration you 
are using: own resolver in each vhost, own logs, etc.  Consider 
moving common settings to the http{} level, it should somewhat reduce 
memory consumption.

Just for the reference: a simple test with 10k SSL vhosts like:

server { listen 8443; ssl on; server_name 1.foo; ssl_certificate test.crt; ssl_certificate_key test.key; }
server { listen 8443; ssl on; server_name 2.foo; ssl_certificate test.crt; ssl_certificate_key test.key; }
...
server { listen 8443; ssl on; server_name 10000.foo; ssl_certificate test.crt; ssl_certificate_key test.key; }

takes about 200mb of memory (on a 32-bit host).  That is, about 
20k of memory per vhost.

With more options it will take more memory, but 1.7mb is a bit too 
many - you are doing something wrong.

Just a guess: if you use ssl_trusted_certificate at http{} level, 
make sure you are loading only needed certificates, not a full 
OS-provided bundle.

> - Is there a way to reduce that memory usage?

Yes, see various hints above.  In summary: configure things at 
http{} level where possible, avoid memory-hungry things you don't 
need.

> - Am I the only one that experiences the doubling of nginx memory usage
> after a nginx reload?
> - Is that a bug?

No and no, see above.

-- 
Maxim Dounin
http://nginx.org/