SSL Memory Usage and Fragmentation

Ben Maurer bmaurer at andrew.cmu.edu
Wed Dec 26 20:34:39 MSK 2007


Hi,

On a production server, I found that nginx appears to leak when using 
ssl. With some investigation, it seems that this is actually memory 
fragmentation due to the session cache. I made a very simple 
configuration for the server:

daemon off;
master_process off;
pid /tmp/x.pid;
error_log /tmp/x.log;
events { use epoll; }
http {
       client_body_temp_path /tmp;
      proxy_temp_path /tmp;
      fastcgi_temp_path /tmp;
      access_log /tmp/access.log;
      server {
      	    listen localhost:8666;
	    ssl on;
	    ssl_certificate      /home/bmaurer/x.pem;
             ssl_certificate_key  /home/bmaurer/x.pem;
	    root /tmp;
      }
}

Then I did a benchmark with the following command:

ab -c500 -n20000 https://localhost:8666/

After doing this, the server uses ~ 30 MB of RSS. Running it once more, 
it uses ~ 40 MB of RSS. Valgrind claims that there are no "leaks", it 
seems that there's just a really bad case of memory fragmentation.

I tried applying this to the SSL configuration:

  ssl_session_cache builtin:2;

Doing so resulted in the memory use of the nginx server staying 
relatively low (it appears the memory was reclaimed from the OS after it 
was used).

It seems like it might be worth switching to something like the shared 
memory cache by default. Keeping the long-lived session cache in a 
different pool of memory avoids the risk of large amounts of memory 
getting pinned in.

One other thing I noticed while investigating this stuff was that nginx 
keeps a 16 KB buffer for each SSL connection for the entire duration of 
the connection. I've attached a patch that keeps this buffer alive only 
while there's a pending write. Sadly, there are some relatively large 
buffers internal to openssl as well, which means the overhead for SSL 
keepalive connections is pretty high.

- Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx-fewer-ssl-buffers.patch
Type: text/x-patch
Size: 1481 bytes
Desc: not available
URL: <http://nginx.org/pipermail/nginx/attachments/20071226/19a6d57b/attachment.bin>


More information about the nginx mailing list