SSL Memory Usage and Fragmentation

[Igor Sysoev]

On Wed, Dec 26, 2007 at 12:34:39PM -0500, Ben Maurer wrote:

> On a production server, I found that nginx appears to leak when using 
> ssl. With some investigation, it seems that this is actually memory 
> fragmentation due to the session cache. I made a very simple 
> configuration for the server:
> 
> daemon off;
> master_process off;
> pid /tmp/x.pid;
> error_log /tmp/x.log;
> events { use epoll; }
> http {
>       client_body_temp_path /tmp;
>      proxy_temp_path /tmp;
>      fastcgi_temp_path /tmp;
>      access_log /tmp/access.log;
>      server {
>      	    listen localhost:8666;
> 	    ssl on;
> 	    ssl_certificate      /home/bmaurer/x.pem;
>             ssl_certificate_key  /home/bmaurer/x.pem;
> 	    root /tmp;
>      }
> }
> 
> Then I did a benchmark with the following command:
> 
> ab -c500 -n20000 https://localhost:8666/
> 
> After doing this, the server uses ~ 30 MB of RSS. Running it once more, 
> it uses ~ 40 MB of RSS. Valgrind claims that there are no "leaks", it 
> seems that there's just a really bad case of memory fragmentation.
> 
> I tried applying this to the SSL configuration:
> 
>  ssl_session_cache builtin:2;
> 
> Doing so resulted in the memory use of the nginx server staying 
> relatively low (it appears the memory was reclaimed from the OS after it 
> was used).
> 
> It seems like it might be worth switching to something like the shared 
> memory cache by default. Keeping the long-lived session cache in a 
> different pool of memory avoids the risk of large amounts of memory 
> getting pinned in.

Well, I will make shared session cache by default. It seems it's quite
stable. Other possible drawback of builtin cache as I think: it uses
a hash to store sessions and cache cleaning may take a long time.

> One other thing I noticed while investigating this stuff was that nginx 
> keeps a 16 KB buffer for each SSL connection for the entire duration of 
> the connection. I've attached a patch that keeps this buffer alive only 
> while there's a pending write. Sadly, there are some relatively large 
> buffers internal to openssl as well, which means the overhead for SSL 
> keepalive connections is pretty high.

Thank you, I will allocate it on demand, and will free using special
function just before a request will go to keep-alive state.

As to OpenSSL it takes about 100K per connection.


-- 
Igor Sysoev
http://sysoev.ru/en/