nginx ssl slow

[Maxim Dounin]

Hello!

On Tue, Jul 12, 2011 at 01:58:38AM -0600, Mark Maunder wrote:

> Igor I did SSL benchmarks with 10 worker processes on a very fast
> multicore machine with multiple ssl_session_cache configs to try and
> disprove this post. My results were also slow:
> 
> On a 4 core Xeon E5410 using:
> 
> ab -c 50 -n 5000
> 
> with 64 bit ubuntu 10.10 and kernel 2.6.35 I get:
> 
> For a 43 byte transparent gif image on regular HTTP:
> 
> Requests per second: 11703.19 [#/sec] (mean)
> 
> Same file via HTTPS with various ssl_session_cache params set:
> 
> ssl_session_cache shared:SSL:10m;
> Requests per second: 180.13 [#/sec] (mean)
> 
> ssl_session_cache builtin:1000 shared:SSL:10m;
> Requests per second: 183.53 [#/sec] (mean)
> 
> ssl_session_cache builtin:1000;
> Requests per second: 182.63 [#/sec] (mean)
> 
> No ssl_session_cache:
> Requests per second: 184.67 [#/sec] (mean)
> 
> 
> I'm assuming the session cache has no effect since each 'ab' request
> is a new session. But I thought I'd try it anyway.

Yes, ab won't reuse sessions.

> 180 per second for a machine this fast compared to 11,703 per second
> on regular HTTP seems like a big difference. 'ab' was run on the
> local machine (it takes very little CPU) so there was zero network
> latency.

I've did some tests on 2 x X5355 (4 cores each, 8 cores total)
server, it should be comparable to yours E5410.  I've used 
empty_gif to test as well.

First of all, ab wasn't even able to saturate *regular* http while 
eating 100% cpu (i.e. the whole cpu core, it just can't eat more 
as it's single thread/single process).  That is, it only shows 
something about 13k r/s, while with 5 ab processes nginx is 
actually able to handle 50k r/s over loopback.

So about "ab takes very little CPU": no it doesn't, it's awfully 
CPU bound.  If you see low numbers in top - make sure top shows 
%CPU for a core, not for all cores in system, or you'll see small 
number like 12.5% (100%/8) for a "whole core loaded, can't eat 
more" case.  Under linux it should be switchable by pressing 'I' 
(Irix mode vs. Solaris mode).

> Let me know if there's anything I should try to speed it up.

The same as the above applies to https as well.  While using 1024 
bit RSA key and DHE-RSA-AES256-SHA cipher, with 8 ab processes 
from another host I see 1800 r/s while system being 100% busy.

Other results include:

1024 bit key, DHE-RSA-AES256-SHA - 1800 r/s
2048 bit key, DHE-RSA-AES256-SHA - 1050 r/s
4096 bit key, DHE-RSA-AES256-SHA - 270 r/s

With ECDHE ciphers (and patch already mentioned in this thread):

1024 bit key, ECDHE-RSA-AES256-SHA - 2740 r/s
2048 bit key, ECDHE-RSA-AES256-SHA - 1340 r/s
4096 bit key, ECDHE-RSA-AES256-SHA - 285 r/s

This is with fairy trivial nginx config:

    worker_processes  8;

    error_log /path/to/error_log;
    
    events {
        worker_connections  10240;
    }
    
    http {
        include       mime.types;
        default_type  application/octet-stream;

        access_log /path/to/access_log;
    
        server {
            listen       8443;
            server_name  localhost;
    
            ssl                  on;
            ssl_certificate      cert.pem;
            ssl_certificate_key  cert.key;
    
            location / {
                empty_gif;
            }
        }
    }

Obviously using other cipher suites will produce much different 
results.

Just to compare, here are results from stunnel on the same 
machine:

1024 bit key, DHE-RSA-AES256-SHA - 1990 r/s
2048 bit key, DHE-RSA-AES256-SHA - 1220 r/s
4096 bit key, DHE-RSA-AES256-SHA - 280 r/s

1024 bit key, ECDHE-RSA-AES256-SHA - 2285 r/s
2048 bit key, ECDHE-RSA-AES256-SHA - 1223 r/s
4096 bit key, ECDHE-RSA-AES256-SHA - 285 r/s

It looks a bit faster with DHE ciphers, and the reason is not 
using SSL_OP_SINGLE_DH_USE option by default.  Setting "options 
SINGLE_DH_USE" in config results in the following DHE performance 
of stunnel:

1024 bit key, DHE-RSA-AES256-SHA - 1480 r/s
2048 bit key, DHE-RSA-AES256-SHA - 953 r/s
4096 bit key, DHE-RSA-AES256-SHA - 260 r/s

Maxim Dounin