Transforming SSL server cert and private key in variables.

[António P. P. Almeida]

On 1 Fev 2013 16h25 CET, mdounin at mdounin.ru wrote:

> Hello!

Hello Maxim,


>> I made some tests and definitely we cannot use in our product, it
>> takes too much time and resources. 
>>
>> Tested on an EC m1.medium instance.
>>
>> HOSTS,DATE,COMMAND,CPU_PERCENTAGE,CPU_SYSTEM,CPU_USER,ELAPSED_TIME,IO_PG_FAULTS,ICONTEXT_SWITCHING,VCONTEXT_SWITCHING,MAX_MEMORY
>> 5001,01.Feb.2013 00:18:33,/usr/sbin/nginx -s
>> reload,92%,0.60,1.84,0:02.62,0,3296,1,138528 10001,01.Feb.2013
>> 00:19:32,/usr/sbin/nginx -s
>> reload,93%,1.67,5.80,0:08.00,0,11627,1,406804 20001,01.Feb.2013
>> 00:20:23,/usr/sbin/nginx -s
>> reload,93%,4.17,13.68,0:19.16,0,25221,1,945164 50001,01.Feb.2013
>> 00:22:02,/usr/sbin/nginx -s
>> reload,60%,13.24,36.37,1:22.46,14,68338,87121,2288668
>>
>> As you can see 50k hosts take more than one minute. That would be
>> acceptable if it weren't for the fact that it uses up a lot of CPU
>> and memory.
>
> 50k ssl hosts is a lot, actually, and loading 50k certificates 
> in 1 minute on a cheap virtual server looks reasonable for me. 
> Especially if you consider total cost of 50k ssl certs.

Yes, but they're the clients/users certs. Not ours.

> Please also make sure you are testing recent enough nginx.  In 
> paticular, the following commit speeds up ssl loading 
> about 2x here (available in nginx 1.3.11):

perusio at dev:~$ /usr/sbin/nginx -V
nginx version: nginx/1.3.11
built by gcc 4.7.2 (Debian 4.7.2-5) 
TLS SNI support enabled
configure arguments: --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-debug --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_fastcgi_module --without-http_autoindex_module --without-http_split_clients_module --without-http_memcached_module --without-http_scgi_module --without-http_browser_module --without-http_split_clients_module --without-http_autoindex_module --without-http_userid_module --without-http_ssi_module --without-select_module --without-poll_module --with-http_gzip_static_module --with-http_realip_module --with-http_stub_status_module --with-http_ssl_module --with-i
 pv6 --with-file-aio

> http://trac.nginx.org/nginx/changeset/5004/nginx
>
> The real limiting factor with many server blocks seems to be 
> memory used for each server{}.  This probably needs some 
> optimization.

Looking at htop, the config parsing is taking a lot of time. I know
that because I also do a nginx -t before. It's just that I ommited
that from the mail. Here they are:

HOSTS,DATE,COMMAND,CPU_PERCENTAGE,CPU_SYSTEM,CPU_USER,ELAPSED_TIME,IO_PG_FAULTS,ICONTEXT_SWITCHING,VCONTEXT_SWITCHING,MAX_MEMORY
5001,01.Feb.2013 00:18:31,/usr/sbin/nginx_ensite 03805-08805.test-ssl.local.conf,93%,0.59,1.85,0:02.61,0,3450,17,138532
10001,01.Feb.2013 00:19:24,/usr/sbin/nginx_ensite 08806-18806.test-ssl.local.conf,93%,1.81,5.63,0:07.95,0,10684,16,406804
20001,01.Feb.2013 00:20:04,/usr/sbin/nginx_ensite 18807-38807.test-ssl.local.conf,93%,4.02,13.92,0:19.17,0,27021,17,945164
50001,01.Feb.2013 00:21:13,/usr/sbin/nginx_ensite 38808-88808.test-ssl.local.conf,93%,10.05,35.70,0:49.07,0,67976,17,2288672

nginx_ensite is a small shell script that creates a symlink and does a
nginx -t.


> I don't think that parsing of the config is a culprit.  More 
> likely it's SSL certificate reading/checking/various random 
> initialization/generation.  (And may be server names hash 
> generation if there are many collisions on server names.)

It's not my experience. In fact, several times I mangled up the cert
names and he generated the not found cert error only quite late in the process.

> Some profiling would be helpful.

I did a ltrace and, hands down, string operations are the main
thing. For a single server block.

awk 'BEGIN {s=0} /strcmp/ {s+=$1} END {print s}' single_ltrace_function_stats_libs.csv 
-> 4417

awk 'BEGIN {s=0} /memcpy/ {s+=$1} END {print s}' single_ltrace_function_stats_libs.csv                                                                                           
-> 245

awk 'BEGIN {s=0} /SSL/ {s+=$1} END {print s}' single_ltrace_function_stats_libs.csv                                                                                              
-> 49

So there are 49 ops for SSL functions against 4417 string comparisons.

Is my reasoning flawed?

--- appa