Occupancy of SSL connections?

[Maxim Dounin]

Hello!

On Thu, Nov 12, 2009 at 01:11:08PM -0500, edwh2 wrote:

> Thanks for your response.
> 
> Maxim Dounin Wrote:
> 
> > 
> > Are you sure you measure allocated memory?  
> 
> Yes, I'm looking at VIRT in top rather than RES.
> 
> > For
> > 16k connections 
> > openssl (at least 0.9.8, but you claim numbers are
> > the same...) 
> > should allocate about 1.2G (~80k per connection)
> > on ssl handshake.
> 
> That's odd, because I don't see that.  I'm using some slightly hacky code derived from some openssl examples, and here's what it looks like:
> 
>     /* Build our SSL context*/
>     ctx=initialize_ctx(KEYFILE,PASSWORD);
> 
>     int loop;
>     for (loop = 0; loop < 20000; loop++)
>     {
>     printf("Loop %d\n", loop);
>     /* Connect the TCP socket*/
>     sock=tcp_connect(host,port);
> 
>     /* Connect the SSL socket */
>     ssl=SSL_new(ctx);
>     sbio=BIO_new_socket(sock,BIO_NOCLOSE);
>     SSL_set_bio(ssl,sbio,sbio);
> 
>     if(SSL_connect(ssl)<=0)
>       berr_exit("SSL connect error");
>     if(require_server_auth)
>       check_cert(ssl,host);
> 
>     /* Now make our HTTP request */
>     http_request(ssl, loop);
> 
>     /* Shutdown the socket */
>     //destroy_ctx(ctx);
>     //close(sock);
>     }
> 
> The difference in occupancy comes from whether or not I comment out the http_request code.  Would you expect the openssl calls above to have incurred the server-side occupancy hit?

See nothing obviously wrong here, but can't say anything 
for sure without full code.

Looking into openssl source, there are at least the following 
allocation right after it got ClientHello message (ssl3/tls1 
case):

~21k in ssl3_accept() for s->init_buf

~34k in ssl3_setup_buffers() for s->s3->rbuf (as nginx uses 
SSL_OP_MICROSOFT_BIG_SSLV3_BUFFER workaround option)

~18k in ssl3_setup_buffers() for s->s3->wbuf

About ~70k in total.

Also there are about 8k for input/ouput bio buffers as well, and 
various small allocations for state information (including copy of 
server cert and so on).  This looks much closer to 270M per 16k 
connections you see, so probably your client doesn't send 
ClientHello for some reason.

You may try running nginx with debug logging enabled and take a 
look if ssl handshake actually happens.  There should be '[debug] ...  
SSL: ..., cipher: ..." lines right after ssl handshake completed.  
Also you may try tcpdump to see what happens on the wire.

[...]

> > For outstanding GET requests there is 16k ssl
> > buffer allocated by 
> > nginx on first output (see
> > src/event/ngx_event_openssl.c) and 
> > freed when connection goes to keepalive state. 
> > This gives 256M 
> > for 16k connections.
> 
> Ok, that's useful.  I'll look at the code, but do you think it's feasible either to 
> - shrink this (how small could it be?  what would happen if it was too small?), or
> - free it in between getting a request in and sending the response?

It's already allocated only when needed (i.e. when nginx starts 
sending the response, and freed when connection goes to keepalive 
state - i.e. after sending the response).  Reducing it will likely 
render it useless - it's here to reduce ssl overhead by feeding 
openssl with big chunks of data instead multiple small ones.

Anyway - it's almost nothing compared to openssl's own buffers.

Maxim Dounin