Occupancy of SSL connections?

[edwh2]

Thanks for your response.

Maxim Dounin Wrote:

> 
> Are you sure you measure allocated memory?  

Yes, I'm looking at VIRT in top rather than RES.

> For
> 16k connections 
> openssl (at least 0.9.8, but you claim numbers are
> the same...) 
> should allocate about 1.2G (~80k per connection)
> on ssl handshake.

That's odd, because I don't see that.  I'm using some slightly hacky code derived from some openssl examples, and here's what it looks like:

    /* Build our SSL context*/
    ctx=initialize_ctx(KEYFILE,PASSWORD);

    int loop;
    for (loop = 0; loop < 20000; loop++)
    {
    printf("Loop %d\n", loop);
    /* Connect the TCP socket*/
    sock=tcp_connect(host,port);

    /* Connect the SSL socket */
    ssl=SSL_new(ctx);
    sbio=BIO_new_socket(sock,BIO_NOCLOSE);
    SSL_set_bio(ssl,sbio,sbio);

    if(SSL_connect(ssl)<=0)
      berr_exit("SSL connect error");
    if(require_server_auth)
      check_cert(ssl,host);

    /* Now make our HTTP request */
    http_request(ssl, loop);

    /* Shutdown the socket */
    //destroy_ctx(ctx);
    //close(sock);
    }

The difference in occupancy comes from whether or not I comment out the http_request code.  Would you expect the openssl calls above to have incurred the server-side occupancy hit?

> 
> Numbers you see in resident memory (RES in top)
> may be quite 
> different, but it just means that relevant memory
> wasn't yet 
> touched.  And grow on request processing is
> expected.
> 

Yes, these are easily confused but I'm familiar with the problem.

> > - 16K connections with the outstanding GET
> request now takes ~1GB (down from estimated 1.6GB
> previously)
> > 
> > So there's still a massive occupancy cost of
> having an outstanding GET request in the HTTPS
> case, which is surprising.  Looks like I'd better
> roll my sleeves up and dig into the code.
> 
> For outstanding GET requests there is 16k ssl
> buffer allocated by 
> nginx on first output (see
> src/event/ngx_event_openssl.c) and 
> freed when connection goes to keepalive state. 
> This gives 256M 
> for 16k connections.

Ok, that's useful.  I'll look at the code, but do you think it's feasible either to 
- shrink this (how small could it be?  what would happen if it was too small?), or
- free it in between getting a request in and sending the response?

That wouldn't account for all the difference I see, but 256M would go some way to reducing the occupancy hit.

> 
> Also there is a bunch of memory in nginx (output
> buffers, large 
> client header buffers, gzip buffers, various
> request-related data 
> and so on) which are freed when connection goes to
> keepalive 
> state.  Depending on your settings, particular
> request and already 
> sent response this may contribute various numbers
> to keepalive vs.  
> outstanding request cases.  This should be almost
> the same in 
> non-https case though, at least with sendfile not
> used.

Yes, that's what I would expect, and the occupancy in the non-https case is pretty good.  So I'm hoping not to have to touch that, and that this is something https-specific.

Edward.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,21531,22405#msg-22405