Occupancy of SSL connections?

[Maxim Dounin]

Hello!

On Thu, Nov 12, 2009 at 09:15:14AM -0500, edwh2 wrote:

> -------------------------------------------------------
> > On Mon, Nov 09, 2009 at 05:34:12PM +0000, Edward
> > Hibbert wrote:
> > 
> > > For HTTP connections, I can get about 20K
> > connections and outstanding requests in 116MB,
> > which is pretty respectable.
> > > 
> > > However with HTTPS, what I see is that:
> > > - having the connection open takes 261MB for
> > about 16K connections
> > > - having the connection open with an outstanding
> > request takes about 1GB for 10L connections.
> 
> Using the OpenSSL 1.0 (thanks for the suggestion, Cliff) there's a significant drop in occupancy.
> - 16K open connections now takes ~277MB (noise level difference from previous measurement)

Are you sure you measure allocated memory?  For 16k connections 
openssl (at least 0.9.8, but you claim numbers are the same...) 
should allocate about 1.2G (~80k per connection) on ssl handshake.

Numbers you see in resident memory (RES in top) may be quite 
different, but it just means that relevant memory wasn't yet 
touched.  And grow on request processing is expected.

> - 16K connections with the outstanding GET request now takes ~1GB (down from estimated 1.6GB previously)
> 
> So there's still a massive occupancy cost of having an outstanding GET request in the HTTPS case, which is surprising.  Looks like I'd better roll my sleeves up and dig into the code.

For outstanding GET requests there is 16k ssl buffer allocated by 
nginx on first output (see src/event/ngx_event_openssl.c) and 
freed when connection goes to keepalive state.  This gives 256M 
for 16k connections.

Also there is a bunch of memory in nginx (output buffers, large 
client header buffers, gzip buffers, various request-related data 
and so on) which are freed when connection goes to keepalive 
state.  Depending on your settings, particular request and already 
sent response this may contribute various numbers to keepalive vs.  
outstanding request cases.  This should be almost the same in 
non-https case though, at least with sendfile not used.

Maxim Dounin