Query about keepalive

[Maxim Dounin]

Hello!

On Tue, Apr 24, 2012 at 05:22:39PM +0530, Fasih wrote:

> Hi All,
> 
> Few days back I was trying to evaluate the performance of upstream
> keepalive feature for a website when I noticed a rather unexpected
> behaviour. It would be help me understand what's going on in the test.
> 
> Here's what I did:
> 1. Setup httperf to run a session load. This basically means that a text
> file with different urls is supplied to httperf. httperf sends all the
> requests in bursts spaced by a sec towards nginx.
> 
> 2. Tcpdump is run on the machine
> 
> 3. Before the tests begin, cache is cleared and nginx restarted
> 
> 4. Test is repeated with httperf "replaying" the requests 1 time first, and
> re-run with repeat count 4 to account for setup of connections/cache
> priming etc
> 
> 5. All the steps are repeated once without keepalive and with keepalive 512;

Just a side note: you may want to avoid setting keepalive bigger 
than your backend is able to handle, keeping in mind that it's 
not a hard limit on connections established, but rather size of 
connection cache kept by each worker.

> Results:
> SessionKeep aliveConns upstreamConn TimeUnique upstream hostsReqs upstreamAvg
> time to 1st byteMax upstream conn reuseClient conns (1)Client reqs (1)Client
> replies (1)Testdur (1)Client connsClient reqsClient repliesTestdurmy-site148
> 8.30858191920.15262331213013031.219852052078.064my-site019220.716919192
> 0.1679461213013025.680852052071.781

Just a side note: please do not use html to post here.  We'll 
won't see it anyway, and plain text of your message is somewhat 
unreadable.

> Analysis:
> * First row with keepalive, second row without keepalive.
> * With keepalive, number of connections upstream (as seen in tcpdump) is
> 48. Note that my-site has multiple (19 - unique upstream hosts) subdomains,
> each of which is individually configured. Without keepalive 192 connections
> are made
> * Total time spent establishing connections is 8.3 vs 20.7
> * Latency is ~0.15secs
> All these are as expected, test duration however goes from 78 -> 71 secs.
> An *increase* in the time for test to complete.
>
> As the number of unique upstream increases, the time increases further.
> This wasnt something that I could explain. Please help me understand, is it
> a bug in the system?

How many times did you run the test?  From numbers it looks like 
you are measuring your network and/or upstream server performance, 
and I suspect this might fluctuate widely.  You might want to do 
the test at least 3 times in each configuration to be able to see 
the difference between two configurations.

If the difference will still be there, you may want to share more 
details about your setup (provide nginx configs, network details, 
description of the upstream servers involved, probably debug log 
for a deeper investigation).

It is possible that keepalive connections to upstreams will lead 
to worse overall performance than no keepalives (notably due to 
various network effects and upstream servers behaviour), but I 
wouldn't expect it to be slower in general.

Maxim Dounin