Bad performance of nginx with Tomcat vs. Apache with Tomcat

[Maxim Dounin]

Hello!

On Thu, Sep 03, 2009 at 01:50:44PM +0900, Chang Song wrote:

> Sometime ago, I posted an message about Nginx performance when paired  
> with Tomcat.
> We recently did extensive in-house testing of various workload against  
> Nginx with Tomcat vs Apache vs Tomcat.
>
> Apache wins hands down.
>
> Here's the basic setup
>
> 1. Nginx (2 proc/8192 connections) -> http/1.0 -> Tomcat (HTTP  
> connector)
> 2. Apache (512 prefork) -> AJP -> Tomcat (AJP)
>
> Both KeepAlive off (we don't use KeepAlive due to L4 switch)

As long as you don't care about concurrency > 512 and client 
keepalive - Apache may be better (or at least easier) choise here.

> The physical server is 2 core Intel Xeon, which is typical web server  
> config here.
> We have three grinder 3.2 load generators.
> We tested 4K and 20K Tomcat simple HTML file, 20K simple HTML with  
> intentional 10% 200ms
> sleep in Tomcat serving (emulate slow DB query), etc.
>
> Every single case, Apache wins by at least 10-15%.
> Throughput and response time.
> Nginx uses a bit less CPU cycles (10-20%), but it is not able drive  
> Tomcat to 100% CPU.
>
> Here's my take on this performance problem.
>
> 1. Lack of AJP support, which is an optimized HTTP protocol
>    First of all, this is a serious bottleneck.
>
>    * AJP has much less communication overhead than HTTP
>
> 2. Lack of HTTP KeepAlive support for proxy
>
>    * Lack of AJP may be compensated with HTTP keepalive support since  
> there are
>      at least twice the number of TIME_WAIT sockets (connection  
> establishment mean time
>      is at least twice - three times slower than that of Apache)

I believe Tomcat has noticeble overhead in connection 
establishment code (though never tested it myself), so lack of 
backend keepalive support may be an issue in your case.

System overhead shouldn't be noticable (it's about 1 ms on usual 
backend networks).

> 3. Lack of connection pooling
>
>    * Ey-balancer makes things a bit easier, response times are stable, 
> but still the same
>      average TPS and response time.
>
> 4. There seems to be a huge bug in connection management code
>
>    Two mix of transactions: 20K HTML serving and 8K HTML with  
> intentional 200ms delay in Tomcat logic
>
>    With Apache, 20K HTML serving took  36 ms on average while 8K HTML  
> took 258 ms
>    With Nginx,  20K HTML serving took 600 ms on average while 8K HTML  
> took 817 ms
>
>    I really cannot explain these difference. Not even TCP connection  
> overhead or lack of AJP.

I believe this time differences has something to do with how 
Tomcat handles connection establishment / close.

Currently nginx will consider request complete only after 
backend's connection close, and if Tomcat waits a bit for some 
reason before closing connection - this may lead to above numbers.

You may try to capture single request between nginx and Tomcat 
with tcpdump to prove this.

Maxim Dounin

> My questions is "should I abandon nginx at this point"?
> I know nginx is great proxy and static file server but I cannot prove my 
> point with Tomcat over and over again.
>
> Thank you
>
> Chang
>
>
>
>