Bad performance of nginx with Tomcat vs. Apache with Tomcat

[István]

well we can argue on the tools you are using but the point is that you have
an obvious performance drop somehow :)


better to move to that direction to be able to find, fix it.

however i have no problem if you are using apache further, just don't see
the point to share your experience here if you don't want to share the
config


regards,
Istvan

On Fri, Sep 4, 2009 at 12:36 AM, Chang Song <changsong at me.com> wrote:

>
>
> Hi, Istvan
> It is not about 50K single node nginx throughput.
> I have a standard TCP/IP tuning settings.
>
> We can reach 70K throughput in some workloads.
> THey ALL depends on workloads.
>
> I have not given all the details so that why you are saying that but
> we are measure and capturing every possible system resource under /proc.
>
> dstat does not tell you everything.
> We are currently using collectl and collectd, and captures everything under
> /proc
>
> This is a sample of nginx access log (proxy service time and nginx service
> time is there)
>
> [03/Sep/2009:12:19:02 +0900] 10.25.131.46 200 8254 gzip:-% conns:199229
> up_response_t:0.392 svc_t:0.831 "GET /index_think.jsp HTTP/1.1"
> [03/Sep/2009:12:19:02 +0900] 10.25.131.48 200 20524 gzip:-% conns:199622
> up_response_t:0.150 svc_t:0.668 "GET /static/20k.jsp HTTP/1.1"
>
> I don't think we are in a position to debug deeper into nginx since we need
> to move on.
> Thanks
>
>
>
> On Sep 4, 2009, at 7:19 AM, István wrote:
>
> I see,
> well I was able to reach about 50K req/s on a single node with nginx with
> tuning linux/tcp stack/nginx and i learned one thing:
>
> measure instead of guess
>
> (and as a side effect: debug instead of guess.)
>
> So, if i were you i would start dstat(dstat -cgilpymn) on that host and
> see the different elements of you system, enable debug logging, even
> stracing nginx
>
> This is the way I think.
>
> Regards,
> Istvan
>
>
>
> On Thu, Sep 3, 2009 at 10:51 PM, Chang Song <changsong at me.com> wrote:
>
>>
>> Istvan.
>> It didn't really matter what config I used.
>>
>> I tried all combinations of
>>
>> worker_process (2-512)
>> worker_connections (512-16000)
>> accept_mutex (on/off)
>> tcp_nopush (on/off)
>> tcp_nodelay (on/off)
>> proxy_buffer* (various sizes)
>> and other proxy related parameters you can imagine.
>> The one I showed you has the best performance
>> The following showed the best performance across the board
>>
>> worker_process 2;           # since we have 2 core machine
>> worker_connections 16000;
>> accept_mutex off;
>> max_connections 256;        # ey-balancer (tomcat had 512 threads)
>> everything else default
>>
>> Thank you.
>>
>>
>>
>> On Sep 3, 2009, at 4:26 PM, István wrote:
>>
>> I think it would be beneficial to show us your nginx config :)
>>
>> Regards,
>> Istvan
>>
>> On Thu, Sep 3, 2009 at 5:50 AM, Chang Song <changsong at me.com> wrote:
>>
>>> Sometime ago, I posted an message about Nginx performance when paired
>>> with Tomcat.
>>> We recently did extensive in-house testing of various workload against
>>> Nginx with Tomcat vs Apache vs Tomcat.
>>>
>>> Apache wins hands down.
>>>
>>> Here's the basic setup
>>>
>>> 1. Nginx (2 proc/8192 connections) -> http/1.0 -> Tomcat (HTTP connector)
>>> 2. Apache (512 prefork) -> AJP -> Tomcat (AJP)
>>>
>>> Both KeepAlive off (we don't use KeepAlive due to L4 switch)
>>>
>>> The physical server is 2 core Intel Xeon, which is typical web server
>>> config here.
>>> We have three grinder 3.2 load generators.
>>> We tested 4K and 20K Tomcat simple HTML file, 20K simple HTML with
>>> intentional 10% 200ms
>>> sleep in Tomcat serving (emulate slow DB query), etc.
>>>
>>> Every single case, Apache wins by at least 10-15%.
>>> Throughput and response time.
>>> Nginx uses a bit less CPU cycles (10-20%), but it is not able drive
>>> Tomcat to 100% CPU.
>>>
>>> Here's my take on this performance problem.
>>>
>>> 1. Lack of AJP support, which is an optimized HTTP protocol
>>>   First of all, this is a serious bottleneck.
>>>
>>>   * AJP has much less communication overhead than HTTP
>>>
>>> 2. Lack of HTTP KeepAlive support for proxy
>>>
>>>   * Lack of AJP may be compensated with HTTP keepalive support since
>>> there are
>>>     at least twice the number of TIME_WAIT sockets (connection
>>> establishment mean time
>>>     is at least twice - three times slower than that of Apache)
>>>
>>> 3. Lack of connection pooling
>>>
>>>   * Ey-balancer makes things a bit easier, response times are stable, but
>>> still the same
>>>     average TPS and response time.
>>>
>>> 4. There seems to be a huge bug in connection management code
>>>
>>>   Two mix of transactions: 20K HTML serving and 8K HTML with intentional
>>> 200ms delay in Tomcat logic
>>>
>>>   With Apache, 20K HTML serving took  36 ms on average while 8K HTML took
>>> 258 ms
>>>   With Nginx,  20K HTML serving took 600 ms on average while 8K HTML took
>>> 817 ms
>>>
>>>   I really cannot explain these difference. Not even TCP connection
>>> overhead or lack of AJP.
>>>
>>> My questions is "should I abandon nginx at this point"?
>>> I know nginx is great proxy and static file server but I cannot prove my
>>> point with Tomcat over and over again.
>>>
>>> Thank you
>>>
>>> Chang
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> the sun shines for all
>>
>>
>>
>
>
> --
> the sun shines for all
>
>
>


-- 
the sun shines for all
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://nginx.org/pipermail/nginx/attachments/20090904/44b0706a/attachment.html>