BIG requests/responses to POST and post_handler return value

[Antoine BONAVITA]

First, a BIG thanks to you and Maxim for helping out newbies like me around. It 
feels good to talk to people who know what they are talking about yet take the 
time to share their knwoledge.


Now, I officially adopted valgrind + no-pool and am trying to configure this 
with Eclipse. That will definitely be very helpful.

Test::Nginx looks good too. To be honest, when I looked at the source code for 
the echo module I looked at the t directory and got scared. And I went to 
writing my test cases in Python. On second look, your library looks much better 
(especially the config/test relationship which is very explicit in the t files). 
I guess I just will have to learn the syntax. I'll try to move my test cases to 
this. I'll ask for help if I get lost in the process.

Looked at taobao.com on google trends. Looks like you guys are handling a little 
bit of load and know what scalable means... ;) Impressive.

Antoine.


----- Original Message ----
> From: agentzh <agentzh at gmail.com>
> To: Antoine BONAVITA <antoine_bonavita at yahoo.com>
> Cc: nginx-devel at nginx.org
> Sent: Mon, February 28, 2011 5:04:05 AM
> Subject: Re: BIG requests/responses to POST and post_handler return value
> 
> On Thu, Feb 24, 2011 at 1:01 AM, Antoine BONAVITA
> <antoine_bonavita at yahoo.com>  wrote:
> > The good old UTSL. Yes, I do that a lot. Except that the code is  not the 
>easiest
> > thing I had to read. And it takes some time/effort to  get used to the 
style,
> > finding what the usual suspects are  (ngx_http_finalize_request,
> > ngx_http_terminate_request, etc.) and  understanding what they are supposed 
>to
> > do.
> 
> Right :)
> 
> >  But, from having done the same with proprietary software in the past (M$, 
>not  to
> > name it), being able to read the code (and debug it) is definitely  
better...
> >
> 
> *GRIN*
> 
> > Actually, I was considering myself  lucky to have run into the bug. It 
>appears
> > only on POST requests with  big request and big response. All my other tests 
>were
> > fine...
> 
> I  want to say more on this.
> 
> It's especially common to get (simple) tests  passing with totally
> broken C code in the context of nginx development,  because of its
> streaming and non-blocking and asynchronous nature. And  nginx's memory
> pool can make things even worse: that is, its various  optimizations
> for space and time tragically prevent tools like valgrind's  memcheck
> from spotting lots of memory-related bugs.
> 
> To overcome these  problems and to find as many bugs as possible in
> dozens of our nginx C  modules, we've developed some tools to help
> uncovering bugs. To be honest, I  was even meant to write an article on
> this topic, so just to name a few  here:
> 
> 1. etcproxy, written by chaoslawful in pure Erlang for emulate  extreme
> network conditions:
> 
>         https://github.com/chaoslawful/etcproxy
> 
>     this simple  TCP proxy has helped us capture lots of state-machine
> related bugs in  ngx_drizzle, ngx_rds_json,
>     ngx_srcache, ngx_form_input, and  even 3rd-party C libraries like
> libdrizzle. There's a (Chinese) post  analyzing a real-world case for
> using this
>    tool to spot a serious  and weird bug in ngx_drizzle:
> 
>          
>http://agentzh.blogspot.com/2010/01/erlang-tcp-proxy-etcproxy-ngxdrizzle.html
> 
> 2.  the no-pool patch from one of my intern students, shrimp:
> 
>          https://github.com/shrimp/no-pool-nginx
> 
>    This simple patch  can disable the nginx memory pool altogether, and
> thus help valgrind's  memcheck to
>    find a lot more memmory invalid read/write and  double-free errors
> that used to be hidden
>    by the memory pool's  optimizations. This patch is for devel version
> of nginx only. One  surely
>    don't want to disable the pool for production ;)
> 
> 3. our  Test::Nginx test scaffold on CPAN for nginx C development
> written in pure  Perl:
> 
>      http://search.cpan.org/perldoc?Test::Nginx
> 
>    This test driver  supports pure declarative test case syntax that
> does *not* require any real  Perl knowledge and our
>    QA enineers who do not know Perl at all have  been offering lots of
> test cases to us.
> 
>    Piotr Sikora has  established a test farm with a small hybrid
> cluster based on lots of  functionalities provided by Test::Nginx.
> 
> These have formed the nginx  development tool-chain used extensively
> here in Taobao.com's Quantum Team. And we cannot live  without it if we
> want to ensure our nginx server running 7x24 hours without  any leaks
> and crashes.
> 
> > I try to avoid luck as much as I can. I  always say "untested code
> > doesn't work". But you cannot test a test case  you don't know to exist...
> 
> Creating (clever) test cases for our C code  written for nginx usually
> cost us much more time ;)
> 
> > If you have  any recommendation on the best way to get going with this thing, 
>I'm
> > all  ears...
> >
> 
> Please see above ;)
> 
> I'm sorry if my previous mail  sounds like some kind of destructive noises.
> 
> > Noticed that. "Devil is  in the details" should be the tagline for nginx... 
>;)
> >
> 
> I cannot  agree more ;) Fortunately there's so many good-quality
> 3rd-party C modules  out there for reference and so many good debugging
> and testing tools that  have been proven useful in practice :D
> 
> Happy hacking!
> -agentzh
>