BIG requests/responses to POST and post_handler return value

[agentzh]

On Thu, Feb 24, 2011 at 1:01 AM, Antoine BONAVITA
<antoine_bonavita at yahoo.com> wrote:
> The good old UTSL. Yes, I do that a lot. Except that the code is not the easiest
> thing I had to read. And it takes some time/effort to get used to the style,
> finding what the usual suspects are (ngx_http_finalize_request,
> ngx_http_terminate_request, etc.) and understanding what they are supposed to
> do.

Right :)

> But, from having done the same with proprietary software in the past (M$, not to
> name it), being able to read the code (and debug it) is definitely better...
>

*GRIN*

> Actually, I was considering myself lucky to have run into the bug. It appears
> only on POST requests with big request and big response. All my other tests were
> fine...

I want to say more on this.

It's especially common to get (simple) tests passing with totally
broken C code in the context of nginx development, because of its
streaming and non-blocking and asynchronous nature. And nginx's memory
pool can make things even worse: that is, its various optimizations
for space and time tragically prevent tools like valgrind's memcheck
from spotting lots of memory-related bugs.

To overcome these problems and to find as many bugs as possible in
dozens of our nginx C modules, we've developed some tools to help
uncovering bugs. To be honest, I was even meant to write an article on
this topic, so just to name a few here:

1. etcproxy, written by chaoslawful in pure Erlang for emulate extreme
network conditions:

        https://github.com/chaoslawful/etcproxy

    this simple TCP proxy has helped us capture lots of state-machine
related bugs in ngx_drizzle, ngx_rds_json,
    ngx_srcache, ngx_form_input, and even 3rd-party C libraries like
libdrizzle. There's a (Chinese) post analyzing a real-world case for
using this
   tool to spot a serious and weird bug in ngx_drizzle:

        http://agentzh.blogspot.com/2010/01/erlang-tcp-proxy-etcproxy-ngxdrizzle.html

2. the no-pool patch from one of my intern students, shrimp:

        https://github.com/shrimp/no-pool-nginx

   This simple patch can disable the nginx memory pool altogether, and
thus help valgrind's memcheck to
   find a lot more memmory invalid read/write and double-free errors
that used to be hidden
   by the memory pool's optimizations. This patch is for devel version
of nginx only. One surely
   don't want to disable the pool for production ;)

3. our Test::Nginx test scaffold on CPAN for nginx C development
written in pure Perl:

    http://search.cpan.org/perldoc?Test::Nginx

   This test driver supports pure declarative test case syntax that
does *not* require any real Perl knowledge and our
   QA enineers who do not know Perl at all have been offering lots of
test cases to us.

   Piotr Sikora has established a test farm with a small hybrid
cluster based on lots of functionalities provided by Test::Nginx.

These have formed the nginx development tool-chain used extensively
here in Taobao.com's Quantum Team. And we cannot live without it if we
want to ensure our nginx server running 7x24 hours without any leaks
and crashes.

> I try to avoid luck as much as I can. I always say "untested code
> doesn't work". But you cannot test a test case you don't know to exist...

Creating (clever) test cases for our C code written for nginx usually
cost us much more time ;)

> If you have any recommendation on the best way to get going with this thing, I'm
> all ears...
>

Please see above ;)

I'm sorry if my previous mail sounds like some kind of destructive noises.

> Noticed that. "Devil is in the details" should be the tagline for nginx... ;)
>

I cannot agree more ;) Fortunately there's so many good-quality
3rd-party C modules out there for reference and so many good debugging
and testing tools that have been proven useful in practice :D

Happy hacking!
-agentzh