Use Test::Nginx with etcproxy and/or valgrind (Was Re: Test::Nginx::LWP vs. Test::Nginx::Socket)

[Antoine BONAVITA]

Agentzh,

Thanks a lot, again. I'm going on a ski trip for a week or so. I'll try that 
when I come back.

Antoine.




----- Original Message ----
> From: agentzh <agentzh at gmail.com>
> To: Antoine BONAVITA <antoine_bonavita at yahoo.com>
> Cc: nginx-devel at nginx.org
> Sent: Thu, March 3, 2011 4:49:17 AM
> Subject: Use Test::Nginx with etcproxy and/or valgrind (Was Re: 
>Test::Nginx::LWP vs. Test::Nginx::Socket)
> 
> On Thu, Mar 3, 2011 at 12:37 AM, Antoine BONAVITA
> <antoine_bonavita at yahoo.com>  wrote:
> > Following agentzh tips, I'm moving the test cases for my module  to 
>Test::Nginx
> > (instead of using my python unit  tests).
> >
> 
> There's a lot of undocumented features in  Test::Nginx::Socket. I'm
> sorry. I'd like to document a bit how to integrate  it with etcproxy
> and/or valgrind here because it's so useful ;)
> 
> Use  Test::Nginx::Socket with  etcproxy
> ============================
> 
> Test::Nginx automatically starts  an nginx instance (from the PATH env)
> rooted at t/servroot/ and the default  config template makes this nginx
> instance listen on the port 1984 by  default.
> 
> The default settings in etcproxy [1] makes this small TCP proxy  split
> the TCP packets into bytes and introduce 1ms latency among  them.
> 
> There's usually various TCP chains that we can put etcproxy into,  for example
> 
> Test::Nginx <=>  nginx
> ----------------------------
> 
>    $ ./etcproxy 1234  1984
> 
> Here we tell etcproxy to listen on port 1234 and to delegate all  the
> TCP traffic to the port 1984, the default port that Test::Nginx  makes
> nginx listen to.
> 
> And then we tell Test::Nginx to test against  the port 1234, where
> etcproxy listens on, rather than the port 1984 that  nginx directly
> listens on:
> 
>    $ TEST_NGINX_CLIENT_PORT=1234 prove  -r t/
> 
> Then the TCP chain now looks like this:
> 
>    Test::Nginx  <=> etcproxy (1234) <=> nginx (1984)
> 
> So etcproxy can  effectively emulate extreme network conditions and
> exercise "unusual" code  paths in your nginx server by your tests.
> 
> In practice, *tons* of weird  bugs can be captured by this setting.
> Even ourselves didn't expect that this  simple approach is so
> effective.
> 
> nginx <=>  memcached
> -----------------------------
> 
> We first start the memcached  server daemon on port 11211:
> 
>     memcached -p 11211  -vv
> 
> and then we another etcproxy instance to listen on port 11984 like  this
> 
>     $ ./etcproxy 11984 11211
> 
> Then we tell our  t/foo.t test script to connect to 11984 rather than 11211:
> 
>    #  foo.t
>    use Test::Nginx::Socket;
>    repeat_each(1);
>     plan tests => 2 * repeat_each() * blocks();
>     $ENV{TEST_NGINX_MEMCACHED_PORT} ||= 11211;  # make this env take  a
> default value
>    run_tests();
> 
>    __DATA__
> 
>     === TEST 1: sanity
>    --- config
>    location /foo {
>          set $memc_cmd set;
>         set  $memc_key foo;
>         set $memc_value bar;
>          memc_pass 127.0.0.1:$TEST_NGINX_MEMCACHED_PORT;
>     }
>    --- request
>        GET /foo
>    ---  response_body_like: STORED
> 
> The Test::Nginx library will automatically  expand the special macro
> "$TEST_NGINX_MEMCACHED_PORT" to the environment with  the same name.
> You can define your own $TEST_NGINX_BLAH_BLAH_PORT macros as  long as
> its prefix is TEST_NGINX_ and all in upper case letters.
> 
> And  now we can run your test script against the etcproxy port 11984:
> 
>      TEST_NGINX_MEMCACHED_PORT=11984 prove t/foo.t
> 
> Then the TCP chains  look like this:
> 
>     Test::Nginx <=> nginx (1984)  <=> etcproxy (11984) <=> memcached (11211)
> 
> If  TEST_NGINX_MEMCACHED_PORT is not set, then it will take the default
> value  11211, which is what we want when there's no  etcproxy
> configured:
> 
>     Test::Nginx <=> nginx (1984)  <=> memcached (11211)
> 
> This approach also works for proxied mysql  and postgres traffic.
> Please see the live test suite of ngx_drizzle and  ngx_postgres for
> more details.
> 
> Usually we set both  TEST_NGINX_CLIENT_PORT and
> TEST_NGINX_MEMCACHED_PORT (and etc) at the same  time, effectively
> yielding the following chain:
> 
>      Test::Nginx <=> etcproxy (1234) <=> nginx (1984) <=>  etcproxy
> (11984) <=> memcached (11211)
> 
> as long as you run two  separate etcproxy instances in two separate terminals.
> 
> It's easy to  verify if the traffic actually goes through your etcproxy
> server. Just check  if the terminal running etcproxy emits outputs. By
> default, etcproxy always  dump out the incoming and outgoing data to
> stdout/stderr.
> 
> Use  Test::Nginx::Socket with valgrind  memcheck
> ====================================
> 
> Test::Nginx has  integrated support for valgrind [2] even though by
> default it does not bother  running it with the tests because valgrind
> will significantly slow down the  test sutie.
> 
> First ensure that your valgrind executable visible in your  PATH env.
> And then run your test suite with the TEST_NGINX_USE_VALGRIND env  set
> to true:
> 
>     TEST_NGINX_USE_VALGRIND=1 prove -r  t
> 
> If you see false alarms, you do have a chance to skip them by  defining
> a ./valgrind.suppress file at the root of your module source tree,  as
> in
> 
>     
>https://github.com/chaoslawful/drizzle-nginx-module/blob/master/valgrind.suppress
>
> 
> This  is the suppression file for ngx_drizzle. Test::Nginx will
> automatically use  it to start nginx with valgrind memcheck if this
> file does exist at the  expected location.
> 
> If you do see a lot of "Connection refused" errors  while running the
> tests this way, then you probably have a slow machine (or a  very busy
> one) that the default waiting time is not sufficient for valgrind  to
> start. You can define the sleep time to a larger value by setting  the
> TEST_NGINX_SLEEP env:
> 
>     TEST_NGINX_SLEEP=1 prove -r  t
> 
> The time unit used here is "second". The default sleep setting  just
> fits my ThinkPad (Core2Duo T9600).
> 
> Applying the no-pool patch to  your nginx core is recommended while
> running nginx with  valgrind:
> 
>     https://github.com/shrimp/no-pool-nginx
> 
> The nginx  memory pool can prevent valgrind from spotting lots of
> invalid memory  reads/writes as well as certain double-free errors. We
> did find a lot more  memory issues in many of our modules when we first
> introduced the no-pool  patch in practice ;)
> 
> There's also more advanced features in Test::Nginx  that have never
> documented. I'd like to write more about them in the near  future ;)
> 
> Cheers,
> -agentzh
> 
> References
> 
> [1] etcproxy: https://github.com/chaoslawful/etcproxy
> [2] valgrind:  http://valgrind.org/
>