[ANNOUNCE] ngx_drizzle v0.0.11 and ngx_rds_json v0.09: significant performance boost

[agentzh]

Hi, folks!

I'm happy to announce that ngx_drizzle v0.0.11 and ngx_rds_json v0.09
are finally out:

    http://github.com/chaoslawful/drizzle-nginx-module/tarball/v0.0.11

    http://github.com/agentzh/rds-json-nginx-module/tarball/v0.09

ngx_drizzle is an upstream module that talks to mysql, drizzle, and
sqlite3 by libdrizzle, and generates output in a binary format known
as RDS (Resty DBD Stream), just like Piotr Sikora's ngx_postgres
module.

ngx_rds_json is an output filter module that converts the RDS outputs
of ngx_drizzle and ngx_postgres, to plain JSON text.

The highlight of these releases are *significant* performance boost
due to extensive refactoring and optimizations in these two modules
recently.

One can observe hundreds of times of improvement for big responses
above 300KB for both ngx_drizzle + ngx_rds_json and ngx_postgres +
ngx_rds_json combination.

We even observed that ngx_drizzle + ngx_rds_json achieved 128MB/sec
(Yes, 1024Mbit/sec!) for a 380KB data-set query while connecting to a
simple mysql server, about 3 ~ 4 times as fast as php + libmysql in
this setting.

Technically, we (partially) adapted the "fixed-size bufs" model used
in ngx_http_gzip_filter_module (and elsewhere in the nginx core) in
both of our modules, effectively eliminating lots of unnecessary
packet splitting and buffer allocations, which contributes most of the
performance boost. And we'd like ngx_postgres to apply this technique
in the near future too (for a typical 470 KB data-set query,
ngx_drizzle + ngx_rds_json is now more than 50% faster than
ngx_postgres + ngx_rds_json).

We also introduce the drizzle_buffer_size and rds_json_buffer_size
directives to allow the user adjust the size of each buf that is used
in the output emitter:

     drizzle_buffer_size 4k;
     rds_json_buffer_size 4k;

The default setting is the page-size, usually 4k ~ 8k. The bigger the
buffer size, the less streamming the output will be.

At last but not least, this release of ngx_drizzle also includes
various new features ported from ngx_postgres, like the
"method-specific queries" support in the drizzle_query directive.

Here's an example that implements a full RESTful interface to a mysql
backend using a single nginx location, and no "if hacks":

   location ~ '^/cat/(\d+)' {
       set $id $1;
       set_form_input $name;
       set_quote_sql_str $quoted_name $name;

       drizzle_query HEAD GET "select * from cats where id=$id";
       drizzle_query DELETE "delete from cats where id=$id";
       drizzle_query POST "insert into cats (name) values($quoted_name)";

       drizzle_pass my_mysql_backend;
   }

There's also an example for ngx_postgres in my slides:

    http://agentzh.org/misc/slides/recent-dev-nginx-conf/#18

Have fun!
-agentzh