Communications Sequence with Upstream

[agentzh]

On Sun, Apr 25, 2010 at 4:58 PM, Magnus Lynch <maglyx at gmail.com> wrote:
> Say I have a location /url that I want to construct responses to based
> on a sequence of communications with an upstream, host:port. To be
> specific, the upstream would be a database, and the sequence of
> communications would comprise commands and responses.

We've been doing this kind of things via ngx_echo's echo_location
directive in our ngx_memc and ngx_drizzle's tests suite. Here's some
an example to do sequential memcached command execution against the
ngx_memc upstream module in a single location /main:

    location /main {
        echo_location '/memc?cmd=flush_all';
        echo_location '/memc?key=foo&cmd=set&val=';
        echo_location '/memc?key=foo&cmd=get';
    }
    location /memc {
        set $memc_cmd $arg_cmd;
        set $memc_key $arg_key;
        set $memc_value $arg_val;
        memc_pass 127.0.0.1:11211;
    }

You can find more examples here:
http://github.com/agentzh/memc-nginx-module/blob/master/test/t/

And here's an example for sequential SQL queries against a mysql backend:

    location /test {
        echo_location /mysql "drop table if exists foo";
        echo;
        echo_location /mysql "create table foo (id serial, name
char(10), age integer);";
        echo;
        echo_location /mysql "insert into foo (name, age) values ('', null);";
        echo;
        echo_location /mysql "insert into foo (name, age) values (null, 0);";
        echo;
        echo_location /mysql "select * from foo order by id";
        echo;
    }
    location /mysql {
        drizzle_pass backend;
        drizzle_module_header off;
        drizzle_query $query_string;
        rds_json on;
    }

Responses look like this

    {"errcode":0}
    {"errcode":0}
    {"errcode":0,"insert_id":1,"affected_rows":1}
    {"errcode":0,"insert_id":2,"affected_rows":1}
    [{"id":1,"name":"","age":null},{"id":2,"name":null,"age":0}]

More examples can be seen here:
http://github.com/agentzh/rds-json-nginx-module/blob/master/test/t/sanity.t

> A simple example
> of what I'd like to do would be:
>  1. client communicates message to /url meaning "if key foo exists,
> add 1 to bar"
>  2. module handling /url sends first database command to check if foo
> exists, if not reply "failed" to client, or it does and
>  3. module sends database command to increment bar, replies "succeeded".
>

It reminds me of the "safe incr" sample in my slides for my talk on
nginx.conf scripting:

    http://agentzh.org/misc/slides/nginx-conf-scripting/nginx-conf-scripting.html

(Use the arrow keys on your keyboard to switch pages there.)

> The details are arbitrary, but I would think there are myriad examples
> where you'd want a module to construct a reply based on back and forth
> communications.
>

Check out ngx_echo module to see if it fits your needs:

    http://wiki.nginx.org/NginxHttpEchoModule

> I'm hoping there's some nice way of handling this case, but from I've
> gathered looking at the code it seems oriented to "1 message to
> module, filter as desired, send to upstream, get response, filter
> again, then response back to client."
>

IMHO, Evan Miller's Guide is very limited in expressing nginx's full
power ;) It's just a good old introductory guide anyway :) Still many
thanks to Evan Miller because the guide has helped so many people
including me :)

I must add that the echo_location and echo_subrequest thingies in
ngx_echo are still limited in power :) We're currently releasing the
full power of nginx's subrequests in our ngx_lua module. Soon we'll be
able to do things like this on the Lua land:

     res = ngx.location.capture('/sub1?id=32');
     if (res ...) {
           res2 = ngx.location.capture('/sub2?id=56');
     } else {
           res2 = ngx.subrequest.capture(ngx.HTTP_POST, '/sub2?id=56',
{ body: 'hello' });
     }

And all the capture operations are transparent non-blocking I/O ones
that are based on nginx's subrequests. Thanks to coco lua's C-level
coroutine support!

There will also be equivalences to the echo_location and
echo_location_async directives on the Lua land, i.e.,
ngx.location.echo and ngx.location.echo_async :)

Stay tuned!
-agentzh