Pointers on writing an Nginx module/application which needs to talk to multiple backends (sometimes sequentially, sometimes parallel) to serve a single request

[Ashish S]

Hi,

I am trying to do something similar to what is described in this older
thread:  http://nginx.2469901.n2.nabble.com/How-can-a-content-handler-block-on-certain-events-before-sending-a-response-td5764856.html

Below is what i am trying to achieve (my use-case):

When my nginx process gets a request,  i want to do the following
things in order:

                1. Connect to a postgres database, retrieve some
information based on a subset of the original request args
                2. Connect to a memcache server with subset of info
from request.  If possible, do 1) and 2) in parallel.
                3. Construct an "upstream request" to one of my (http
based) backend service with info from original request + memcache +
postgres ( I wrote this upstream service by modifying the "upstream
proxy" standard module)
                4. Do some additional processing (filtering, ranking
etc) on the response from my backend
                5. Send final response to the caller.
                6. Update cache if needed.


Is there some sample code for modules that might have similarities to
the above, that i can follow and learn?


For #1 above, I am playing around with postgres module code
http://labs.frickle.com/nginx_ngx_postgres/ to communicate with
postgres DB.  Currently, for the sake of learning/experiments, i
defined a separate upstream block for postgres in my config, and also
defined a separate location block for db query, with the above
postgres module hooked up to it, and in my own module i just do a
subrequest to this postgres block.  I followed the eval module code to
do the subrequest (based on suggestions in this older thread
http://forum.nginx.org/read.php?29,114237), but figured that eval does
an in-memory subrequest, while Piotr's postgres module above doesn't
support in-memory subrequest.  Has anyone modified postgres module
code to work with in-memory sub-requests?


I also experimented with a simple case of firing in-memory subrequests
from my module, to fetch a static file (instead of calling postgres
location above), before i do my upstream backend call.  Similar to the
code in eval module
(https://github.com/vkholodkov/nginx-eval-module/blob/master/ngx_http_eval_module.c),
in my ctx object, i have a "done", and "in_progress" flags.  During
the initial call to my handler, ctx->done is not set, and i create and
fire a in-memory subrequest to a url that points to the static file,
and return NGX_DONE.  And in my sub-request handler, i set the
ctx->done, and return NGX_OK. In my main request  handler, if
ctx->done is set,  i have code which will do a call to my upstream
backend (step #3 in my list) . However, the call never returns from
sub-request handler to my main handler. What might i be doing wrong?


I am currently not looking into nginx_lua and other extensions, until
i have a certain level of comfort with stock nginx module development.


I've also tried suggestions and code from Piotr and Shaun, from the
following thread:  http://www.ruby-forum.com/topic/199392 (Parallel
subrequests for multi-source long polling) for creating independent
sub-requests for step #1,  Basically, i create an independent
subrequest during the content phase of my module's handler, and was
hoping that i could (when i get response), setup the hooks for
upstream functionality (create request etc) for my own module. (maybe
i was wrong here)  I had to call the ngx_indep_subreq_postconf in the
sample code,  during the post-configuration phase of MY module, to get
around some initial segfaults.  But now i am stuck at another sigsegv,
with backtrace below:

Program received signal SIGSEGV, Segmentation fault.
0x080607a6 in ngx_event_connect_peer (pc=0x9fbf1c4) at
src/event/ngx_event_connect.c:30
30	    s = ngx_socket(pc->sockaddr->sa_family, SOCK_STREAM, 0);
(gdb) where
#0  0x080607a6 in ngx_event_connect_peer (pc=0x9fbf1c4) at
src/event/ngx_event_connect.c:30
#1  0x08080333 in ngx_http_upstream_connect (r=0x9fd0eac, u=0x9fbf1bc)
at src/http/ngx_http_upstream.c:1104
#2  0x08080da4 in ngx_http_upstream_init_request (r=0x9fd0eac) at
src/http/ngx_http_upstream.c:632
#3  0x0808116b in ngx_http_upstream_init (r=0x9fd0eac) at
src/http/ngx_http_upstream.c:433
#4  0x080aa4c5 in ngx_indep_subreq_init_upstream (r=0x9fd0eac,
url=0xbf965e60, callback=0x80a34cb
<ngx_http_my_broker_subrequest_handler(ngx_buf_t*,
ngx_http_request_t*, ngx_int_t, void*)>,
    callback_data=0x9fd0360, upstream_extensions=0xbf965d88)
#5  0x080aa57f in ngx_indep_subreq_fetch (r=0x9fc9698, pool=0x9fcfe30,
url=0xbf965e60,
    callback=0x80a34cb
<ngx_http_my_broker_subrequest_handler(ngx_buf_t*,
ngx_http_request_t*, ngx_int_t, void*)>, callback_data=0x9fd0360,
upstream_extensions=0xbf965d88)
#6  0x080a51d3 in ngx_http_my_broker_handler (r=0x9fc9698)
#7  0x0806e4c4 in ngx_http_core_content_phase (r=0x9fc9698,
ph=0x9fd846c) at src/http/ngx_http_core_module.c:1367
#8  0x0806a075 in ngx_http_core_run_phases (r=0x9fc9698) at
src/http/ngx_http_core_module.c:862
......


I am trying to see if either sub-requests, or independent subrequests
(with some modifications to the postgres module code)  will suffice
for my use-case, or  will i need to work directly with nginx
events/nginx_event_connect_peer etc.  Any help is appreciated.


Thanks!
Ashish