Breaking content generation phase into multiple phases or adding custom events in content phase

[Yichun Zhang (agentzh)]

Hello!

On Thu, Jun 18, 2015 at 2:17 AM, Kaustubh Deorukhkar wrote:
> I am working on a custom module where I need to use a third party library
> and make sync/async calls to APIs. I do not have control over what the
> library does internally but any async API call on library would call a
> callback which indicates that content generation phase continue with forming
> response and sending it back to client.
>

If you MUST use this 3rd-party library, then you can check out our
ngx_drizzle [1] (for nonblocking MySQL communication via libdrizzle)
and ngx_postgres [2] (for nonblocking PostgreSQL communication via
libpq) for such 3rd-party library integration examples (both of them
are production ready for years.

But in retrospect, it took a *lot* of developer efforts to get them
exactly right due to the inherent limitations in nginx's upstream
mechanism and you MAY run into bugs in 3rd-party libraries when using
edge-triggered (ET) events (well, we had to work around such issues in
libpq, at least).

The recommended way is to re-implement the wire protocol for I/O
directly in Lua atop the cosocket API [3] provided by the ngx_lua
module (or better, use the openresty bundle directly), in the same
spirit of the existing lua-resty-mysql [4] and lua-resty-redis [5]
libraries out there.

> We want to avoid upstream server model if this is already possible with
> nginx.
>

Both ngx_postgres an ngx_drizzle on based on a good part of the stock
nginx's upstream mechanism. It's easier to reuse it than coding
everything from scratch if you stick with that road.

It's worth mentioning that the cosocket mechanism in ngx_lua is a
*parallel* implementation to the official upstream thing and overcomes
all those limitations in "upstream" and makes things much cleaner and
nicer at least on the Lua land. Still, we inherit most (if not all) of
the good stuff from the "upstream" facility. You can check out the
picture below for some ideas:

http://agentzh.org/misc/slides/nginx-conf-2014/images/cosocket.png

Best regards,
-agentzh

[1] https://github.com/openresty/drizzle-nginx-module
[2] https://github.com/FRiCKLE/ngx_postgres
[3] https://github.com/openresty/lua-nginx-module#ngxsockettcp
[4] https://github.com/openresty/lua-resty-mysql
[5] https://github.com/openresty/lua-resty-redis