Breaking content generation phase into multiple phases or adding custom events in content phase

[Jeff Kaufman]

ngx_pagespeed does this by giving nginx a pipe to watch, setting up a
handler for that pipe, calling an async api that uses threads, then
the from the callback writing a byte to the pipe.  Now when the async
code finishes we're back on the nginx event loop in the pipe's
handler.

On Wed, Jun 17, 2015 at 11:27 PM, Yichun Zhang (agentzh)
<agentzh at gmail.com> wrote:
> Hello!
>
> On Thu, Jun 18, 2015 at 2:17 AM, Kaustubh Deorukhkar wrote:
>> I am working on a custom module where I need to use a third party library
>> and make sync/async calls to APIs. I do not have control over what the
>> library does internally but any async API call on library would call a
>> callback which indicates that content generation phase continue with forming
>> response and sending it back to client.
>>
>
> If you MUST use this 3rd-party library, then you can check out our
> ngx_drizzle [1] (for nonblocking MySQL communication via libdrizzle)
> and ngx_postgres [2] (for nonblocking PostgreSQL communication via
> libpq) for such 3rd-party library integration examples (both of them
> are production ready for years.
>
> But in retrospect, it took a *lot* of developer efforts to get them
> exactly right due to the inherent limitations in nginx's upstream
> mechanism and you MAY run into bugs in 3rd-party libraries when using
> edge-triggered (ET) events (well, we had to work around such issues in
> libpq, at least).
>
> The recommended way is to re-implement the wire protocol for I/O
> directly in Lua atop the cosocket API [3] provided by the ngx_lua
> module (or better, use the openresty bundle directly), in the same
> spirit of the existing lua-resty-mysql [4] and lua-resty-redis [5]
> libraries out there.
>
>> We want to avoid upstream server model if this is already possible with
>> nginx.
>>
>
> Both ngx_postgres an ngx_drizzle on based on a good part of the stock
> nginx's upstream mechanism. It's easier to reuse it than coding
> everything from scratch if you stick with that road.
>
> It's worth mentioning that the cosocket mechanism in ngx_lua is a
> *parallel* implementation to the official upstream thing and overcomes
> all those limitations in "upstream" and makes things much cleaner and
> nicer at least on the Lua land. Still, we inherit most (if not all) of
> the good stuff from the "upstream" facility. You can check out the
> picture below for some ideas:
>
> http://agentzh.org/misc/slides/nginx-conf-2014/images/cosocket.png
>
> Best regards,
> -agentzh
>
> [1] https://github.com/openresty/drizzle-nginx-module
> [2] https://github.com/FRiCKLE/ngx_postgres
> [3] https://github.com/openresty/lua-nginx-module#ngxsockettcp
> [4] https://github.com/openresty/lua-resty-mysql
> [5] https://github.com/openresty/lua-resty-redis
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel