Buffers Chain

Mon May 26 09:17:35 UTC 2014

On Mon, May 26, 2014 at 9:28 AM, SplitIce <mat999 at gmail.com> wrote:
> Yes, connecting with SOCK_NONBLOCK shouldnt block. I don't believe this is
> mentioned previously. If your code blocks (e.g blocking connect or blocking
> send) then it would reduce nginx throughput substantially. Thats the point I
> was trying to make.
>

Yap, I didn't mention SOCK_NONBLOCK. Sorry about that.

Although I'm working on a prototype to benchmark, my concern is about
socket() and connect() calls on every nginx filter execution. How
would you manage this?

> Have you investigated srcache
> (https://github.com/openresty/srcache-nginx-module/)? srcache_store could be
> used without srcache_fetch to store only. Your store handler could be a
> proxy_pass call to pass a restful server or whatever you use for storing the
> HTML responses.
>

I didn't know about this module. Thanks, I will have a deep look on it.
Do you think that I can pick the proxy_pass response and store it on
memcache easily?

> Perhaps a better solution would be to use the nginx lua extension instead
> (less of a hack).
>

Do you think that lua extension has advantages regarding an nginx filter module.

Sorry about so many questions but at the beginning, nginx is a all
mystery, and you'll worried to do not break its performance.

Thanks a bunch,

> Regards,
> Mathew
>
>
> On Mon, May 26, 2014 at 6:18 PM, Paulo Silva <pauloasilva at gmail.com> wrote:
>>
>> On Mon, May 26, 2014 at 9:09 AM, SplitIce <mat999 at gmail.com> wrote:
>> > As in blocking send and connect? I don't know the specifics of Unix
>> > Sockets,
>> > but don't they block when the buffer fills (I know FIFO queues do)?
>> >
>>
>> Sorry, I don't fully understand your question.
>> I was expecting that with the SOCK_NONBLOCK it would not block.
>>
>> What would be your approach?
>> Do you know about any nginx internal mechanism to accomplish this goal
>> (get the upstream response body out of nginx)?
>>
>> >
>> > On Mon, May 26, 2014 at 9:22 AM, Paulo Silva <pauloasilva at gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >> I'm not sure whether I will face problems with other filters modifying
>> >> the response body after mine, but for know I'm comfortable as I can
>> >> rebuild the full response body just iterating buffers chains.
>> >>
>> >> As I said before I'm using nginx as reverse proxy and my main goal is
>> >> to pass the upstream (proxy_pass) response to another local process
>> >> (relative to nginx).
>> >>
>> >> I am benchmarking Unix sockets and Shared memory as IPC.
>> >> I did it already for Unix Sockets and with my prototype the nginx
>> >> "performance" dropped for half the number of requests per second. Of
>> >> course I'm doing something really bad.
>> >>
>> >> Is it OK to use socket/connect/send from inside an nginx module?
>> >>
>> >> I would be glad to hear from you.
>> >> Thanks,
>> >>
>> >> On Fri, May 23, 2014 at 2:50 PM, Paulo Silva <pauloasilva at gmail.com>
>> >> wrote:
>> >> > Because I don't have deep knowledge of nginx internal and I can not
>> >> > find a proper resource about it, the best I can do and with what I am
>> >> > comfortable is with body_filter.
>> >> >
>> >> > Do you think I can notice whether all other 3rd party module filters
>> >> > finish modifying the ngx_chain_t *in ?
>> >> >
>> >> >
>> >> >
>> >> > On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin <mdounin at mdounin.ru>
>> >> > wrote:
>> >> >> Hello!
>> >> >>
>> >> >> On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote:
>> >> >>
>> >> >>> Hi,
>> >> >>>
>> >> >>> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin <mdounin at mdounin.ru>
>> >> >>> wrote:
>> >> >>> > Hello!
>> >> >>> >
>> >> >>> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote:
>> >> >>> >
>> >> >>> >> there is other option than modify the auto/modules file?
>> >> >>> >>
>> >> >>> >> According to my goal (capture the full request response body) I
>> >> >>> >> would
>> >> >>> >> say that my module must run right before the postpone.
>> >> >>> >
>> >> >>> > Before the postpone filter you'll get subrequest bodies in your
>> >> >>> > filter, which is probably not what you want (the postpone filter
>> >> >>> > is to glue subrequest together, correctly ordered).
>> >> >>> >
>> >> >>> >> Am I supposed to modify the auto/modules like follows?
>> >> >>> >>
>> >> >>> >> if [ $HTTP_POSTPONE = YES ]; then
>> >> >>> >>     HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES
>> >> >>> >> $HTTP_POSTPONE_FILTER_MODULE"
>> >> >>> >>     HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS"
>> >> >>> >> fi
>> >> >>> >>
>> >> >>> >> # insert my module here!
>> >> >>> >>
>> >> >>> >> if [ $HTTP_SSI = YES ]; then
>> >> >>> >>     have=NGX_HTTP_SSI . auto/have
>> >> >>> >>     HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES
>> >> >>> >> $HTTP_SSI_FILTER_MODULE"
>> >> >>> >>     HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS"
>> >> >>> >>     HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS"
>> >> >>> >> fi
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> I did check my modules config file and I did realize that it is
>> >> >>> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different queues
>> >> >>> >> for
>> >> >>> >> core modules and addons?
>> >> >>> >
>> >> >>> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the
>> >> >>> > only one currently officially supported for 3rd party modules.
>> >> >>> >
>> >> >>> > If you want your filter to be called right before/after postpone
>> >> >>> > filter, it should be relatively safe to put it into the
>> >> >>> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some
>> >> >>> > additional checks to make sure the postpone filter is enabled, or
>> >> >>> > just a code to enable it unconditionally).
>> >> >>> >
>> >> >>>
>> >> >>> And this is also valid when compiling nginx with the --add-module
>> >> >>> flag?
>> >> >>> How does config file look like?
>> >> >>>
>> >> >>> My knowledge is restricted to Emiller's Guide To Nginx Module
>> >> >>> Development (http://www.evanmiller.org/nginx-modules-guide.html)
>> >> >>> and a
>> >> >>> few debugging hours.
>> >> >>
>> >> >> Uhm, looking again into auto/modules I think I was wrong, and
>> >> >> modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work
>> >> >> (added module config scripts are executed later on), you should
>> >> >> modify HTTP_FILTER_MODULES variable instead, and put your module
>> >> >> into a proper position.
>> >> >>
>> >> >> Note that the "config" file of a module is just a shell script,
>> >> >> and you are free to do more or less anything there.
>> >> >>
>> >> >> --
>> >> >> Maxim Dounin
>> >> >> http://nginx.org/
>> >> >>
>> >> >> _______________________________________________
>> >> >> nginx-devel mailing list
>> >> >> nginx-devel at nginx.org
>> >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Paulo A. Silva
>> >> > http://tech.pauloasilva.com
>> >> > http://linkedin.com/in/devpauloasilva/
>> >>
>> >>
>> >>
>> >> --
>> >> Paulo A. Silva
>> >> http://tech.pauloasilva.com
>> >> http://linkedin.com/in/devpauloasilva/
>> >>
>> >> _______________________________________________
>> >> nginx-devel mailing list
>> >> nginx-devel at nginx.org
>> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>> >
>> >
>> >
>> > _______________________________________________
>> > nginx-devel mailing list
>> > nginx-devel at nginx.org
>> > http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>
>>
>>
>> --
>> Paulo A. Silva
>> http://tech.pauloasilva.com
>> http://linkedin.com/in/devpauloasilva/
>>
>> _______________________________________________
>> nginx-devel mailing list
>> nginx-devel at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
>
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

-- 
Paulo A. Silva
http://tech.pauloasilva.com
http://linkedin.com/in/devpauloasilva/