Buffers Chain

Tue May 27 08:15:28 UTC 2014

Hi,
One more question about buffers chains.

Is it possible to a body_filter to say to the engine to hang up client
streaming?
Maybe say "keep it in memory" (there are a plenty undocumented
ngx_chain_t members).

I'm searching for "</head>" on response body and one possibility is to
get, i.e., "</" on the last buffer of a chain and the "head>" at the
beginning of the next chain. But on the next body_filter call I will
not have access to the previous chain, right?

I was thinking about telling to the engine to hang up the client
streaming whenever a chain ends with a substring of "</head>".
The purpose is to inject some HTML right before the </head> so I have
to manipulate the content of chain->buf.

It is not possible to use SSI, because nginx is being used as reverse
proxy and the backend application doesn't know about nginx.

Do you know about some nginx internals resources other than Evan Millers post?

I would like to say "sorry" about my (possibly) lame questions.
Thanks a bunch,

On Mon, May 26, 2014 at 10:29 AM, SplitIce <mat999 at gmail.com> wrote:
> Theres actually an example using the memcached module (and echo module to
> get the request body). The main advantage to the lua module is if you wanted
> to do something custom (e.g processing) or transmit it over a different
> protocol not available as an extension.
>
>
> On Mon, May 26, 2014 at 7:17 PM, Paulo Silva <pauloasilva at gmail.com> wrote:
>>
>> On Mon, May 26, 2014 at 9:28 AM, SplitIce <mat999 at gmail.com> wrote:
>> > Yes, connecting with SOCK_NONBLOCK shouldnt block. I don't believe this
>> > is
>> > mentioned previously. If your code blocks (e.g blocking connect or
>> > blocking
>> > send) then it would reduce nginx throughput substantially. Thats the
>> > point I
>> > was trying to make.
>> >
>>
>> Yap, I didn't mention SOCK_NONBLOCK. Sorry about that.
>>
>> Although I'm working on a prototype to benchmark, my concern is about
>> socket() and connect() calls on every nginx filter execution. How
>> would you manage this?
>>
>> > Have you investigated srcache
>> > (https://github.com/openresty/srcache-nginx-module/)? srcache_store
>> > could be
>> > used without srcache_fetch to store only. Your store handler could be a
>> > proxy_pass call to pass a restful server or whatever you use for storing
>> > the
>> > HTML responses.
>> >
>>
>> I didn't know about this module. Thanks, I will have a deep look on it.
>> Do you think that I can pick the proxy_pass response and store it on
>> memcache easily?
>>
>> > Perhaps a better solution would be to use the nginx lua extension
>> > instead
>> > (less of a hack).
>> >
>>
>> Do you think that lua extension has advantages regarding an nginx filter
>> module.
>>
>> Sorry about so many questions but at the beginning, nginx is a all
>> mystery, and you'll worried to do not break its performance.
>>
>> Thanks a bunch,
>>
>> > Regards,
>> > Mathew
>> >
>> >
>> > On Mon, May 26, 2014 at 6:18 PM, Paulo Silva <pauloasilva at gmail.com>
>> > wrote:
>> >>
>> >> On Mon, May 26, 2014 at 9:09 AM, SplitIce <mat999 at gmail.com> wrote:
>> >> > As in blocking send and connect? I don't know the specifics of Unix
>> >> > Sockets,
>> >> > but don't they block when the buffer fills (I know FIFO queues do)?
>> >> >
>> >>
>> >> Sorry, I don't fully understand your question.
>> >> I was expecting that with the SOCK_NONBLOCK it would not block.
>> >>
>> >> What would be your approach?
>> >> Do you know about any nginx internal mechanism to accomplish this goal
>> >> (get the upstream response body out of nginx)?
>> >>
>> >> >
>> >> > On Mon, May 26, 2014 at 9:22 AM, Paulo Silva <pauloasilva at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >> I'm not sure whether I will face problems with other filters
>> >> >> modifying
>> >> >> the response body after mine, but for know I'm comfortable as I can
>> >> >> rebuild the full response body just iterating buffers chains.
>> >> >>
>> >> >> As I said before I'm using nginx as reverse proxy and my main goal
>> >> >> is
>> >> >> to pass the upstream (proxy_pass) response to another local process
>> >> >> (relative to nginx).
>> >> >>
>> >> >> I am benchmarking Unix sockets and Shared memory as IPC.
>> >> >> I did it already for Unix Sockets and with my prototype the nginx
>> >> >> "performance" dropped for half the number of requests per second. Of
>> >> >> course I'm doing something really bad.
>> >> >>
>> >> >> Is it OK to use socket/connect/send from inside an nginx module?
>> >> >>
>> >> >> I would be glad to hear from you.
>> >> >> Thanks,
>> >> >>
>> >> >> On Fri, May 23, 2014 at 2:50 PM, Paulo Silva <pauloasilva at gmail.com>
>> >> >> wrote:
>> >> >> > Because I don't have deep knowledge of nginx internal and I can
>> >> >> > not
>> >> >> > find a proper resource about it, the best I can do and with what I
>> >> >> > am
>> >> >> > comfortable is with body_filter.
>> >> >> >
>> >> >> > Do you think I can notice whether all other 3rd party module
>> >> >> > filters
>> >> >> > finish modifying the ngx_chain_t *in ?
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin <mdounin at mdounin.ru>
>> >> >> > wrote:
>> >> >> >> Hello!
>> >> >> >>
>> >> >> >> On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote:
>> >> >> >>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin
>> >> >> >>> <mdounin at mdounin.ru>
>> >> >> >>> wrote:
>> >> >> >>> > Hello!
>> >> >> >>> >
>> >> >> >>> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote:
>> >> >> >>> >
>> >> >> >>> >> there is other option than modify the auto/modules file?
>> >> >> >>> >>
>> >> >> >>> >> According to my goal (capture the full request response body)
>> >> >> >>> >> I
>> >> >> >>> >> would
>> >> >> >>> >> say that my module must run right before the postpone.
>> >> >> >>> >
>> >> >> >>> > Before the postpone filter you'll get subrequest bodies in
>> >> >> >>> > your
>> >> >> >>> > filter, which is probably not what you want (the postpone
>> >> >> >>> > filter
>> >> >> >>> > is to glue subrequest together, correctly ordered).
>> >> >> >>> >
>> >> >> >>> >> Am I supposed to modify the auto/modules like follows?
>> >> >> >>> >>
>> >> >> >>> >> if [ $HTTP_POSTPONE = YES ]; then
>> >> >> >>> >>     HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES
>> >> >> >>> >> $HTTP_POSTPONE_FILTER_MODULE"
>> >> >> >>> >>     HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS"
>> >> >> >>> >> fi
>> >> >> >>> >>
>> >> >> >>> >> # insert my module here!
>> >> >> >>> >>
>> >> >> >>> >> if [ $HTTP_SSI = YES ]; then
>> >> >> >>> >>     have=NGX_HTTP_SSI . auto/have
>> >> >> >>> >>     HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES
>> >> >> >>> >> $HTTP_SSI_FILTER_MODULE"
>> >> >> >>> >>     HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS"
>> >> >> >>> >>     HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS"
>> >> >> >>> >> fi
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> I did check my modules config file and I did realize that it
>> >> >> >>> >> is
>> >> >> >>> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different
>> >> >> >>> >> queues
>> >> >> >>> >> for
>> >> >> >>> >> core modules and addons?
>> >> >> >>> >
>> >> >> >>> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the
>> >> >> >>> > only one currently officially supported for 3rd party modules.
>> >> >> >>> >
>> >> >> >>> > If you want your filter to be called right before/after
>> >> >> >>> > postpone
>> >> >> >>> > filter, it should be relatively safe to put it into the
>> >> >> >>> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with
>> >> >> >>> > some
>> >> >> >>> > additional checks to make sure the postpone filter is enabled,
>> >> >> >>> > or
>> >> >> >>> > just a code to enable it unconditionally).
>> >> >> >>> >
>> >> >> >>>
>> >> >> >>> And this is also valid when compiling nginx with the
>> >> >> >>> --add-module
>> >> >> >>> flag?
>> >> >> >>> How does config file look like?
>> >> >> >>>
>> >> >> >>> My knowledge is restricted to Emiller's Guide To Nginx Module
>> >> >> >>> Development (http://www.evanmiller.org/nginx-modules-guide.html)
>> >> >> >>> and a
>> >> >> >>> few debugging hours.
>> >> >> >>
>> >> >> >> Uhm, looking again into auto/modules I think I was wrong, and
>> >> >> >> modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work
>> >> >> >> (added module config scripts are executed later on), you should
>> >> >> >> modify HTTP_FILTER_MODULES variable instead, and put your module
>> >> >> >> into a proper position.
>> >> >> >>
>> >> >> >> Note that the "config" file of a module is just a shell script,
>> >> >> >> and you are free to do more or less anything there.
>> >> >> >>
>> >> >> >> --
>> >> >> >> Maxim Dounin
>> >> >> >> http://nginx.org/
>> >> >> >>
>> >> >> >> _______________________________________________
>> >> >> >> nginx-devel mailing list
>> >> >> >> nginx-devel at nginx.org
>> >> >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Paulo A. Silva
>> >> >> > http://tech.pauloasilva.com
>> >> >> > http://linkedin.com/in/devpauloasilva/
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Paulo A. Silva
>> >> >> http://tech.pauloasilva.com
>> >> >> http://linkedin.com/in/devpauloasilva/
>> >> >>
>> >> >> _______________________________________________
>> >> >> nginx-devel mailing list
>> >> >> nginx-devel at nginx.org
>> >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > nginx-devel mailing list
>> >> > nginx-devel at nginx.org
>> >> > http://mailman.nginx.org/mailman/listinfo/nginx-devel
>> >>
>> >>
>> >>
>> >> --
>> >> Paulo A. Silva
>> >> http://tech.pauloasilva.com
>> >> http://linkedin.com/in/devpauloasilva/
>> >>
>> >> _______________________________________________
>> >> nginx-devel mailing list
>> >> nginx-devel at nginx.org
>> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>> >
>> >
>> >
>> > _______________________________________________
>> > nginx-devel mailing list
>> > nginx-devel at nginx.org
>> > http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>
>>
>>
>> --
>> Paulo A. Silva
>> http://tech.pauloasilva.com
>> http://linkedin.com/in/devpauloasilva/
>>
>> _______________________________________________
>> nginx-devel mailing list
>> nginx-devel at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
>
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

-- 
Paulo A. Silva
http://tech.pauloasilva.com
http://linkedin.com/in/devpauloasilva/