Buffers Chain

Mon May 26 08:28:50 UTC 2014

Yes, connecting with SOCK_NONBLOCK shouldnt block. I don't believe this is
mentioned previously. If your code blocks (e.g blocking connect or blocking
send) then it would reduce nginx throughput substantially. Thats the point
I was trying to make.

Have you investigated srcache (
https://github.com/openresty/srcache-nginx-module/)? srcache_store could be
used without srcache_fetch to store only. Your store handler could be a
proxy_pass call to pass a restful server or whatever you use for storing
the HTML responses.

Perhaps a better solution would be to use the nginx lua extension instead
(less of a hack).

Regards,
Mathew

On Mon, May 26, 2014 at 6:18 PM, Paulo Silva <pauloasilva at gmail.com> wrote:

> On Mon, May 26, 2014 at 9:09 AM, SplitIce <mat999 at gmail.com> wrote:
> > As in blocking send and connect? I don't know the specifics of Unix
> Sockets,
> > but don't they block when the buffer fills (I know FIFO queues do)?
> >
>
> Sorry, I don't fully understand your question.
> I was expecting that with the SOCK_NONBLOCK it would not block.
>
> What would be your approach?
> Do you know about any nginx internal mechanism to accomplish this goal
> (get the upstream response body out of nginx)?
>
> >
> > On Mon, May 26, 2014 at 9:22 AM, Paulo Silva <pauloasilva at gmail.com>
> wrote:
> >>
> >> Hi,
> >> I'm not sure whether I will face problems with other filters modifying
> >> the response body after mine, but for know I'm comfortable as I can
> >> rebuild the full response body just iterating buffers chains.
> >>
> >> As I said before I'm using nginx as reverse proxy and my main goal is
> >> to pass the upstream (proxy_pass) response to another local process
> >> (relative to nginx).
> >>
> >> I am benchmarking Unix sockets and Shared memory as IPC.
> >> I did it already for Unix Sockets and with my prototype the nginx
> >> "performance" dropped for half the number of requests per second. Of
> >> course I'm doing something really bad.
> >>
> >> Is it OK to use socket/connect/send from inside an nginx module?
> >>
> >> I would be glad to hear from you.
> >> Thanks,
> >>
> >> On Fri, May 23, 2014 at 2:50 PM, Paulo Silva <pauloasilva at gmail.com>
> >> wrote:
> >> > Because I don't have deep knowledge of nginx internal and I can not
> >> > find a proper resource about it, the best I can do and with what I am
> >> > comfortable is with body_filter.
> >> >
> >> > Do you think I can notice whether all other 3rd party module filters
> >> > finish modifying the ngx_chain_t *in ?
> >> >
> >> >
> >> >
> >> > On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin <mdounin at mdounin.ru>
> >> > wrote:
> >> >> Hello!
> >> >>
> >> >> On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin <mdounin at mdounin.ru>
> >> >>> wrote:
> >> >>> > Hello!
> >> >>> >
> >> >>> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote:
> >> >>> >
> >> >>> >> there is other option than modify the auto/modules file?
> >> >>> >>
> >> >>> >> According to my goal (capture the full request response body) I
> >> >>> >> would
> >> >>> >> say that my module must run right before the postpone.
> >> >>> >
> >> >>> > Before the postpone filter you'll get subrequest bodies in your
> >> >>> > filter, which is probably not what you want (the postpone filter
> >> >>> > is to glue subrequest together, correctly ordered).
> >> >>> >
> >> >>> >> Am I supposed to modify the auto/modules like follows?
> >> >>> >>
> >> >>> >> if [ $HTTP_POSTPONE = YES ]; then
> >> >>> >>     HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES
> >> >>> >> $HTTP_POSTPONE_FILTER_MODULE"
> >> >>> >>     HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS"
> >> >>> >> fi
> >> >>> >>
> >> >>> >> # insert my module here!
> >> >>> >>
> >> >>> >> if [ $HTTP_SSI = YES ]; then
> >> >>> >>     have=NGX_HTTP_SSI . auto/have
> >> >>> >>     HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES
> >> >>> >> $HTTP_SSI_FILTER_MODULE"
> >> >>> >>     HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS"
> >> >>> >>     HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS"
> >> >>> >> fi
> >> >>> >>
> >> >>> >>
> >> >>> >> I did check my modules config file and I did realize that it is
> >> >>> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different queues
> for
> >> >>> >> core modules and addons?
> >> >>> >
> >> >>> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the
> >> >>> > only one currently officially supported for 3rd party modules.
> >> >>> >
> >> >>> > If you want your filter to be called right before/after postpone
> >> >>> > filter, it should be relatively safe to put it into the
> >> >>> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some
> >> >>> > additional checks to make sure the postpone filter is enabled, or
> >> >>> > just a code to enable it unconditionally).
> >> >>> >
> >> >>>
> >> >>> And this is also valid when compiling nginx with the --add-module
> >> >>> flag?
> >> >>> How does config file look like?
> >> >>>
> >> >>> My knowledge is restricted to Emiller's Guide To Nginx Module
> >> >>> Development (http://www.evanmiller.org/nginx-modules-guide.html)
> and a
> >> >>> few debugging hours.
> >> >>
> >> >> Uhm, looking again into auto/modules I think I was wrong, and
> >> >> modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work
> >> >> (added module config scripts are executed later on), you should
> >> >> modify HTTP_FILTER_MODULES variable instead, and put your module
> >> >> into a proper position.
> >> >>
> >> >> Note that the "config" file of a module is just a shell script,
> >> >> and you are free to do more or less anything there.
> >> >>
> >> >> --
> >> >> Maxim Dounin
> >> >> http://nginx.org/
> >> >>
> >> >> _______________________________________________
> >> >> nginx-devel mailing list
> >> >> nginx-devel at nginx.org
> >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel
> >> >
> >> >
> >> >
> >> > --
> >> > Paulo A. Silva
> >> > http://tech.pauloasilva.com
> >> > http://linkedin.com/in/devpauloasilva/
> >>
> >>
> >>
> >> --
> >> Paulo A. Silva
> >> http://tech.pauloasilva.com
> >> http://linkedin.com/in/devpauloasilva/
> >>
> >> _______________________________________________
> >> nginx-devel mailing list
> >> nginx-devel at nginx.org
> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel
> >
> >
> >
> > _______________________________________________
> > nginx-devel mailing list
> > nginx-devel at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
>
>
> --
> Paulo A. Silva
> http://tech.pauloasilva.com
> http://linkedin.com/in/devpauloasilva/
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20140526/ebf9ff62/attachment.html>