Processing responses of unbounded sizes from upstream servers

[Sorin Manole]

Hey Maxime! The solution you described makes sense and should work. And
yes, you should probably use HTTP_AUX_FILTER_MODULES to register your
handler.
You will have to do a little more juggling with buffers however. From what
I remember, the default behaviour when using not NGX_HTTP_SUBREQUEST_IN_MEMORY
is to pass on the subrequest response to the client. You probably don't
want that, so you will have to catch the response buffers in the output
handler and not pass anything to the next filter.
https://trac.nginx.org/nginx/browser/nginx/src/http/modules/ngx_http_ssi_filter_module.c
is probably the best source of inspiration for this.
Anyway, good luck!

2015-10-22 15:30 GMT+03:00 Maxime Henrion <mhenrion at appnexus.com>:

> Hello all,
>
>
> I am currently developing a module that has to send a number of
> subrequests to upstream servers, and aggregate them through application
> logic. I am currently doing that through a post-subrequest handler, using
> the NGX_HTTP_SUBREQUEST_IN_MEMORY flag. My problem is that it is possible
> to receive very large responses from the upstream servers, and I end up
> with the "upstream buffer is too small" error, even after bumping the
> buffer sizes a number of times.
>
> It is my understanding that if I drop this subrequest flag, nginx wouldn't
> try to make the response fit in a single buffer anymore, but then I have no
> idea how to get at that buffer chain - my post-subrequest handler only
> knows about the single buffer in the upstream structure and I haven't been
> able to locate a piece of code that would do things differently.
>
> I suppose it would be possible to use an output filter instead of a
> post-subrequest handler for that use case, would that make sense? And last
> but not least, if I go down that road, can I just move my module
> "declaration" to the HTTP_AUX_FILTER_MODULES variable (from HTTP_MODULES),
> and still have the rest of module work fine, or will I need to use a second
> module for that?
>
> Thanks a lot in advance!
> Maxime
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151026/c62c7075/attachment.html>