Processing responses of unbounded sizes from upstream servers

[Maxime Henrion]

Hello all,


I am currently developing a module that has to send a number of subrequests to upstream servers, and aggregate them through application logic. I am currently doing that through a post-subrequest handler, using the NGX_HTTP_SUBREQUEST_IN_MEMORY flag. My problem is that it is possible to receive very large responses from the upstream servers, and I end up with the "upstream buffer is too small" error, even after bumping the buffer sizes a number of times.

It is my understanding that if I drop this subrequest flag, nginx wouldn't try to make the response fit in a single buffer anymore, but then I have no idea how to get at that buffer chain - my post-subrequest handler only knows about the single buffer in the upstream structure and I haven't been able to locate a piece of code that would do things differently.

I suppose it would be possible to use an output filter instead of a post-subrequest handler for that use case, would that make sense? And last but not least, if I go down that road, can I just move my module "declaration" to the HTTP_AUX_FILTER_MODULES variable (from HTTP_MODULES), and still have the rest of module work fine, or will I need to use a second module for that?

Thanks a lot in advance!
Maxime