Processing responses of unbounded sizes from upstream servers

[Maxime Henrion]

I realized that the "pos" field is actually pointing right at the body of that response, so that solves one of my issues.

I would however appreciate any documentation on those numerous flags sometimes set in buffers and their actual meaning.

Maxime
________________________________
From: nginx-devel [nginx-devel-bounces at nginx.org] on behalf of Maxime Henrion [mhenrion at appnexus.com]
Sent: Tuesday, October 27, 2015 5:00 PM
To: nginx-devel at nginx.org
Subject: RE: Processing responses of unbounded sizes from upstream servers

Hey Sorin!

First and foremost, thanks a lot for your answer.

I started working on implementing a body filter for my needs, and am already facing some strange issues. When my body filter is called on the subrequest I sent, despite it being a body filter, I get a buffer chain containing the whole upstream response, including headers. Since this is not a header filter, this has gotten me quite confused... For what it's worth, my module doesn't even register a header filter. Here is some gdb output:


Breakpoint 1, ngx_http_dispatcher_body_filter (r=0x5402050, in=0x5403b38) at /home/mhenrion/dispatcher/ngx_http_dispatcher_module.c:207
207 {
(gdb) p *in->buf
$1 = {pos = 0x5404441 "N", last = 0x5404498 "", file_pos = 0, file_last = 0,
  start = 0x54043c0 "HTTP/1.1 200 OK\r\n[more data here that I stripped]",
  end = 0x54083c0 "", tag = 0x7671e0 <ngx_http_proxy_module>, file = 0x0, shadow = 0x5403698, temporary = 1, memory = 0, mmap = 0, recycled = 0, in_file = 0, flush = 0,
  sync = 0, last_buf = 0, last_in_chain = 0, last_shadow = 1, temp_file = 0, num = 0}
(gdb)

Unless I'm missing something, this makes it a lot harder for me to actually work on the body of those responses, as I would end up having to parse the HTTP header to find the part I'm interested in.

I'm also at a loss as to the semantics of the last_buf and last_in_chain flags. The module development guide doesn't contain any information about the "last_in_chain" flag; it only talks about last_buf as a marker indicating that the buffer chain is complete. Intuitively, I would think of "last_in_chain" as a marker that we're at the end of the buffer list, but that doesn't make a lot of sense since this would be redundant with the next pointer being NULL. The ngx_buf.h header is equally unhelpful there and I couldn't locate a piece of code that would make the semantics clear yet (some modules seem to consider both flags as being equal, while others aren't).

Last but not least, is there any documentation as to when exactly the output filters are being called? Mine seems to be called two times for the same subrequest. Once with the buffer that I showed above, and a second time with a buffer that is mostly all zero'ed, except for the "sync" and "last_in_chain" flags both set to 1. That seems to be some kind of a flush request, but again, I don't see any documentation on that.

Thanks in advance,
Maxime
________________________________
From: nginx-devel [nginx-devel-bounces at nginx.org] on behalf of Sorin Manole [sorin.v.manole at gmail.com]
Sent: Monday, October 26, 2015 8:05 PM
To: nginx-devel at nginx.org
Subject: Re: Processing responses of unbounded sizes from upstream servers

Hey Maxime! The solution you described makes sense and should work. And yes, you should probably use HTTP_AUX_FILTER_MODULES to register your handler.
You will have to do a little more juggling with buffers however. From what I remember, the default behaviour when using not NGX_HTTP_SUBREQUEST_IN_MEMORY is to pass on the subrequest response to the client. You probably don't want that, so you will have to catch the response buffers in the output handler and not pass anything to the next filter.
https://trac.nginx.org/nginx/browser/nginx/src/http/modules/ngx_http_ssi_filter_module.c is probably the best source of inspiration for this.
Anyway, good luck!

2015-10-22 15:30 GMT+03:00 Maxime Henrion <mhenrion at appnexus.com<mailto:mhenrion at appnexus.com>>:
Hello all,


I am currently developing a module that has to send a number of subrequests to upstream servers, and aggregate them through application logic. I am currently doing that through a post-subrequest handler, using the NGX_HTTP_SUBREQUEST_IN_MEMORY flag. My problem is that it is possible to receive very large responses from the upstream servers, and I end up with the "upstream buffer is too small" error, even after bumping the buffer sizes a number of times.

It is my understanding that if I drop this subrequest flag, nginx wouldn't try to make the response fit in a single buffer anymore, but then I have no idea how to get at that buffer chain - my post-subrequest handler only knows about the single buffer in the upstream structure and I haven't been able to locate a piece of code that would do things differently.

I suppose it would be possible to use an output filter instead of a post-subrequest handler for that use case, would that make sense? And last but not least, if I go down that road, can I just move my module "declaration" to the HTTP_AUX_FILTER_MODULES variable (from HTTP_MODULES), and still have the rest of module work fine, or will I need to use a second module for that?

Thanks a lot in advance!
Maxime
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org<mailto:nginx-devel at nginx.org>
http://mailman.nginx.org/mailman/listinfo/nginx-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151027/4ff91c67/attachment.html>