<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle"></style><style type="text/css"></style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Hey Sorin!
<div><br>
</div>
<div>First and foremost, thanks a lot for your answer.</div>
<div><br>
</div>
<div>I started working on implementing a body filter for my needs, and am already facing some strange issues. When my body filter is called on the subrequest I sent, despite it being a body filter, I get a buffer chain containing the whole upstream response,
including headers. Since this is not a header filter, this has gotten me quite confused... For what it's worth, my module doesn't even register a header filter. Here is some gdb output:</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>Breakpoint 1, ngx_http_dispatcher_body_filter (r=0x5402050, in=0x5403b38) at /home/mhenrion/dispatcher/ngx_http_dispatcher_module.c:207</div>
<div>207<span class="Apple-tab-span" style="white-space:pre"> </span>{</div>
<div>(gdb) p *in->buf</div>
<div>$1 = {pos = 0x5404441 "N", last = 0x5404498 "", file_pos = 0, file_last = 0,</div>
<div> start = 0x54043c0 "HTTP/1.1 200 OK\r\n[more data here that I stripped]",</div>
<div> end = 0x54083c0 "", tag = 0x7671e0 <ngx_http_proxy_module>, file = 0x0, shadow = 0x5403698, temporary = 1, memory = 0, mmap = 0, recycled = 0, in_file = 0, flush = 0,</div>
<div> sync = 0, last_buf = 0, last_in_chain = 0, last_shadow = 1, temp_file = 0, num = 0}</div>
<div>(gdb)<br>
<br>
Unless I'm missing something, this makes it a lot harder for me to actually work on the body of those responses, as I would end up having to parse the HTTP header to find the part I'm interested in.<br>
<br>
I'm also at a loss as to the semantics of the last_buf and last_in_chain flags. The module development guide doesn't contain any information about the "last_in_chain" flag; it only talks about last_buf as a marker indicating that the buffer chain is complete.
Intuitively, I would think of "last_in_chain" as a marker that we're at the end of the buffer list, but that doesn't make a lot of sense since this would be redundant with the next pointer being NULL. The ngx_buf.h header is equally unhelpful there and I couldn't
locate a piece of code that would make the semantics clear yet (some modules seem to consider both flags as being equal, while others aren't).</div>
<div><br>
</div>
<div>Last but not least, is there any documentation as to when exactly the output filters are being called? Mine seems to be called two times for the same subrequest. Once with the buffer that I showed above, and a second time with a buffer that is mostly all
zero'ed, except for the "sync" and "last_in_chain" flags both set to 1. That seems to be some kind of a flush request, but again, I don't see any documentation on that.</div>
<div><br>
</div>
<div>Thanks in advance,</div>
<div>Maxime</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF7462" style="direction: ltr;"><font face="Tahoma" size="2" color="#000000"><b>From:</b> nginx-devel [nginx-devel-bounces@nginx.org] on behalf of Sorin Manole [sorin.v.manole@gmail.com]<br>
<b>Sent:</b> Monday, October 26, 2015 8:05 PM<br>
<b>To:</b> nginx-devel@nginx.org<br>
<b>Subject:</b> Re: Processing responses of unbounded sizes from upstream servers<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">Hey Maxime! The solution you described makes sense and should work. And yes, you should probably use <span style="font-size:12.8px">HTTP_AUX_FILTER_MODULES to register your handler.</span>
<div><span style="font-size:12.8px">You will have to do a little more juggling with buffers however. From what I remember, the default behaviour when using not </span><span style="font-size:12.8px">NGX_HTTP_SUBREQUEST_IN_MEMORY is to pass on the subrequest
response to the client. You probably don't want that, so you will have to catch the response buffers in the output handler and not pass anything to the next filter.</span></div>
<div><span style="font-size:12.8px"><a href="https://trac.nginx.org/nginx/browser/nginx/src/http/modules/ngx_http_ssi_filter_module.c" target="_blank">https://trac.nginx.org/nginx/browser/nginx/src/http/modules/ngx_http_ssi_filter_module.c</a> is probably the
best source of inspiration for this.</span><br>
</div>
<div><span style="font-size:12.8px">Anyway, good luck!</span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2015-10-22 15:30 GMT+03:00 Maxime Henrion <span dir="ltr">
<<a href="mailto:mhenrion@appnexus.com" target="_blank">mhenrion@appnexus.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
Hello all,<br>
<br>
<br>
I am currently developing a module that has to send a number of subrequests to upstream servers, and aggregate them through application logic. I am currently doing that through a post-subrequest handler, using the NGX_HTTP_SUBREQUEST_IN_MEMORY flag. My problem
is that it is possible to receive very large responses from the upstream servers, and I end up with the "upstream buffer is too small" error, even after bumping the buffer sizes a number of times.<br>
<br>
It is my understanding that if I drop this subrequest flag, nginx wouldn't try to make the response fit in a single buffer anymore, but then I have no idea how to get at that buffer chain - my post-subrequest handler only knows about the single buffer in the
upstream structure and I haven't been able to locate a piece of code that would do things differently.<br>
<br>
I suppose it would be possible to use an output filter instead of a post-subrequest handler for that use case, would that make sense? And last but not least, if I go down that road, can I just move my module "declaration" to the HTTP_AUX_FILTER_MODULES variable
(from HTTP_MODULES), and still have the rest of module work fine, or will I need to use a second module for that?<br>
<br>
Thanks a lot in advance!<br>
Maxime<br>
_______________________________________________<br>
nginx-devel mailing list<br>
<a href="mailto:nginx-devel@nginx.org" target="_blank">nginx-devel@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx-devel" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx-devel</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>