nginx+lua reverse proxy empty body

[Bart van Deenen]

Hi Agentz

But wouldn't the statement 
          client_body_in_single_buffer on;
cause the whole body of the proxied server to go into ngx.arg[1] ? 

And I also don't understand that my example code shouldn't work reliably, even if the proxied data is passed through it in chunks (unless the chunk boundary would accidentally be right in the middel of my short match string). I've done a very similar setup proxying and modification of a simple website (vandeenensupport.com), and that works perfectly.

I have also noticed that when I add a 'print(ngx.arg[1])' in the first line of the lua section of my example, the html replacement works reliably, no more empty ngx.arg[1]! 
But that print only goes into the nginx logging, so maybe it's only its timing that has some effect?

So I still don't understand it.

Thanks for all your good work on nginx.

Bart

________________________________________
From: nginx-bounces at nginx.org [nginx-bounces at nginx.org] on behalf of agentzh [agentzh at gmail.com]
Sent: Wednesday, November 28, 2012 12:43 AM
To: nginx at nginx.org
Subject: Re: nginx+lua reverse proxy empty body

Hello!

On Tue, Nov 27, 2012 at 2:28 AM, Bart van Deenen wrote:
> The problem I have basically that the ngx.arg[1] is an empty string
> (sometimes, timing dependent?) on url's that are definitely not empty.
>

It is normal that ngx.arg[1] is an empty string in the body filters
when the upstream module generates "pure special bufs" like those with
only the "last_buf" flag set (i.e., the eof flag set on the Lua land).

It's normal that for a given response, the output body filter gets
called multiple times because that's exactly how streaming processing
works in Nginx (you surely do not want to buffer all the data at a
time for huge responses).

And the response body may be fed into your body filter in multiple
data chunks. You should always be prepared for that in your Lua code.

Please refer to the documentation for body_filter_by_lua for more information:

    http://wiki.nginx.org/HttpLuaModule#body_filter_by_lua

BTW, doing simple regex match in body filters may not always work as
expected because the nginx upstream module may split the response body
into chunks in an arbitrary way (e.g., splitting in the middle of the
word "Speel", for example).

I've been working on the sregex C library that will support streaming
match just like Ragel:

    https://github.com/agentzh/sregex

It's still in progress though but it'll soon be usable on the Lua land :)

Best regards,
-agentzh

_______________________________________________
nginx mailing list
nginx at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx