Upstream module usage to process data

[Maxim Dounin]

Hello!

On Tue, May 30, 2017 at 03:45:15AM -0400, isolomka wrote:

> Hi Maxim,
> Thanks for quick response.
> I've implemented all upstream callbacks and upstream seems to work fine
> now.
> 
> But i still have the open question how to avoid sending received data from
> upstream to the downstream client.
> As I said, I need to process received data first and after that send result
> to output. 
> 
> As i see in the ngx_http_upstream_process_non_buffered_request()  buffers
> are sent to output if out_bufs are not empty:
>  if (u->out_bufs || u->busy_bufs) {
>                 rc = ngx_http_output_filter(r, u->out_bufs);
> 
> So is it a normal design from nginx point of view to store data in own
> buffer (not in u->out_bufs ) to avoid call to ngx_http_output_filter?
> I will call it later with my own chain of buffers when all data will be
> correctly processed.

There are two basic options you can use:

- Keep the data in u->buffer untill you can do appropriate process, 
  then put the result into u->out_bufs.  This will work as long as 
  u->buffer is big enough for responses you are expected to 
  handle.  For example, this what memcached module does when it 
  parses a response trailer.

- Copy the data elsewhere.  Once the response is complete, process 
  it, then put to u->out_bufs.

Trying to call ngx_http_output_filter() yourself is not likely to 
work correctly, and certainly not how input filters are expected 
to work.

Note well that if you need to do sophisticated processing of input 
data in an input filter, it might indicate that you are using it 
wrong.  Input filters are to remove various protocol-dependent 
transport encapsulation of data, such as chunked transfer encoding 
in HTTP and/or FastCGI records.  If you want to change data 
representation from one form to another, implementing this as a 
response body filter might be better solution, see 
http://nginx.org/en/docs/dev/development_guide.html#http_body_filters.

-- 
Maxim Dounin
http://nginx.org/