[dev] how to send data to client

[Manlio Perillo]

Hi again!

I'm here with another question.

I'm adding the [dev] tag to questions about internals of nginx.
Unfortunately I'm not good at writing documentation, but I hope that 
someone will take note about all these discussions.

Igor, thanks for your help!


After the question on how to read data, the problem is now how to write 
data.

Reading data is quite easy, since nginx will call my callback function 
when all the data is available (and for now I'm not interested in how to 
register callbacks that will be called when each chunk of data is read).

Writing data seems to be bit more tricky.

I have a lot of questions:


Body filters
============

Body filters are called by ngx_http_output_filter.
Is the chunked body filter enabled by default?

When the WSGI application does not set the Content-Lenght header, should 
I set r->headers_out.content_length_n == -1 by hand, or is it 
initialized by nginx?


Headers filters
===============

The WSGI application returns the headers in a list, so I can easily copy 
the headers to the r->headers_out->headers list.

Does nginx have an header filter that process the 
r->headers_out->headers list and fills the attributes on the 
r->headers_out object?

I suspect that I must do the job by myself, parsing each header and 
checking if it supported in the ngx_http_headers_out_t structure...

Does nginx at least fills missing headers like Server and Date?


Asyncronous writing
===================

The WSGI specification has some problems here.

Here is the text 
http://www.python.org/dev/peps/pep-0333/#buffering-and-streaming:

WSGI servers, gateways, and middleware must not delay the transmission 
of any block; they must either fully transmit the block to the client, 
or guarantee that they will continue transmission even while the 
application is producing its next block. A server/gateway or middleware 
may provide this guarantee in one of three ways:

    1. Send the entire block to the operating system (and request that
       any O/S buffers be flushed) before returning control to the
       application, OR
    2. Use a different thread to ensure that the block continues to be
       transmitted while the application produces the next block.
    3. (Middleware only) send the entire block to its parent
       gateway/server

By providing this guarantee, WSGI allows applications to ensure that 
transmission will not become stalled at an arbitrary point in their 
output data. This is critical for proper functioning of e.g. multipart 
"server push" streaming, where data between multipart boundaries should 
be transmitted in full to the client.


This is simply impossible for nginx, since it is fully asincronous and 
does not uses threads to send available buffers.

buffers are sent to the client when the application returns from the 
request handler, so that the nginx reactor (main loop) can check the events.

This means that no data is sent to the client until the WSGI application 
has produced all of its data.

I'm planning to add an extension to enable true asyncronous support in WSGI.


End of request processing
=========================

I need to be able to register a callback that will be called when all 
the response has been sent to the client.

The reason is that, to avoid to copy buffers, I want to send to the 
chain buffers the content of:

     size = PyString_GET_SIZE(object);
     result = (u_char *) PyString_AsString(object);


PyString_AsString returns:
    A pointer that refers to the internal buffer of string, not a copy.


This means that I must keep the String object alive, but I must 
decrement its reference count when nginx has sent the entire content of 
the chain buffer to the client.

Reading the nginx source, it seems that I can use the cleanup attribute 
of the request object.

This is nice, since I can store in the "data" a list of all the Python 
String objects returned by the WSGI application.

What is the right way to use the cleanup handler?
Should I search the entire linked list until I find a free next slot?





Thanks  Manlio Perillo