Thread Pool HTTP POST issue

[Maxim Dounin]

Hello!

On Wed, Dec 16, 2020 at 03:34:38PM +0100, st.gabrielli at libero.it wrote:

> Hi,we are doing some tests using the nginx thread pool feature. The scenario is the following:1) the worker process register with ngx_http_read_client_request_body fuction a handler for the request. We can call it "post_handler"
> 2) the worker process receive a POST bid request. The payload is a json that contains an id field (unique among requests that will be used in the response generated by our web service)
> 3) the worker process in the "post_handler" create the task context and launch the task
> 4) the request payload is read in the task code that is executed in the thread pool
> 5) the blocking code is executed in the task code that is executed in the thread pool
> 6) task completion handler (worker process - main thread - outside thread pool): 
> the        ngx_http_send_header(r) 
> ngx_http_output_filter(r, out);    ngx_http_finalize_request(r, r->headers_out.status);    code is executed in the task completion handler Now some details:
> * in point 3:
> -- the "r" (ngx_http_request_t) pointer and "b" (ngx_buf_t *) the response buffer pointer are saved as fields in the task context (a simple c structure used as the context of the task)
> -- just before the code for the task launch (call to "ngx_thread_task_post" function) we put the following lines of code:
> r->main->blocked++;
> r->aio = 1;* in point 5, after the blocking code execution and after the generated response string has been saved in a task context field, 
> -- the buffer output chain is allocated into the heap and stored into a task context field:
> //buffer chain needed to put response buffer in
> ngx_chain_t  *out = ngx_alloc_chain_link(r->pool);  -- the buffers are filled in (r, b were stored into the task context before task launch - response is a local var where the response saved into the task context has been copied):
> r->headers_out.content_type_len = sizeof("application/json") - 1;
> r->headers_out.content_type.data = (u_char *) "application/json";  r->headers_out.content_length_n = strlen(response);  /* adjust the pointers of the buffer */
> b->pos = (u_char *)response;
> b->last = (u_char *)response + strlen(response);  b->memory = 1;    /* this buffer is in memory */
> b->last_buf = 1;  /* this is the last buffer in the buffer chain */
> (*out).buf = b;
> (*out).next = NULL;* in point 6, just before the instructions described above (for point 6) we put the following lines of code:
> r->main->blocked--;
> r->aio = 0; Issue we are trying to fix:- in point 6 comparing the response string saved into the task context with the buffer chain "out" var content (the comparison is done just before calling "ngx_http_output_filter" and "ngx_http_finalize_request") we discovered that about the 40% of the times the 2 values are different for two reasons:
> - "out" content is an empty string
> - "out" content is a response with id different than the one of the response string in the task context (that is the one used to fill in "out" buffer chain in point 5)We think there is probably an error either about what it is possible to put into the task context structure (maybe it is not possible to manipulate "r", "b", "out" outside the main thread...) or the right usage and position of "r->main->blocked" counter and "r->aio" flag. 
> Any hints about how to solve this issue?Thanks.

Thread pools are designed to execute short self-contained blocking 
code like system calls for reading a file, and the code executed 
in a thread pool is expected to be thread-safe.  In particular, 
this means that it cannot call most of nginx functions or access 
nginx structures.  For example, accessing the request structure 
might lead to undefined results if it is modified by the main 
thread at the same time.

If I understand the above text correctly, you are doing 
allocations from the request pool and various modifications of the 
request structure within a thread.  This is not going to work, as 
these operations are not tread-safe, see above.

You should do relevant operations from the main thread instead.  
That is, you should either pre-allocate appropriate buffers for 
your threaded code to return results in (for example, nginx's own 
ngx_thread_read() reads the data into pre-allocated buffer), or 
provide the result within your own buffers (for example, allocated 
with thread-safe malloc() within your thread code) and free these 
buffers afterwards as appropriate.

-- 
Maxim Dounin
http://mdounin.ru/