Reading large request body using ngx_http_read_client_request_body

[Maxim Dounin]

Hello!

On Sat, May 18, 2019 at 05:10:29PM -0400, NginxNewbee wrote:

> Apologies if this is a trivial question. I have searched it on web and none
> of the answers have solved my problem. I am trying to read request body and
> thing seem to work fine if request body is small (4  kb). As soon as it
> becomes 4+ megabytes,ngx_http_read_client_request_body returns NGX_AGAIN
> (-2) in rc and my RequestBodyHandler is never called. 
> 
> snippet of nginx.conf is like this for buffer settings. I have removed lot
> of other stuff from nginx.conf to keep it brief.
> 
> http {
>     include       mime.types;
>     default_type  application/octet-stream;
> 
>     sendfile        on;
>     keepalive_timeout  65;
> 
> 	client_body_buffer_size 1m;
> 	client_max_body_size 0;
> }
> 
> / * code sample start */
>  r->request_body_in_single_buf = 1;
>  r->request_body_in_persistent_file = 1;
>  r->request_body_in_clean_file = 1;
>  r->request_body_file_log_level = 0;
> 
>         rc = ngx_http_read_client_request_body(r, RequestBodyHandler);
> 	if (rc >= NGX_HTTP_SPECIAL_RESPONSE) {
> 
> 		CORE_TRACE(Error, L"Nginx read request error: %d", rc);
>                 return from thread.                
> 	}
> 
>         return NGX_DONE.
> 
> / * code sample end */
> 
> Just to add some more context here. I use nginx thread pool. So my module's
> content handler basically queues a task to thread pool. Callback of thread
> eventually calls ngx_http_read_client_request_body to read request body. If
> there is an error (as stated in above if condition), thread simply returns
> and the completion handler of thread is called by nginx. In the completion
> handler, I do finalize request with NGX_DONE always.

In no particular order:

- You shouldn't try to use nginx functions such as 
  ngx_http_read_client_request_body() and objects such as "r" in 
  threads.  While this may appear to work, in fact it's not - as 
  nginx provides no thread safety for these functions and objects.

- You should finalize the request with NGX_DONE only after the 
  request body handler is called and you've done with the request.  
  By finalizing the request earlier you basically say that you've 
  done with the request, and nginx should stop processing - so there 
  is no surprie your body handler is never called.

-- 
Maxim Dounin
http://mdounin.ru/