rate limit request body read from a body filter? (nginx-upload-module)

[Maxim Dounin]

Hello!

On Thu, Jan 03, 2019 at 12:21:15AM -0500, Frankie Dintino wrote:

> I'm currently trying to rewrite 
> https://github.com/fdintino/nginx-upload-module/ 
> <https://github.com/fdintino/nginx-upload-module/> as a request 
> body filter, in order to simplify the code and future-proof it 
> (as it's currently written, there's a lot of copypasta from 
> ngx_http_request_body.c that is very fragile and probably 
> already incompatible with new features and third party modules).
> 
> I'm nearly done this work, but I've hit a stumbling block trying 
> to port the upload rate limiting feature. The current 
> implementation completely overrides the request handler, so 
> rate-limiting the request body is pretty trivial: if you need to 
> delay the next buffer read, to stay within the rate limit, you 
> can set c->read->delayed = 1, call ngx_add_timer(c->read, 
> delay), and return NGX_AGAIN.
> 
> When I try this from the request body filter, I'm able to delay 
> the very next read event, but afterwards when control is 
> returned to the outer request handler, the rate that read 
> buffers get consumed is no longer within my control (also, 
> returning NGX_AGAIN from the body handler to the http2 request 
> handler doesn't work at all—it raises a 500—but that's another 
> issue).
> 
> So it seems to me that—if I want to keep the upload rate 
> limiting feature—I'm going to have to do it somewhere else, 
> maybe in a custom read event handler. Is there another option 
> I'm not thinking of? And if I have to do it in a custom 
> read_event_handler, does anyone have pointers for how I can do 
> this in the least obtrusive way?

With HTTP/1.x, I think it should be possible to implement rate 
limiting the way you are doing it as long as requests larger than 
client_body_buffer_size are concerned - that is, when writing 
requests to files.  So nginx will read buffers up to 
client_body_buffer_size, and writing to a temporary file will be 
delayed by your code.

With HTTP/2, I don't think there is a way to implement upload rate 
limiting without changes in the HTTP/2 code.

-- 
Maxim Dounin
http://mdounin.ru/