nginx limit_rate if in location - strange behaviour - possible bug ?

Bozhidara Marinchovska quintessence at bulinfo.net
Mon Feb 17 10:02:47 UTC 2014 

Hi, again

FreeBSD 9.1-STABLE #0: Sat May 18 00:32:18 EEST 2013 amd64

nginx version: nginx/1.4.4
TLS SNI support enabled
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I 
/usr/local/include' --with-ld-opt='-L /usr/local/lib' 
--conf-path=/usr/local/etc/nginx/nginx.conf 
--sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid 
--error-log-path=/var/log/nginx-error.log --user=www --group=www 
--http-client-body-temp-path=/var/tmp/nginx/client_body_temp 
--http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp 
--http-proxy-temp-path=/var/tmp/nginx/proxy_temp 
--http-scgi-temp-path=/var/tmp/nginx/scgi_temp 
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp 
--http-log-path=/var/log/nginx-access.log 
--with-http_image_filter_module --with-http_stub_status_module 
--add-module=/usr/ports/www/nginx/work/nginx_upload_module-2.2.0 
--add-module=/usr/ports/www/nginx/work/masterzen-nginx-upload-progress-module-a788dea 
--add-module=/usr/ports/www/nginx/work/naxsi-core-0.50/naxsi_src 
--with-pcre --with-http_ssl_module

I've meet strange behaviour when connection is established via software 
download manager.

My configuration is as follows:

location some_extension {

limit_rate_after 15m;
limit_rate 250k;

}

When I open or save as URL with matched extension via browser (desktop, 
mobile, etc), limit_rate works perfectly, after I hit 15MB of the file, 
my speed goes down up to 1Mb/s regarding my configuration.

When I place the same URL with matched extension in some download 
manager like IDM (internet download manager), FDM (free download 
manager) or others, limit_rate is not matched, download speed proceeded 
at max possible. [the actual problem]

As differences from browser and software download manager I met only:

- when download is processed via browser - if download is requested on 
chunks I get 206 partial content (multiple http sessions sent, multiple 
entries for this connection in my access log), download if processed as 
expected, after hitting 15MB of the file, download speed goes down to 1 Mb/s

- when download is processed via browser - if download is requested on 
one piece I get 200 ok (1 http session, 1 entry in my access log 
regarding this connection), download if processed as expected, after 
hitting 15MB of the file, download speed goes down to 1 Mb/s

- when download is processed via software download manager FDM (free 
download manager) I receive 1 http connection with status 200 OK (1 
entry in my access log, multiple established connections from remote 
addr X.X.X.X and remote ports A,B,C,D,...) , but after inspecting 
headers, I see multiple requests regarding this single http session, in 
multiple requests is changed only start byte in Range header without 
sending end byte in range header; netstat confirms multiple connections 
via this single session and limit_rate is not matched in that case.

tcpdump - download performed via browser Firefox - 200 OK single 
connection (1 entry in my access_log), no Range header:
GET [requested_file] HTTP/1.1
Host: [host]
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:27.0) Gecko/20100101 
Firefox/27.0 AlexaToolbar/alxf-2.19
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: [referer]
Cookie: [cookie]
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

tcpdump - download performed via browser Firefox 206 partial (multiple 
connections, multiple entries in my access log) - range header sent with 
end byte specified:
GET [requested_file] HTTP/1.1
Host: [host]
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:27.0) Gecko/20100101 
Firefox/27.0 AlexaToolbar/alxf-2.19
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: [referer]
Cookie: [cookie]
Connection: keep-alive
Range: bytes=2621440-2686975

tcpdump - download performed via FDM (single session, 1 entry in my 
access log, multiple established connections) without end byte specified 
in Range header:
GET [requested_file] HTTP/1.1
Referer: [referer]
Accept: */*
Range: bytes=2541702-
User-Agent: FDM 3.x
Host: [host]
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: [cookie]

My question is what may be the reason when downloading the example file 
with download manager not to match limit_rate directive and is there any 
way to debug somehow limit_rate ?

I've used to inspect in the past the behaviour when end byte in Range 
header is not specified (unfortunately I don't remember if is was with 
some browser, but I can reproduce it if needed), then Nginx returns 416 
(and when I see 416 in my access log, I block certain client for some 
time, let say 1h hour ).

Actually is not a problem for me, to match these download managers and 
block them, or to set some inspection regarding Range header, or to 
limit the connection with limit_conn instead limit_rate, but I don't 
want to limit speed by IP address or to block download managers, but I 
would like to use limit_rate regarding file type extension.

Thank you

On 10/9/2013 5:41 PM, Bozhidara Marinchovska wrote:
> I'm sorry, misread documentation.
>
> Placed set $limit_rate 0 in my case1 instead limit_rate 0 and now 
> works as expected.
>
>
> On 09.10.2013 17:21 ч., Bozhidara Marinchovska wrote:
>> Hi,
>>
>> nginx -V
>> nginx version: nginx/1.4.2
>> configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I 
>> /usr/local/include' --with-ld-opt='-L /usr/local/lib' 
>> --conf-path=/usr/local/etc/nginx/nginx.conf 
>> --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid 
>> --error-log-path=/var/log/nginx-error.log --user=www --group=www 
>> --http-client-body-temp-path=/var/tmp/nginx/client_body_temp 
>> --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp 
>> --http-proxy-temp-path=/var/tmp/nginx/proxy_temp 
>> --http-scgi-temp-path=/var/tmp/nginx/scgi_temp 
>> --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp 
>> --http-log-path=/var/log/nginx-access.log 
>> --with-http_image_filter_module --with-http_stub_status_module 
>> --add-module=/usr/ports/www/nginx/work/naxsi-core-0.50/naxsi_src 
>> --with-pcre
>>
>> FreeBSD 9.1-STABLE #0: Sat May 18 00:32:18 EEST 2013 amd64
>>
>> I'm using limit_rate case if in location. Regarding documentation "if 
>> in location" context is avaiable.
>>
>> My configuration is as follows:
>>
>> location some_extension {
>>
>> # case 1
>> if match_something {
>> root ...
>> break;
>> }
>>
>> # case 2
>> if match_another {
>> root ...
>> break;
>> }
>>
>> # else (case3)
>> root ...
>> something other ...
>> here it is placed also limit_rate / limit_after directives
>> }
>>
>> There is a root inside location with a (strong) reason :) (nginx 
>> pitfails case "root inside location block - BAD").
>>
>> When I open in my browser http://my.website.com/myfile.ext it matches 
>> case 3 from the cofiguration. Limit_rate/limit_after works correct as 
>> expected.
>> I want case1 not to have limit_rate / limit_after.
>>
>> Test one:
>> In case1 I place limit_rate 0, case3 is the same limit_rate_after 
>> XXm; limit_rate some_rate. When I open in my browser URL matching 
>> case1 - limit_rate 0 is ignored. After hitting XXm from the file I 
>> get limit_rate from case 3.
>>
>> Test 2:
>> In case 1 I place limit_rate_after 0; limit_rate 0, case3 is the same 
>> limit_rate_after XXm; limit_rate some rate. When I open in my browser 
>> URL matching case 1 - limit_rate_after 0 and limit_rate 0 are 
>> ignored. Worst is that when I try to download the file, I even didn't 
>> match case3 - my download starts from the first MB with limit_rate 
>> bandwidth from case3.
>>
>> Both tests are made in interval from 20 minutes, 1 connection from my 
>> IP, etc.
>>
>> I don't post my whole configuration, because may be it is 
>> unnessesary. If cases are with http_referer.
>>
>> Case 1 - if I see referer some_referer, I do something. (here I don't 
>> want to place any limits)
>> Case 2 - If I see another referer , I do something else.
>> Case 3 - Else ... something other (here I have some limits)
>>
>> I'm sure I match case1 when I test (nginx-error.log with debug say 
>> it), I mean my configuration with cases is working as expected, the 
>> new thing is limit_rate and limit_rate_after which is not working as 
>> expected.
>>
>> Any thoughts ? Meanwhile I will test on another version.
>>
>> Thanks
>>
>>
>>
>> _______________________________________________
>> nginx mailing list
>> nginx at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>

More information about the nginx mailing list