nginx and Apache killer

[Gena Makhomed]

On 28.08.2011 19:36, Maxim Dounin wrote:

>>> Following "Apache Killer" discussions and the advisory from 2011-08-24
>>> (Advisory: Range header DoS vulnerability Apache HTTPD 2.x CVE-2011-3192)
>>> we'd like to clarify a couple of things in regards to nginx behavior
>>> either in standalone or "combo" (nginx+apache) modes.

>> CVE-2011-3192 updated 26 Aug 2011. UPDATE 2 version available from
>> http://mail-archives.apache.org/mod_mbox/httpd-announce/201108.mbox/browser

>>> In order to mitigate this attack when your installation includes
>>> apache behind nginx we recommend you the following:

>>> 1. Refer to the above mentioned security advisory CVE-2011-3192 for apache
>>> and implement described measures accordingly.

>> these workarounds are needed only if "naked" apache open to internet.
>> if apache listen only at 127.0.0.1 and located behind nginx frontend,
>> enough ipmlement protection only at nginx level.

> I don't recommend relying on only nginx level protection even if
> your backend server is only reachable from localhost.  It's
> always a good idea to follow vendor recommendation and apply
> needed security fixes to affected software.  This applies to other
> cases as well, not only to this particular Apache problem.

quote from the CVE-2011-3192 UPDATE 2 version from 26 Aug 2011:

"There is currently no patch/new version of Apache HTTPD which fixes 
this vulnerability. This advisory will be updated when a long term fix
is available."

"A full fix is expected in the next 24 hours".

>> to emulate directive "max_ranges 5;" to allow max 5 ranges:
>>
>> if ($http_range ~ "(?:\d*\s*-\s*\d*\s*,\s*){5,}") {return 416;}
>> if ($http_request_range ~ "(?:\d*\s*-\s*\d*\s*,\s*){5,}") {return 416;}
>>
>> to emulate directive "max_ranges 1;" to allow only one range:
>>
>> if ($http_range ~ ",") {return 416;}
>> if ($http_request_range ~ ",") {return 416;}
>>
>> to completely remove these headers while proxying requests to apache:
>>
>> proxy_set_header Range "";
>> proxy_set_header Request-Range "";
>
> In case of Request-Range you don't need any checks, just unset it
> with
>
>      proxy_set_header Request-Range "";
>
> in nginx or equivalent
>
>      RequestHeader unset Request-Range
>
> in Apache.
>
> It's long obsolete one used by ancient browsers, never defined in any
> standard.  It's not even supported by nginx.

Ok.

May be it will be better unset "Request-Range" request header
as built-in feature of nginx to protect all vulnerable backends ?
like as built-it feature "merge_slashes" with default merge_slashes on;

>>> We'd also like to notify you that for standalone nginx installations
>>> we've produced the attached patch. This patch prevents handling
>>> malicious range requests at all, instead outputting just the entire file
>>> if the total size of all ranges is greater than the expected response.

>> this not protect nginx from "frequent nginx disk seek (D)Dos attack",
>> and additional max_ranges checks/protections for nginx is required!!!

> I don't think the "attack" you are talking about is something
> practical.  It requires prior knowledge of urls of many really
> large (and "cold", i.e. not cached) files on the attacked site,
> and it as well relies on disk seeks to be costly which is not
> always true (and almost always not true for a single file, as even
> "really large" still usually means "much less than disk size",
> i.e. one can't force full disk seeks).

one - can't. but, multiple such requests can make very high seek rate
of disk subsystem, and performance of disk subsystem will be very low

> Additionally, maximum
> number of ranges requested in such "attack" is effectively limited
> by maximum header length to something about 500 by default.

no, it is limited by large_client_header_buffers
directive, by default it is 8k for 64-bit systems.

and one such malicious "Range:" request
can make a few hundred "seek" operations.

and few hundred such malicious requests easy can
generate very high seek rate via vulnerable nginx.

and none of nginx built-in protection directives
limit_req / limit_conn / keepalive_requests / ...
can help to protect from such type of DDoS attack.

workaround/fix for this "vulnerability" can be implemented
only via optional rewrite module if nginx compiled with it.

and yes, this vector of attack applicable only for web servers
with many large files (video/iso) and non-SSD storage devices.

> (On the other hand, I *do* think that limiting number of ranges to
> low number like 5 suggested here and there *is* harmfull.  Quick
> look over logs on my server for a couple of days reveals perfectly
> valid requests from Adobe Reader with up to 17 ranges.  Minimum
> sane value will be something about 50.)

probably - this is (special) feature only of some pdf readers software,
and for all other file types "max_ranges 1;" will be safe and harmless?

P.S.

how to force 411 seek operations by one malicious request:

Range: bytes=0-1,1072694271-1072694272,2097154-2097155,...

========================================================================

#!/usr/bin/python

size = 1 * 1024 * 1024 * 1024
step = 1 * 1024 * 1024 + 1

line = 'Range: bytes='
limit = 8 * 1024

count = 0

def point( x ): return str( x ) + '-' + str( x + 1 ) + ','

seq = [ point( x ) for x in range( 0, size - 1 , step ) ]

seq[ 1 :: 2 ] = reversed( seq[ 1 :: 2 ] )

for range_ in seq:
     if len( line + range_ ) < limit:
         line += range_
         count+= 1
     else:
         line = line[ : -1 ]
         break

print 'how to force', count, 'seek operations by one malicious request:'
print
print line

========================================================================

-- 
Best regards,
  Gena