nginx and Apache killer

[Maxim Dounin]

Hello!

On Sun, Aug 28, 2011 at 11:39:14PM +0300, Gena Makhomed wrote:

> On 28.08.2011 19:36, Maxim Dounin wrote:
> 
> >>>Following "Apache Killer" discussions and the advisory from 2011-08-24
> >>>(Advisory: Range header DoS vulnerability Apache HTTPD 2.x CVE-2011-3192)
> >>>we'd like to clarify a couple of things in regards to nginx behavior
> >>>either in standalone or "combo" (nginx+apache) modes.
> 
> >>CVE-2011-3192 updated 26 Aug 2011. UPDATE 2 version available from
> >>http://mail-archives.apache.org/mod_mbox/httpd-announce/201108.mbox/browser
> 
> >>>In order to mitigate this attack when your installation includes
> >>>apache behind nginx we recommend you the following:
> 
> >>>1. Refer to the above mentioned security advisory CVE-2011-3192 for apache
> >>>and implement described measures accordingly.
> 
> >>these workarounds are needed only if "naked" apache open to internet.
> >>if apache listen only at 127.0.0.1 and located behind nginx frontend,
> >>enough ipmlement protection only at nginx level.
> 
> >I don't recommend relying on only nginx level protection even if
> >your backend server is only reachable from localhost.  It's
> >always a good idea to follow vendor recommendation and apply
> >needed security fixes to affected software.  This applies to other
> >cases as well, not only to this particular Apache problem.
> 
> quote from the CVE-2011-3192 UPDATE 2 version from 26 Aug 2011:
> 
> "There is currently no patch/new version of Apache HTTPD which fixes
> this vulnerability. This advisory will be updated when a long term
> fix
> is available."
> 
> "A full fix is expected in the next 24 hours".

[facepalm.jpg should be here]

Just follow vendor recommendations.  "There are several immediate 
options to mitigate this issue..."

> >>to emulate directive "max_ranges 5;" to allow max 5 ranges:
> >>
> >>if ($http_range ~ "(?:\d*\s*-\s*\d*\s*,\s*){5,}") {return 416;}
> >>if ($http_request_range ~ "(?:\d*\s*-\s*\d*\s*,\s*){5,}") {return 416;}
> >>
> >>to emulate directive "max_ranges 1;" to allow only one range:
> >>
> >>if ($http_range ~ ",") {return 416;}
> >>if ($http_request_range ~ ",") {return 416;}
> >>
> >>to completely remove these headers while proxying requests to apache:
> >>
> >>proxy_set_header Range "";
> >>proxy_set_header Request-Range "";
> >
> >In case of Request-Range you don't need any checks, just unset it
> >with
> >
> >     proxy_set_header Request-Range "";
> >
> >in nginx or equivalent
> >
> >     RequestHeader unset Request-Range
> >
> >in Apache.
> >
> >It's long obsolete one used by ancient browsers, never defined in any
> >standard.  It's not even supported by nginx.
> 
> Ok.
> 
> May be it will be better unset "Request-Range" request header
> as built-in feature of nginx to protect all vulnerable backends ?
> like as built-it feature "merge_slashes" with default merge_slashes on;

I don't think so.  Unsetting arbitrary headers just because 
there are some vulnerable software looks wrong for me.  I believe 
Apache folks will be able to release a fix in a week or so, and 
everybody will be happy enough after that.  And I don't expect 
this vulnerability to appear in other software.

> >>>We'd also like to notify you that for standalone nginx installations
> >>>we've produced the attached patch. This patch prevents handling
> >>>malicious range requests at all, instead outputting just the entire file
> >>>if the total size of all ranges is greater than the expected response.
> 
> >>this not protect nginx from "frequent nginx disk seek (D)Dos attack",
> >>and additional max_ranges checks/protections for nginx is required!!!
> 
> >I don't think the "attack" you are talking about is something
> >practical.  It requires prior knowledge of urls of many really
> >large (and "cold", i.e. not cached) files on the attacked site,
> >and it as well relies on disk seeks to be costly which is not
> >always true (and almost always not true for a single file, as even
> >"really large" still usually means "much less than disk size",
> >i.e. one can't force full disk seeks).
> 
> one - can't. but, multiple such requests can make very high seek rate
> of disk subsystem, and performance of disk subsystem will be very low

For multiple requests you need multiple large and cold files.  
That is what I was talking about.

> >Additionally, maximum
> >number of ranges requested in such "attack" is effectively limited
> >by maximum header length to something about 500 by default.
> 
> no, it is limited by large_client_header_buffers
> directive, by default it is 8k for 64-bit systems.

It's not clear why you said "no" here.  Maximum header length as 
you rightfully outlined is 8k by default, and this gives us about 
500 as a maximum number of ranges possible with at least 1M 
distance between them.

[...]

> >(On the other hand, I *do* think that limiting number of ranges to
> >low number like 5 suggested here and there *is* harmfull.  Quick
> >look over logs on my server for a couple of days reveals perfectly
> >valid requests from Adobe Reader with up to 17 ranges.  Minimum
> >sane value will be something about 50.)

FWIW: grepped a bit more logs, and it looks like Adobe Reader uses 
up to 200 ranges in a single request.  At least I see multiple  
requests with 200 ranges used (or less), but not any single 
request with more than 200.

> probably - this is (special) feature only of some pdf readers software,
> and for all other file types "max_ranges 1;" will be safe and harmless?

I've not seen myself any applications using multiple ranges except 
Adobe Reader.  Though there were at least attempts to implement 
JPEG2000 streaming using multiple ranges requests, and probably 
there are other applications as well.

Maxim Dounin