DDoS Attack Log Analysis Question

[Jim Ohlstein]

Payam Chychi wrote:
> On Fri, Oct 9, 2009 at 4:40 PM, Jim Ohlstein <jim at ohlste.in> wrote:
>> The nginx forum had a DDoS attack which took the site down this morning.  In
>> approximately 23 seconds there were just under 900,000 lines in the error
>> log that looked like:
>>
>> 2009/10/09 10:21:38 [alert] 32576#0: accept() failed (24: Too many open
>> files)
>>
>> First question is do each of these entries represent an attempted
>> connection?
>>
>>
>> Looking at the access log there were thousands of requests for the same page
>> from roughly 400 IP's in that same 23 second span like this:
>>
>> 58.53.85.229 - - [09/Oct/2009:10:21:38 -0400] "GET /list.php?2,page=1
>> HTTP/1.1" 500 553 "http://forum.nginx.org" "Mozilla/4.0
>> (compatible; MSIE 6.0; Windows 5.1)"
>> 60.177.29.231 - - [09/Oct/2009:10:21:38 -0400] "GET /list.php?2,page=1
>> HTTP/1.1" 500 553 "http://forum.nginx.org" "Mozilla/4.0
>>  (compatible; MSIE 6.0; Windows 5.1)"
>> 125.91.207.11 - - [09/Oct/2009:10:21:38 -0400] "GET /list.php?2,page=1
>> HTTP/1.1" 500 553 "http://forum.nginx.org" "Mozilla/4.0
>>  (compatible; MSIE 6.0; Windows 5.1)"
>> 125.119.65.194 - - [09/Oct/2009:10:21:38 -0400] "GET /list.php?2,page=1
>> HTTP/1.1" 500 553 "http://forum.nginx.org" "Mozilla/4.
>> 0 (compatible; MSIE 6.0; Windows 5.1)"
>>
>>
>> All of the IP's that I checked were legitimate IP's that localize to various
>> regions in China.
>>
>> I have set up limit_zone and limit_conn directives to hopefully mitigate
>> this in the future.
>>
>> Second question is where to set limit_conn and what are the effects on users
>> if set low? The site generally responds quickly, at least here in the US,
>> and I don't want it to be especially sluggish for people using less fast
>> connections in other parts of the world, but of course I want to reduce the
>> chances of this happening again. Bear in mind this is a low traffic site
>> (16K visits in the last month) on a small VPS.
>>
>> Any advice would be appreciated.
>>
>> --
>> Jim Ohlstein
>>
>>
> 
> Jim,
> 
> Your main issue is going ot be the system 'open files' setting,  can
> you show me the output of the following please, run the command as
> room or the user that runs the nginx proc.  'ulimit -a'
> 
> by default linux sets this value to  which really means that you cant
> have more than 1024 file descriptors open on the system at one time.
> 
> You can modify this setting to safely allow up to 150,000 on a more
> resent dual cor box.
> 'ulimit -n 150000'

It's hardly a dual core box. :) It's a small VPS with 256 MB of RAM that 
has a great deal of room to spare. Once again, we only receive ~500 
visitors per day on average. Open files were set at 4096 which I 
*thought* would have been more than enough for the (one) nginx worker 
process max connections which is set at 1024. I have already increased 
the number to 32768. I'm not totally convinced that will do much in this 
situation given the intensity of the attack on such a small system. I 
would think that eventually some system resource would have given even 
if file descriptors had not run out, though I must admit that I have 
been fortunate not to have dealt with this much in the past.

> 
> try reading up on ulimit, its a huge for system performance, also make
> sure your not running any iptables connection tracking else you will
> need to fine tune those settings as well.
> 
> 


-- 
Jim Ohlstein