Logging inconsistencies during apparent DoS

[John Barratt]

Hi,
	We have been having problems with an apparent SYN-flood DoS attack. 
However there are are inconsistencies with the resulting log entries in 
nginx that along with the environment it is in, make me wonder if it 
really is a DoS attack, and/or there is something else going wrong.

We are running nginx 0.6.31 on OSX 10.5 Server.  Details of the problem 
go something like this :

- We would see hundreds of active connections start to pile up on nginx, 
to the point where the site would become un-responsive.

- It appears that one of the internal buffers had filled up at times, 
though this was not exactly matched to the site being un-responsive. 
The warning in the error log was :

2008/07/26 07:29:55 [warn] 36499#0: *15481336 kqueue change list is 
filled up, client: 210.8.118.47, server: www.host.com, request: "GET 
/index.html HTTP/1.1", host: "www.host.com", referrer: 
"http://www.host.com/"

- At the same time as this was happening we would be getting a bunch of 
400's from reserved, or unroutable IPs being logged eg :

248.48.44.0 - - [26/Jul/2008:08:05:03 +1000] "-" 400 0 "-" "-" "-"

- We would get 100's of these logged per minute, I am assuming many 
didn't get a chance to be logged, waiting for a timeout.

- Mixed in amongst these requests though were occasional requests that 
looked like perfectly valid redirects, except for the fact that they had 
the same invalid IP as above (248.48.44.0), and there was a zero byte 
response reported, eg :

248.240.43.0 host.com - [05/Jul/2008:06:26:38 +1000] "GET / HTTP/1.1" 
301 0 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; 
.NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)" "-"

- Normal redirects typically record a 178 byte response size.

- These requests seem to form a valid part of a user browsing the site, 
as at similar times there were often other valid 200, and 301 requests 
logged that had matching user agents and sometimes cookies, and with a 
different, valid IP address.

- Also of note is that that particular IP address is explicitly blocked 
upstream via the range it is in, and also nginx is behind a load 
balancer that should only pass off connections that have already been 
established.  So un-routable connections should not be able to get through.

Any comments/thoughts/suggestions veria much appreciated as to whether 
this is indeed perhaps a problem with nginx and/or somewhere upstream.

Thanks,

John Barratt.