Logging inconsistencies during apparent DoS
djon00 at gmail.com
Sat Jul 26 10:45:56 MSD 2008
We have been having problems with an apparent SYN-flood DoS attack.
However there are are inconsistencies with the resulting log entries in
nginx that along with the environment it is in, make me wonder if it
really is a DoS attack, and/or there is something else going wrong.
We are running nginx 0.6.31 on OSX 10.5 Server. Details of the problem
go something like this :
- We would see hundreds of active connections start to pile up on nginx,
to the point where the site would become un-responsive.
- It appears that one of the internal buffers had filled up at times,
though this was not exactly matched to the site being un-responsive.
The warning in the error log was :
2008/07/26 07:29:55 [warn] 36499#0: *15481336 kqueue change list is
filled up, client: 220.127.116.11, server: www.host.com, request: "GET
/index.html HTTP/1.1", host: "www.host.com", referrer:
- At the same time as this was happening we would be getting a bunch of
400's from reserved, or unroutable IPs being logged eg :
248.48.44.0 - - [26/Jul/2008:08:05:03 +1000] "-" 400 0 "-" "-" "-"
- We would get 100's of these logged per minute, I am assuming many
didn't get a chance to be logged, waiting for a timeout.
- Mixed in amongst these requests though were occasional requests that
looked like perfectly valid redirects, except for the fact that they had
the same invalid IP as above (248.48.44.0), and there was a zero byte
response reported, eg :
248.240.43.0 host.com - [05/Jul/2008:06:26:38 +1000] "GET / HTTP/1.1"
301 0 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1;
.NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)" "-"
- Normal redirects typically record a 178 byte response size.
- These requests seem to form a valid part of a user browsing the site,
as at similar times there were often other valid 200, and 301 requests
logged that had matching user agents and sometimes cookies, and with a
different, valid IP address.
- Also of note is that that particular IP address is explicitly blocked
upstream via the range it is in, and also nginx is behind a load
balancer that should only pass off connections that have already been
established. So un-routable connections should not be able to get through.
Any comments/thoughts/suggestions veria much appreciated as to whether
this is indeed perhaps a problem with nginx and/or somewhere upstream.
More information about the nginx