Logging inconsistencies during apparent DoS
is at rambler-co.ru
Sun Jul 27 09:55:33 MSD 2008
On Sat, Jul 26, 2008 at 04:45:56PM +1000, John Barratt wrote:
> We have been having problems with an apparent SYN-flood DoS attack.
> However there are are inconsistencies with the resulting log entries in
> nginx that along with the environment it is in, make me wonder if it
> really is a DoS attack, and/or there is something else going wrong.
> We are running nginx 0.6.31 on OSX 10.5 Server. Details of the problem
> go something like this :
> - We would see hundreds of active connections start to pile up on nginx,
> to the point where the site would become un-responsive.
> - It appears that one of the internal buffers had filled up at times,
> though this was not exactly matched to the site being un-responsive.
> The warning in the error log was :
> 2008/07/26 07:29:55 [warn] 36499#0: *15481336 kqueue change list is
> filled up, client: 18.104.22.168, server: www.host.com, request: "GET
> /index.html HTTP/1.1", host: "www.host.com", referrer:
This is not error, but just warning indicating peak load.
You may have other kernel network buffers filled up.
It seems that MacOSX has no FreeBSD's syncache and syncookies.
> - At the same time as this was happening we would be getting a bunch of
> 400's from reserved, or unroutable IPs being logged eg :
> 248.48.44.0 - - [26/Jul/2008:08:05:03 +1000] "-" 400 0 "-" "-" "-"
> - We would get 100's of these logged per minute, I am assuming many
> didn't get a chance to be logged, waiting for a timeout.
> - Mixed in amongst these requests though were occasional requests that
> looked like perfectly valid redirects, except for the fact that they had
> the same invalid IP as above (248.48.44.0), and there was a zero byte
> response reported, eg :
> 248.240.43.0 host.com - [05/Jul/2008:06:26:38 +1000] "GET / HTTP/1.1"
> 301 0 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1;
> .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)" "-"
> - Normal redirects typically record a 178 byte response size.
> - These requests seem to form a valid part of a user browsing the site,
> as at similar times there were often other valid 200, and 301 requests
> logged that had matching user agents and sometimes cookies, and with a
> different, valid IP address.
> - Also of note is that that particular IP address is explicitly blocked
> upstream via the range it is in, and also nginx is behind a load
> balancer that should only pass off connections that have already been
> established. So un-routable connections should not be able to get through.
> Any comments/thoughts/suggestions veria much appreciated as to whether
> this is indeed perhaps a problem with nginx and/or somewhere upstream.
If you have no set set_real_ip_from directive, then nginx loggs IP that
it gets from accept() syscall.
You may set debug logging for request from these addresses:
More information about the nginx