Large number of invalid packets detected

Mon May 19 01:11:57 MSD 2008

Hi,

On Sun, May 18, 2008 22:11, Rt Ibmer wrote:
>>You didn't show us what was important: the log that was produced.
>
> I included various samples from the different types of invalid state
> errors at the end of this message.
>
>> If the rejected packet was a ACK/FIN, then it is harmless.
>
> About 25% of the invalid packets are of this type. Should I drop them
> using DROP or using RESET?

I think you can drop them.

> How about for the other types?  I take it
> there is no harm in dropping or rejecting these - as they are obviously
> bad packets anyway?

I think you really might better ask those questions on the netfilter list:
http://www.netfilter.org/mailinglists.html
There is certainly lot of people way more knowledgeable than on the nginx
list about iptables/netfilter :-)

> I should note that in EVERY case it appears that nginx served the content
> successfully and that browser got it just fine (no obvious errors), and
> these bad packet log messages are showing up anywhere from 2-5 minutes
> after the user hits the page and not part of a new request.  So it seems
> there is some packet activity that occurs minutes after the user has left
> the site.

Maybe you could tcpdump the whole traffic to analyze exactly what's
happening?

>>On busy servers conntrack removes the connection from its table as soon
>> as
>>it gets the FIN packet. When the ACK/FIN then comes, the connection is
>> not
>>known and this is not a SYN. It logs it as invalid.
>
> My server is not very busy (yet). It is handling about 500-1500 requests
> per hour and no more than a 2 or 3 concurrently at times.

Yes, definitely not busy. Anyway, I think what I wrote still applies,
although I'm really not a netfilter expert.

>>This could also have been a scan (nmap can trigger invalid packets too).
>
> I really don't think this could be a scan. I took the site live for the
> first time and immediately started seeing these invalid packets.

You bet? I get so much automatic scans per seconds, that I don't even look
to the logs anymore :-(

>> What kernel version are you running?
>
> uname -a returns the following:
> 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 athlon i386
> GNU/Linux
>
> Here are some log samples. Thank you and I look forward to hearing your
> thoughts on this. I really appreciate your help!!
> ==========
> Out of 436 log entries here is the break down (these are not exact counts
> but will give you an idea of what % are for what type of bad packet):
>
> 218 are type "ACK RST" on the INPUT chain. None of this type are for the
> OUTPUT chain. Examples:
> --------
> May 18 15:21:44 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
> MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
> TTL=238 ID=13770 DF PROTO=TCP SPT=1391 DPT=80 WINDOW=0 RES=0x00 ACK RST
> URGP=0
>
> May 18 15:17:25 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
> MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
> TTL=240 ID=17970 DF PROTO=TCP SPT=49671 DPT=80 WINDOW=0 RES=0x00 ACK RST
> URGP=0

It is indeed the same issue than ACK/FIN
http://lists.netfilter.org/pipermail/netfilter/2004-June/053782.html

> 110 are of type "ACK FIN". Out of this group 82 are on the INPUT chain and
> 28 are on the OUTPUT chain. Examples:
> -------- (OUTPUT)
> May 18 15:14:13 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0
> SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=31199 DF
> PROTO=TCP SPT=80 DPT=3055 WINDOW=6432 RES=0x00 ACK FIN URGP=0
>
> May 18 13:39:17 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0
> SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=299 DF
> PROTO=TCP SPT=80 DPT=1157 WINDOW=6432 RES=0x00 ACK FIN URGP=0
> -------- (INPUT)
> May 18 15:24:08 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
> MAC=12:....:ff:08:00 SRC==x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
> TTL=49 ID=25196 DF PROTO=TCP SPT=57214 DPT=80 WINDOW=65535 RES=0x00 ACK
> FIN URGP=0
>
> May 18 12:58:10 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
> MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=52 TOS=0x00 PREC=0x00
> TTL=47 ID=52221 DF PROTO=TCP SPT=55817 DPT=80 WINDOW=65535 RES=0x00 ACK
> FIN URGP=0

We already talked about this one above. This could also be a Maimon scan
which uses ACK+FIN.

> 99 are for type "RST", out of those 22 are for the INPUT chain and 77 are
> for the OUTPUT chain. Examples:
> -------- (OUTPUT)
> May 18 10:49:15 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0
> SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF
> PROTO=TCP SPT=80 DPT=19144 WINDOW=0 RES=0x00 RST URGP=0
>
> May 18 15:35:40 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0
> SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF
> PROTO=TCP SPT=80 DPT=62353 WINDOW=0 RES=0x00 RST URGP=0
> -------- (INPUT)
> May 18 13:54:58 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
> MAC=12:....:ff:08:00 SRC==x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
> TTL=238 ID=9943 PROTO=TCP SPT=4361 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
>
>
> May 18 12:31:37 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
> MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
> TTL=237 ID=65259 PROTO=TCP SPT=12568 DPT=80 WINDOW=6432 RES=0x00 RST
> URGP=0

Hum, that one seems strange. I don't see any easy explanation.
Maybe with a full capture of the traffic you will be able to understand
what's that.

> 9 are for type "ACK PSH FIN". Example:
> --------
> May 18 02:00:03 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0
> SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=29585 DF
> PROTO=TCP SPT=80 DPT=5178 WINDOW=54 RES=0x00 ACK PSH FIN URGP=0

http://lists.netfilter.org/pipermail/netfilter/2005-August/062059.html

As a side note, I'm also running conntrack on my servers, and I do see
some of the ACK+FIN and ACK+RST invalid packets. I drop them and doesn't
really care: nobody ever complained :-)

Good luck :-)
-- 
Brice Figureau
Days of Wonder