Nginx securiy problem

Fri Dec 4 03:09:41 MSK 2009

-------- Original-Nachricht --------
> Datum: Thu, 3 Dec 2009 04:22:22 -0500
> Von: "egerci" <nginx-forum at nginx.us>
> An: nginx at sysoev.ru
> Betreff: Nginx securiy problem

> Hello,
> 
> I am using nginx for one year. 
> 
> Server info :
> 2 x 8 core -  16GB  (one for web server and other for mysql)
> OS : linux RH 5
> Nginx version : 0.8.x
> web application :  vbulletin 3.8.4 PL1
> 
> I have experienced some security issues in last month. My server was under
> attack with 300Mbit. I don't know what is type of attack. But when I ask
> my service provider to add my server behind cisco guard, firewall could
> handle yhese attacks.
> 
> By the way my server located in softlayer.  So, they give this firewall
> only limited time (only 24 hours) adn thenyou have to ask again to add server
> behind firewall...
> 
> At these day, somebody (one of my forum member)  add some files to my
> server as attachment. I saw that this files contain virusus. I think these
> files botnet clients. I deleted this forum messages and attachment. (I think
> some of my other members download this files. :( )
> 
> But at that time my server is up with the help of cisco firewall. 
> And I began to receive HACKING / MALICIOUS ACTIVITY complaint mails from
> diffirent locations and they claim that my IP address is attack their
> server.
> 
> below are some log lines that they sent :
> 
> 
> #Nov 3 02:00:24 2009 .. Nov 3 02:33:14 2009
> # Scan from xxx.xxx.xxx.xxx affecting at least
> # 65 addresses targeting TCP:1024, TCP:3072.
> #
> 
> #Nov 3 01:00:50 2009 .. Nov 3 01:59:00 2009
> # Scan from xxx.xxx.xxx.xxx affecting at least
> # 104 addresses targeting TCP:1024, TCP:3072.
> #
> 
> #Nov 3 00:23:25 2009 .. Nov 3 00:59:55 2009
> # Scan from xxx.xxx.xxx.xxx affecting at least
> # 100 addresses targeting TCP:1024, TCP:3072.
> #
> 
> 
> #Nov 2 23:00:15 2009 .. Nov 2 23:59:58 2009
> # Scan from xxx.xxx.xxx.xxx affecting at least
> # 54 addresses targeting TCP:1024, TCP:3072.
> 
> 
> UIDL Date Source Destination Port Protocole Nombre ASN Pays
> 4aefcca000000000 2009-11-02 22:52:03 xxx.xxx.xxx.xxx u-bordeaux.fr 3072
> tcp 31 11897
> 4aefcca000000000 2009-11-02 22:40:53 xxx.xxx.xxx.xxx u-bordeaux.fr 1024
> tcp 31 11897
> 4aef69ee00000000 2009-11-02 22:29:11 xxx.xxx.xxx.xxx lmd.ens.fr 3072 tcp 8
> 11897
> 4aefcca000000000 2009-11-02 22:52:03 xxx.xxx.xxx.xxx u-bordeaux.fr 3072
> tcp 31 11897
> 4aefcca000000000 2009-11-02 22:40:53 xxx.xxx.xxx.xxx u-bordeaux.fr 1024
> tcp 31 11897
> 4aef69ee00000000 2009-11-02 22:29:11 xxx.xxx.xxx.xxx lmd.ens.fr 3072 tcp 8
> 11897 
> 
> #Nov 20 06:00:59 2009 .. Nov 20 06:59:51 2009
> # Scan from xxx.xxx.xxx.xxx affecting at least
> # 58 addresses targeting TCP:1025, TCP:1057, TCP:1537, TCP:1569,
> TCP:16897, TCP:16929, TCP:17409, TCP:17441, TCP:17921, TCP:17953, TCP:18433,
> TCP:18465, TCP:18945, TCP:18977, TCP:19457, TCP:19489, TCP:19969, TCP:2049,
> TCP:2081, TCP:2561, TCP:2593, TCP:3073, TCP:3105, TCP:33, TCP:513, TCP:545.
> #
> 
> #Nov 20 13:47:47 2009 .. Nov 20 13:59:51 2009
> # Scan from xxx.xxx.xxx.xxx affecting at least
> # 149 addresses targeting TCP:1, TCP:1025, TCP:1057, TCP:1537, TCP:1569,
> TCP:16385, TCP:16417, TCP:16897, TCP:16929, TCP:17409, TCP:17921, TCP:17953,
> TCP:18433, TCP:18465, TCP:18945, TCP:18977, TCP:19457, TCP:19489,
> TCP:19969, TCP:20001, TCP:2049, TCP:2081, TCP:2561, TCP:3073, TCP:3105, TCP:33,
> TCP:3585, TCP:3617, TCP:513, TCP:545.
> #
> 
> Event Date Time, Destination IP, IP Protocol, Target Port, Issue
> Description, Source Port, Event Count
> EventRecord: 20 Nov 2009 11:12:36, 67.34.x.x, 6, 16385, Research Pending ,
> 80, 1
> EventRecord: 20 Nov 2009 11:12:22, 156.99.x.x, 6, 2561, Research Pending ,
> 80, 1
> EventRecord: 20 Nov 2009 11:09:26, 64.128.x.x, 6, 3617, Research Pending ,
> 80, 1
> EventRecord: 20 Nov 2009 11:08:47, 83.170.x.x, 6, 16929, Research Pending
> , 80, 1
> EventRecord: 20 Nov 2009 11:07:47, 24.220.x.x, 6, 20001, Research Pending
> , 80, 1
> EventRecord: 20 Nov 2009 11:07:40, 173.15.x.x, 6, 19969, Research Pending
> , 80, 1
> EventRecord: 20 Nov 2009 11:07:40, 173.15.x.x, 6, 19969, Research Pending
> , 80, 1
> EventRecord: 20 Nov 2009 11:06:38, 156.99.x.x, 6, 3585, Research Pending ,
> 80, 1
> EventRecord: 20 Nov 2009 11:06:12, 194.85.x.x, 6, 20001, Research Pending
> , 80, 1
> EventRecord: 20 Nov 2009 11:05:43, 194.85.x.x, 6, 16417, Research Pending
> , 80, 1
> EventRecord: 20 Nov 2009 11:05:36, 156.99.x.x, 6, 3617, Research Pending ,
> 80, 1
> EventRecord: 20 Nov 2009 11:05:20, 64.128.x.x, 6, 19969, Research Pending
> , 80, 1
> EventRecord: 20 Nov 2009 11:03:37, 84.12.x.x, 6, 3105, Research Pending ,
> 80, 1
> EventRecord: 20 Nov 2009 11:02:34, 84.12.x.x, 6, 16897, Research Pending ,
> 80, 1
> 
> 
> 33:42.1 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.68,
> 1537, sbg.fmew.com -
> 47:31.9 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.71,
> 2561, mac.fmew.com -
> 49:40.6 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.5, 1,
> fmewservices.fmew.com -
> 51:56.5 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.27,
> 2593 -
> 53:23.7 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.37,
> 18433, jma.fmew.com -
> 54:37.5 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.42,
> 17953, mjt.fmew.com -
> 55:41.4 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.46,
> 16385, emp.fmew.com -
> 56:51.6 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.86,
> 16417 -
> 57:59.0 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.94,
> 18977 -
> 59:21.5 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.21,
> 1057 -
> 03:50.4 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.104,
> 2049 -
> 04:56.8 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.36,
> 1057 -
> 06:13.6 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.79,
> 16897 -
> 07:19.8 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.33,
> 1025 -
> 10:27.5 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.116,
> 3585 -
> 11:34.2 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.126,
> 17953 -
> 12:34.7 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.16,
> 16929 -
> 13:50.4 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.99,
> 19457 -
> 14:57.8 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.110,
> 545 -
> 16:15.6 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.13,
> 20001 -
> 17:17.1 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.27,
> 18465 -
> 20:41.4 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.77,
> 17409 -
> 21:52.4 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.81,
> 17953 -
> 24:24.6 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.92,
> 17441 -
> 29:41.8 TCP checksum error - xxx.xxx.xxx.xxx, 500, WAN - 67.151.154.44,
> 20001 -
> 
> 
> The following is a list of types of activity that may appear in this
> report:
> BEAGLE BEAGLE3 BLASTER BOTNETS BOTS BRUTEFORCE
> DAMEWARE DEFACEMENT DIPNET DNSBOTS MALWAREURL MYDOOM
> NACHI PHATBOT PHISHING ROUTERS SCAN445 SCANNERS
> SINIT SLAMMER SPAM SPYBOT TOXBOT
> 
> etc. ....
> 
> 
> Like this tens of mail sent to me and softlayer abuse department.
> 
> And softlayer ask me to stop this activity or stop my server.
> And I check my server with know security, system auditing tool and rootkit
> scanners. Rootkit Hunter, lynsis and chkrootkit.
> 
> nothing found.
> 
> Also third party management company audit my server and give me a report
> that my server is clean and make hardening on myserver. But they advise me
> switch back to apache (because they no experience with nginx)
> 
> After that I receive complaint mails again.
> 
> So, 3 days ago made a os reload, setup a clean system and I switched back
> to apache and complaint mails stop for 3 days.
> 
> But Apache couldn't handle request. my server load is very high over 100,
> sometimes over 300..
> I lose my google indexes also my members complaint about unreachable site.
> 
> I want to switch back to nginx. But Softlayer warn me about if they
> receive this kind od abuse mails cut my server activities.
> 
> Have you ever been experiencing this kinf of situation ? What do you
> advise me ? (sorry for my english)
> 
Fix your application (vbulletin). If you can't do that then go back to your Apache setup and use something like mod_security (http://www.modsecurity.org/) with it or any other WAF. Harden your PHP since it seems that all your attacks where introduced by something tunneled over vbulletin (which is PHP) into your system and then executed/triggered from/by within PHP. I would say that one of your users has uploaded some kind of scanning toolkit on your server and then misusing your server to scan other systems. Don't allow the user that is running PHP to execute tools that a normal PHP setup does not need. Nail down your file system (for example: mount your temporary directories with "noexec" and do the same for your upload directory, etc). Use something like SELinux / RBAC / grsecurity / etc to prevent your PHP interpreter to go wild. Add an IDS / NIDS / PIDS / etc and act as soon as possible if something strange is going on. Use something like Fail2Ban to parse logs and act on significant issues. Use something like PSAD to prevent idiots scanning your system. Use a firewall / IPtables / etc to prevent your system making strange connections to the outside world. If you are not familiar with IPtables then use something like Shorewall and install it on your system and don't just check inbound but do check outbound as well. Close every not needed port or application on your system. Double secure your logins from external (don't allow root to log into ssh, use AllowGroups/AllowUsers to limit who can log in, use unprivileged user to log into ssh and su to root, etc). If you are still staying on Apache then use something like mod_evasive to prevent one single system from outside to bring your Apache down. If you are still staying on Apache then use something recent that is not such a big security issue as the older Apache versions (look up the therm "Slowloris" if you need a good example what I mean). etc, etc, etc... Just do the normal things every good sysadmin/hoster would do. I am pretty sure that nginx is not your problem. But I understand if you say that with Apache you don't have those issues. It's normal human behavior to think in pictures (I have problems with my page. Hmmm.... I use nginx. Hmmm. Format system, install fresh OS, install Apache. Hmm... No problem so far. Okay! I got it! It's nginx.) instead of taking the time to understand what the problem is and THINK on the problem and solution. But hey! It's your install. If you think that it is nginx then it MUST be nginx. I would not be surprised if in some days you would come back here and tell us the same story has happened with Apache as HTTPD.

Oh! And one last advice: Do not trust anybody! If a security company is telling you that YOUR system is secure then fine and dandy but it's you that need to guarantee and understand the security of your system. Not any one else. You need to UNDERSTAND what is going on with your system and YOU need to KNOW that and why your system is secure. Some one telling you that is secure is not going to take away that responsibility from you. A drug dealer will always ensure that what you buy from him is 100% risk free and and and... but it's you that is going to consume that stuff and it's you that is risking to die. Not him. So don't just blindly trust. Turn on the gears in your head and THINK and ACT but don't just follow blindly. You are not a sheep!

> Best regards
> 
> Posted at Nginx Forum:
> http://forum.nginx.org/read.php?2,27636,27636#msg-27636
> 
> 
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://nginx.org/mailman/listinfo/nginx

-- 
Sarah Kreuz, die DSDS-Siegerin der Herzen, mit ihrem eindrucksvollen           
Debütalbum "One Moment in Time". http://portal.gmx.net/de/go/musik