Nginx Log Parser

[valor]

Hello Everyone,

First pardon me if this have already been discussed in the russian
mailinglist.
Second, thank you igor for this wonderful software, it's amazing speed
and stability.

After googling and searching around I've not found a single page with
information related to take advantage of nginx cache logging and tools
to parse this data. So decided to create this topic to first check
what's everyone else using and then if there's nothing ready for
production, do it myself and share it here :-).

1) Common basic web stats

By default nginx combined format is identical to apache, so out of the
box anyone can generate graphs of nginx traffic by using any software
that supports apache: Awstats, Webalizer, etc. This is not a problem in
nginx, setup of these softwares for nginx have not a single difference
with setup for apache logs. 

2) Web stats of cache usage

It's well documented on the wiki (
http://wiki.nginx.org/NginxHttpUpstreamModule ) how to add to the log
useful information regarding caching:
upstream_cache_status (hit,miss,expired,stale,updating,etc)
upstream_status (status returned by apache)
upstream_response_time (time taken by backend to process the request)

However, I've been completely unable to find a single reference to
anyone having this information processed nicely by log analyzers
allowing administrators to see nginx cache stats. 

What are other people using for this? I could not find any hack around
to show this info in awstats, webalizer or visitors(hping).  The only
thing i found is a script made by a guy named Eliot from saltycrane
using python to export log files into mongodb and then use
orbited+stomp+jquery to plot that data into graphs. 
(You can see a presentation about it here:
http://www.saltycrane.com/site_media/html/lla.html )

3) Live status of nginx for sever monitoring/alert software like
Nagios/Zabbix/Cacti/etc.

Another thing I'm looking for it's a way to get live data from nginx to
pipe it into my monitoring software (zabbix).
I'm right now using 0.7.67 so I can't use the new status module, however
AFAIK the status module do not provide information per virtualhost but
nginx-wide stats so even by parsing that status I won't be able to find
the culprit domain for a traffic peak until I parse that domain log
offline with another software.

One way to do this would be to parse log files each minute parsing only
last 60 seconds of activity, however I'm afraid this may result in the
solution being worst than the problem as it will generate load in the
server each minute. Have anyone done something like this already for
nginx cache logs?

Well.... Hope this would be a very long and productive topic :-)

Best Regards from Uruguay

Guzmán Brasó

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,138438,138438#msg-138438