Log Parsing - Near Real Time

[Ed W]

On 01/08/2011 14:53, John Macleod wrote:
> I'm looking for a near real-time script to parse log files and insert interesting data into a db.
> 
> Does anyone know of an existing script to do this?

I don't think anyone said rsyslog yet?  Logs directly into a database if
you want, optionally passing through some kind of parser first. It has
spooling to disk in case target cant keep up and semi reliable network
modes.  I heard someone say that Fedora had switched their default
syslog to rsyslog (confirmation?) so hopefully it's not too niche for
you... (obviously I believe it can read directly from a file or pipe...)

Alternatively ask nginx to log to some local fifo and write your own
spooler? Beware blocking if you don't keep up though...

Finally note that there have been some previous patches to nginx to add
syslog logging. I personally believe this is useful for many classes of
problem, but I believe that the position by Igor is that it's considered
too slow to keep up with nginx? (I think I have seen Thrift patches in
the past also?).

I personally don't use nginx to the limit and would love to see syslog
logging in standard nginx, even if it limited maximum performance...
Perhaps if there are a core of similar interested users we could
interest (or pay) Igor to consider adding such a feature, caveat the
limitations?

Good luck

Ed W

P.S. This link has some suggestions on logging to a fifo and catching
the output in syslog-ng
	http://www.spinics.net/lists/centos/msg109747.html