Nginx error log parser

[mohit Agrawal]

I finally end up writing my own error log fluentd custom parser in ruby.
It's working now.

Thanks for help anyways, much appreciated


On 11 January 2018 at 14:26, Aziz Rozyev <arozyev at nginx.com> wrote:

> Hi,
>
> seems, that fluentd has an nginx_parser plugin already, another solution
> that probably should work is to use the grep filters,
> something as follows:
>
> <fitler foo.bar>
>    @type grep
>    <regexp>
>       key client
>       patter ^client.*\ $
>    </regexp>
>    <regexp>
>       key server
>       pattern ^server.*\ $
>    </regexp>
>    <regexp>
>       key host
>       pattern ^host.*$
>    </regexp>
>    <regexp>
>       key zone
>       pattern ^zone.*\ $
>    </regexp>
>     …..
> </filter>
>
>
> then use record_trasformer type, to make further modifications. But, I
> didn’t tried above,
> probably it’s something that better to be asked from fluentd community..
>
>
> br,
> Aziz.
>
>
>
>
>
> > On 10 Jan 2018, at 15:23, mohit Agrawal <mohit3081989 at gmail.com> wrote:
> >
> > Thanks Aziz for this, I get your point, but can we do awking in fluentd
> cons file ? Basically we are looking for realtime awking a nginx error log
> file, how heavy this would be according to you.
> >
> > On 10 January 2018 at 17:44, Aziz Rozyev <arozyev at nginx.com> wrote:
> > If you need parse exactly the same format, as you’ve shown in you
> question, it’s fairly easy to create something e.g. perl/awk/sed script.
> >
> > for instance:
> >
> > ################# tst.awk #################
> > BEGIN {FS = "," }
> > {
> >     split($1, m, "\ ")
> >     printf "%s", "{ "
> >     printf "%s",$2
> >     printf "%s",$3
> >     printf "%s",$5
> >     printf "%s",$4
> >     printf "reason: %s %s %s %s \"%s\"\n", m[6], m[7], m[8], m[9], m[10]
> >     print " }”
> >
> > }
> > #############################################
> >
> >
> > result:
> >
> > echo 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com" | awk -f
> /tmp/test.awk
> > {  client: xx.xx.xx.xx server: www.xyz.com host: www.xyz.com request:
> GET /api/xyz HTTP/1.1reason: limiting connections by zone "rl_conn"
> >  }
> >
> >
> > br,
> > Aziz.
> >
> >
> >
> >
> >
> > > On 10 Jan 2018, at 14:45, mohit Agrawal <mohit3081989 at gmail.com>
> wrote:
> > >
> > > Yeah I have tried grok / regex pattern as well. But not extensive
> success that I achieved. grok didn't work for me, I tried regex then it was
> able to segregate time , pid, tid, log_level and message. I also need
> message break up for above pattern
> > >
> > > On 10 January 2018 at 17:12, Aziz Rozyev <arozyev at nginx.com> wrote:
> > > Hi Mohit,
> > >
> > > check the second reply. I’m not sure that there is a conventional
> pretty printing
> > > tools for nginx error log.
> > >
> > >
> > > br,
> > > Aziz.
> > >
> > >
> > >
> > >
> > >
> > > > On 10 Jan 2018, at 14:37, mohit Agrawal <mohit3081989 at gmail.com>
> wrote:
> > > >
> > > > Hi Aziz,
> > > >
> > > > log_format directive only provides formatting for access log, I am
> looking to format error.log which doesn't take log_format directive.
> > > > Above example that I gave is just for nginx error logs.
> > > >
> > > > Thanks
> > > >
> > > > On 10 January 2018 at 15:26, Aziz Rozyev <arozyev at nginx.com> wrote:
> > > > btw, after re-reading the your questing, it looks like you need
> something like logstash grok filter.
> > > >
> > > > br,
> > > > Aziz.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > On 10 Jan 2018, at 11:45, mohit Agrawal <mohit3081989 at gmail.com>
> wrote:
> > > > >
> > > > > Hi ,
> > > > >
> > > > > I am looking to parse nginx error log so as to find out which
> particular IP is throttled during specific amount of time on connection
> throttling  / request throttling. The format looks like :
> > > > >
> > > > > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
> > > > > And the sample that I am looking for is :
> > > > >
> > > > > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com",
> "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone
> "rl_conn""}
> > > > > so that I can pass it through ELK stack and find out the root ip
> which is causing issue.
> > > > >
> > > > >
> > > > > --
> > > > > Mohit Agrawal
> > > > > _______________________________________________
> > > > > nginx mailing list
> > > > > nginx at nginx.org
> > > > > http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > > > _______________________________________________
> > > > nginx mailing list
> > > > nginx at nginx.org
> > > > http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > > >
> > > >
> > > > --
> > > > Mohit Agrawal
> > >
> > >
> > >
> > >
> > > --
> > > Mohit Agrawal
> >
> >
> >
> >
> > --
> > Mohit Agrawal
>
>


-- 
Mohit Agrawal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20180111/df513eed/attachment.html>