Nginx error log parser

[mohit Agrawal]

Hi All,


I have something like this. I tested the `tail -f /var/log/nginx/error.log
| awk -f  /var/log/nginx/test.awk` part and it just works fine. But when i
try to run it through fluentd, it doesn't do anything. Any idea why ?


<source>

      @type exec

      format json

      tag sample

      command tail -f /var/log/nginx/error.log | awk -f
/var/log/nginx/test.awk

  </source>


<match sample >

  @type stdout

</match>


Also /var/log/nginx/test.awk, is as follow :


################# tst.awk #################

BEGIN {FS = "," }

{

    split($1, m, "\ ")

    gsub(/ /, "", $2)

    split($2, a, ":")

    gsub(/ /, "", $3)

    split($3, b, ":")

    gsub(/ /, "", $4)

    split($4, c, ":")

    gsub(/ /, "", $5)

    split($5, d, ":")

    printf "%s", "{"

    printf "\"%s\" : \"%s\",",a[1], a[2]

    printf "\"%s\" : \"%s\",",b[1], b[2]

    #printf "%s",$3 ","

    #printf "%s",$5 ","

    #printf "%s",$4 ","

    printf "\"%s\" : %s,",c[1], c[2]

    printf "\"%s\" : %s,",d[1], d[2]

    split(m[10], e, "\"")

    printf " \"reason\": \"%s %s %s %s %s\"}\n", m[6], m[7], m[8], m[9], e[2
]




}

#############################################




On 10 January 2018 at 17:53, mohit Agrawal <mohit3081989 at gmail.com> wrote:

> Thanks Aziz for this, I get your point, but can we do awking in fluentd
> cons file ? Basically we are looking for realtime awking a nginx error log
> file, how heavy this would be according to you.
>
> On 10 January 2018 at 17:44, Aziz Rozyev <arozyev at nginx.com> wrote:
>
>> If you need parse exactly the same format, as you’ve shown in you
>> question, it’s fairly easy to create something e.g. perl/awk/sed script.
>>
>> for instance:
>>
>> ################# tst.awk #################
>> BEGIN {FS = "," }
>> {
>>     split($1, m, "\ ")
>>     printf "%s", "{ "
>>     printf "%s",$2
>>     printf "%s",$3
>>     printf "%s",$5
>>     printf "%s",$4
>>     printf "reason: %s %s %s %s \"%s\"\n", m[6], m[7], m[8], m[9], m[10]
>>     print " }”
>>
>> }
>> #############################################
>>
>>
>> result:
>>
>> echo 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
>> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
>> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com" | awk -f
>> /tmp/test.awk
>> {  client: xx.xx.xx.xx server: www.xyz.com host: www.xyz.com request:
>> GET /api/xyz HTTP/1.1reason: limiting connections by zone "rl_conn"
>>  }
>>
>>
>> br,
>> Aziz.
>>
>>
>>
>>
>>
>> > On 10 Jan 2018, at 14:45, mohit Agrawal <mohit3081989 at gmail.com> wrote:
>> >
>> > Yeah I have tried grok / regex pattern as well. But not extensive
>> success that I achieved. grok didn't work for me, I tried regex then it was
>> able to segregate time , pid, tid, log_level and message. I also need
>> message break up for above pattern
>> >
>> > On 10 January 2018 at 17:12, Aziz Rozyev <arozyev at nginx.com> wrote:
>> > Hi Mohit,
>> >
>> > check the second reply. I’m not sure that there is a conventional
>> pretty printing
>> > tools for nginx error log.
>> >
>> >
>> > br,
>> > Aziz.
>> >
>> >
>> >
>> >
>> >
>> > > On 10 Jan 2018, at 14:37, mohit Agrawal <mohit3081989 at gmail.com>
>> wrote:
>> > >
>> > > Hi Aziz,
>> > >
>> > > log_format directive only provides formatting for access log, I am
>> looking to format error.log which doesn't take log_format directive.
>> > > Above example that I gave is just for nginx error logs.
>> > >
>> > > Thanks
>> > >
>> > > On 10 January 2018 at 15:26, Aziz Rozyev <arozyev at nginx.com> wrote:
>> > > btw, after re-reading the your questing, it looks like you need
>> something like logstash grok filter.
>> > >
>> > > br,
>> > > Aziz.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > > On 10 Jan 2018, at 11:45, mohit Agrawal <mohit3081989 at gmail.com>
>> wrote:
>> > > >
>> > > > Hi ,
>> > > >
>> > > > I am looking to parse nginx error log so as to find out which
>> particular IP is throttled during specific amount of time on connection
>> throttling  / request throttling. The format looks like :
>> > > >
>> > > > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
>> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
>> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
>> > > > And the sample that I am looking for is :
>> > > >
>> > > > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com",
>> "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone
>> "rl_conn""}
>> > > > so that I can pass it through ELK stack and find out the root ip
>> which is causing issue.
>> > > >
>> > > >
>> > > > --
>> > > > Mohit Agrawal
>> > > > _______________________________________________
>> > > > nginx mailing list
>> > > > nginx at nginx.org
>> > > > http://mailman.nginx.org/mailman/listinfo/nginx
>> > >
>> > > _______________________________________________
>> > > nginx mailing list
>> > > nginx at nginx.org
>> > > http://mailman.nginx.org/mailman/listinfo/nginx
>> > >
>> > >
>> > >
>> > > --
>> > > Mohit Agrawal
>> >
>> >
>> >
>> >
>> > --
>> > Mohit Agrawal
>>
>>
>
>
> --
> Mohit Agrawal
>



-- 
Mohit Agrawal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20180110/ff0ba632/attachment-0001.html>