Log Parsing - Near Real Time

[David Yu]

On Tue, Aug 2, 2011 at 3:08 AM, Harold Sinclair <haroldsinclair at gmail.com>wrote:

> I cobbled something like this together with open source tools and have been
> using it on hundreds of servers.. pls contact me offline if you'd like a
> copy :)
>
Or you can share it on github to make it open to anyone.

>
> -Harold
>
>
> On Mon, Aug 1, 2011 at 2:57 PM, Dennis Jacobfeuerborn <
> dennisml at conversis.de> wrote:
>
>> An alternative is to tail -F (aka. "--follow=name --retry") the log file
>> and pipe the output into a script. This allows you to parse the entries as
>> they come in and rotate the log file as often as you want independently of
>> the parsing script.
>>
>> Regards,
>>  Dennis
>>
>> On 08/01/2011 04:57 PM, Randy Parker wrote:
>>
>>> My app has a request that opens the log file, fseeks to the end, backs up
>>> as many bytes as it takes to get to the size the log file was on the last
>>> similar request by that user, and runs a regex over the novel part to get
>>> interesting metrics before closing the file.  Since this happens less
>>> than
>>> once per minute, I have not done anything fancy to optimize.
>>>
>>> - Randy
>>>
>>> On Mon, Aug 1, 2011 at 10:39 AM, Reinis Rozitis <r at roze.lv
>>> <mailto:r at roze.lv>> wrote:
>>>
>>>        I'm looking for a near real-time script to parse log files and
>>>        insert interesting data into a db.
>>>        Does anyone know of an existing script to do this?
>>>
>>>
>>>    You can check/try http://www.splunk.com
>>>
>>>    rr
>>>
>>>
>>>    ______________________________**___________________
>>>    nginx mailing list
>>>    nginx at nginx.org <mailto:nginx at nginx.org>
>>>    http://mailman.nginx.org/__**mailman/listinfo/nginx<http://mailman.nginx.org/__mailman/listinfo/nginx>
>>>    <http://mailman.nginx.org/**mailman/listinfo/nginx<http://mailman.nginx.org/mailman/listinfo/nginx>
>>> >
>>>
>>>
>>>
>>>
>>> --
>>> http://mobiledyne.com
>>>
>>>
>>> ______________________________**_________________
>>> nginx mailing list
>>> nginx at nginx.org
>>> http://mailman.nginx.org/**mailman/listinfo/nginx<http://mailman.nginx.org/mailman/listinfo/nginx>
>>>
>>
>> ______________________________**_________________
>> nginx mailing list
>> nginx at nginx.org
>> http://mailman.nginx.org/**mailman/listinfo/nginx<http://mailman.nginx.org/mailman/listinfo/nginx>
>>
>
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
>


-- 
When the cat is away, the mouse is alone.
- David Yu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20110802/1ba8aecd/attachment-0001.html>