Log Parsing - Near Real Time

[Harold Sinclair]

I cobbled something like this together with open source tools and have been
using it on hundreds of servers.. pls contact me offline if you'd like a
copy :)

-Harold

On Mon, Aug 1, 2011 at 2:57 PM, Dennis Jacobfeuerborn <dennisml at conversis.de
> wrote:

> An alternative is to tail -F (aka. "--follow=name --retry") the log file
> and pipe the output into a script. This allows you to parse the entries as
> they come in and rotate the log file as often as you want independently of
> the parsing script.
>
> Regards,
>  Dennis
>
> On 08/01/2011 04:57 PM, Randy Parker wrote:
>
>> My app has a request that opens the log file, fseeks to the end, backs up
>> as many bytes as it takes to get to the size the log file was on the last
>> similar request by that user, and runs a regex over the novel part to get
>> interesting metrics before closing the file.  Since this happens less than
>> once per minute, I have not done anything fancy to optimize.
>>
>> - Randy
>>
>> On Mon, Aug 1, 2011 at 10:39 AM, Reinis Rozitis <r at roze.lv
>> <mailto:r at roze.lv>> wrote:
>>
>>        I'm looking for a near real-time script to parse log files and
>>        insert interesting data into a db.
>>        Does anyone know of an existing script to do this?
>>
>>
>>    You can check/try http://www.splunk.com
>>
>>    rr
>>
>>
>>    ______________________________**___________________
>>    nginx mailing list
>>    nginx at nginx.org <mailto:nginx at nginx.org>
>>    http://mailman.nginx.org/__**mailman/listinfo/nginx<http://mailman.nginx.org/__mailman/listinfo/nginx>
>>    <http://mailman.nginx.org/**mailman/listinfo/nginx<http://mailman.nginx.org/mailman/listinfo/nginx>
>> >
>>
>>
>>
>>
>> --
>> http://mobiledyne.com
>>
>>
>> ______________________________**_________________
>> nginx mailing list
>> nginx at nginx.org
>> http://mailman.nginx.org/**mailman/listinfo/nginx<http://mailman.nginx.org/mailman/listinfo/nginx>
>>
>
> ______________________________**_________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/**mailman/listinfo/nginx<http://mailman.nginx.org/mailman/listinfo/nginx>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20110801/bf5bdc84/attachment-0001.html>