Status of Nginx centralised logging

[Matthieu Tourne]

On Wed, Jun 6, 2012 at 3:39 PM, Jonathan Matthews
<contact at jpluscplusm.com> wrote:
> Hi all -
>
> As many of us will have discovered over the course of our careers,
> centralised/non-local logging is an important part of any non-trivial
> infrastructure.
>
> I'm aware of different patches that add syslogging capabilities to
> different Nginx versions, but I've yet to see an official description
> of how we should achieve non-local logging. Preferably syslog,
> personally speaking, but anything scalable, supportable, debug-able
> and sane would, I feel, be acceptable to the wider community.
>
> I'm aware of at least the following options, but I feel they're all
> lacking to some degree:
>
> * log to local disk and syslog/logstash/rsync them off: undesirable
> due to the management overhead of the additional logging process/logic
> + the wasted disk I/O when creating the per-request logs
>
> * use post_action to hit a logging endpoint after each request: would
> add overhead to both configuration and network traffic; post_action
> has been described as "a bit of a hack", IIRC
>
> * use syslog patches: not "official", hence troublesome to debug
> on-list; rightly or wrongly, the usual response of "please replicate
> problem without 3rd party patches" would cause problems when debugging
> production systems
>
> * use another 3rd-party logging protocol, e.g. statsd, redis: as
> similarly unsupportable as syslog patches
>
> I wonder if someone could officially comment on the potential for
> resolving this gap in Nginx's feature set? Or perhaps I'm missing
> something ... :-)
>

Hi!

I think this thread from a few days ago might answer some of your questions :
http://forum.nginx.org/read.php?2,227234

We use log_by_lua to aggregate data, and then fetch the aggregated
data from another location.
In turns this feeds into a bigger centralized system (OpenTSDB).
This way we don't have to output one log line, or hit one logging
endpoint for each request that goes through.

One interesting thing we've noticed while developing this is that even
an error in our logging code really doesn't affect serving pages, so
this might be the safest use case for 3rd party modules you could
imagine.

Currently your only other official alternative to get data, besides
the access_log is the stub_status module :
http://wiki.nginx.org/HttpStubStatusModule

Hope this helps,
Matthieu