Nginx Logging to Zeromq Module - Sparkngin

[W-Mark Kubacki]

2013/12/10 MAGNIEN, Thierry <thierry.magnien at sfr.com>:
>
> I don't know much about Zeromq performance, but we pushed redis to about 300k commands per second on our production servers (8 cores), and benchmarks on better hardware (24 cores) show it could handle close to 500k/s.

ØMQ should come pretty close or even exceed that. You will require a
free core for its process, though.
      Nevertheless implementing it looks rather straightforward (I've
done it in a different product) and with an event consumer (Nginx
being the producer) you will get one part of a fantastic local
analytics tool, similar to Piwik.

Here's an introduction to authentication and encryption:
[6] http://hintjens.com/blog:49#toc6

Here's how you can prototype the plugin using Lua:

[7a] http://wiki.nginx.org/HttpLuaModule#init_by_lua — initialize context
[7b] http://wiki.nginx.org/HttpLuaModule#Data_Sharing_within_an_Nginx_Worker
— the place for PUB sockets
[7c] https://github.com/mtourne/nginx_log_by_lua/blob/master/conf/nginx.conf
— collect data for logging, and send a multipart message here…
[7d] https://github.com/Neopallium/lua-zmq/blob/master/examples/client_multipart.lua
— … that's how the call looks like

Everything else on the consumer will be counters of some sort. You
could still use Redis there for small sites, though.

Other noteworthy data structures for the consumer:
[8] http://blog.aggregateknowledge.com/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/
[9] http://billmill.org/bloomfilter-tutorial/
[10] http://www.igvita.com/2010/01/06/flow-analysis-time-based-bloom-filters/

I've written this in the hope that other developers of Nginx modules
have an idea on where to start (their sockets, for example ;-) ). We
should move discussion of further details to a sparkngin group.

-- 
Mark