Nginx Logging to Zeromq Module - Sparkngin

[Steve Morin]

Piotr,
  You've worked on something similar in the past what's your opinion on the
performance of a zeromq module vs redis?
-Steve


On Tue, Dec 10, 2013 at 7:00 AM, W-Mark Kubacki <wmark+nginx at hurrikane.de>wrote:

> 2013/12/10 MAGNIEN, Thierry <thierry.magnien at sfr.com>:
> >
> > I don't know much about Zeromq performance, but we pushed redis to about
> 300k commands per second on our production servers (8 cores), and
> benchmarks on better hardware (24 cores) show it could handle close to
> 500k/s.
>
> ØMQ should come pretty close or even exceed that. You will require a
> free core for its process, though.
>       Nevertheless implementing it looks rather straightforward (I've
> done it in a different product) and with an event consumer (Nginx
> being the producer) you will get one part of a fantastic local
> analytics tool, similar to Piwik.
>
> Here's an introduction to authentication and encryption:
> [6] http://hintjens.com/blog:49#toc6
>
> Here's how you can prototype the plugin using Lua:
>
> [7a] http://wiki.nginx.org/HttpLuaModule#init_by_lua — initialize context
> [7b]
> http://wiki.nginx.org/HttpLuaModule#Data_Sharing_within_an_Nginx_Worker
> — the place for PUB sockets
> [7c]
> https://github.com/mtourne/nginx_log_by_lua/blob/master/conf/nginx.conf
> — collect data for logging, and send a multipart message here…
> [7d]
> https://github.com/Neopallium/lua-zmq/blob/master/examples/client_multipart.lua
> — … that's how the call looks like
>
> Everything else on the consumer will be counters of some sort. You
> could still use Redis there for small sites, though.
>
> Other noteworthy data structures for the consumer:
> [8]
> http://blog.aggregateknowledge.com/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/
> [9] http://billmill.org/bloomfilter-tutorial/
> [10]
> http://www.igvita.com/2010/01/06/flow-analysis-time-based-bloom-filters/
>
> I've written this in the hope that other developers of Nginx modules
> have an idea on where to start (their sockets, for example ;-) ). We
> should move discussion of further details to a sparkngin group.
>
> --
> Mark
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20131210/9804cd51/attachment-0001.html>