You've worked on something similar in the past what's your opinion on the performance of a zeromq module vs redis?

On Tue, Dec 10, 2013 at 7:00 AM, W-Mark Kubacki <wmark+nginx@hurrikane.de> wrote:
2013/12/10 MAGNIEN, Thierry <thierry.magnien@sfr.com>:
> I don't know much about Zeromq performance, but we pushed redis to about 300k commands per second on our production servers (8 cores), and benchmarks on better hardware (24 cores) show it could handle close to 500k/s.

MQ should come pretty close or even exceed that. You will require a
free core for its process, though.
Nevertheless implementing it looks rather straightforward (I've
done it in a different product) and with an event consumer (Nginx
being the producer) you will get one part of a fantastic local
analytics tool, similar to Piwik.

Here's an introduction to authentication and encryption:
[6] http://hintjens.com/blog:49#toc6

Here's how you can prototype the plugin using Lua:

[7a] http://wiki.nginx.org/HttpLuaModule#init_by_lua initialize context
[7b] http://wiki.nginx.org/HttpLuaModule#Data_Sharing_within_an_Nginx_Worker
the place for PUB sockets
[7c] https://github.com/mtourne/nginx_log_by_lua/blob/master/conf/nginx.conf
collect data for logging, and send a multipart message here
[7d] https://github.com/Neopallium/lua-zmq/blob/master/examples/client_multipart.lua
that's how the call looks like

Everything else on the consumer will be counters of some sort. You
could still use Redis there for small sites, though.

Other noteworthy data structures for the consumer:
[8] http://blog.aggregateknowledge.com/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/
[9] http://billmill.org/bloomfilter-tutorial/
[10] http://www.igvita.com/2010/01/06/flow-analysis-time-based-bloom-filters/

I've written this in the hope that other developers of Nginx modules
have an idea on where to start (their sockets, for example ;-) ). We
should move discussion of further details to a sparkngin group.


nginx-devel mailing list