Support for 3rd party zlib libraries

[Maxim Dounin]

Hello!

On Tue, Jul 29, 2014 at 10:46:03PM +0200, Richard Stanway wrote:

> Hello,
> I recently came across a modified version of zlib with code contributed by
> Intel [1] that makes use of modern CPU instructions to increase
> performance. In testing, the performance gains seemed substantial, however
> when I tried to use this version with nginx, the following alert types
> appeared in the error_log on gzip requests:
> 
> 2014/07/30 05:20:29 [alert] 22285#0: *739837 gzip filter failed to use
> preallocated memory: 65536 of 65520 while sending to client
> 2014/07/30 05:40:42 [alert] 29462#0: *230460 gzip filter failed to use
> preallocated memory: 1040 of 4112
> 
> It appears nginx is pre-allocating a buffer based on the original zlib
> memory usage patterns, however the Intel optimized version has slightly
> higher memory requirements due to padding for SSE functions etc.
> 
> Is there a chance this version could be supported by nginx, or a
> configuration option made available to control the allocation size?
> 
> Thanks.
> 
> [1] https://github.com/jtkukunas/zlib

This version indeed uses larger allocations (much larger in some 
cases), and also uses different sizes which confuses nginx code 
when it tries to allocate 8k for deflate state.

Quick hack to do correct preallocations for this zlib version:

--- a/src/http/modules/ngx_http_gzip_filter_module.c	Wed Jul 09 12:27:15 2014 -0700
+++ b/src/http/modules/ngx_http_gzip_filter_module.c	Wed Jul 30 05:57:50 2014 +0400
@@ -521,7 +521,16 @@ ngx_http_gzip_filter_memory(ngx_http_req
      *  *) 5920 bytes on amd64 and sparc64
      */
 
+#if 0
     ctx->allocated = 8192 + (1 << (wbits + 2)) + (1 << (memlevel + 9));
+#endif
+
+    if (conf->level == 1) {
+        wbits = 13;
+    }
+
+    ctx->allocated = 8192 + 16 + (1 << (wbits + 2))
+                     + (1 << (ngx_max(memlevel, 8) + 8)) + (1 << (memlevel + 8));
 }
 
 
@@ -987,7 +996,7 @@ ngx_http_gzip_filter_alloc(void *opaque,
 
     alloc = items * size;
 
-    if (alloc % 512 != 0 && alloc < 8192) {
+    if (alloc % 512 != 0 && alloc < 8192 && items == 1) {
 
         /*
          * The zlib deflate_state allocation, it takes about 6K,

This is expected to work with normal zlib as well, but will be 
suboptimal from memory usage point of view.

Note well that alerts logged aren't really serious - nginx is able 
to handle such cases properly, and will fallback to normal pool 
allocations instead of using preallocated block.  It's just not 
something it expects to ever happen, so it logs alerts to make 
sure it's noticed.  With different zlib libraries out there, we 
probably want to just silence the alert, keeping preallocation 
tuned for normal zlib.

-- 
Maxim Dounin
http://nginx.org/