nginx worker stuck, potential mod_zip bug

[Maxim Dounin]

Hello!

On Fri, Feb 17, 2012 at 04:03:46PM -0800, W. Andrew Loe III wrote:

> I'm running an older version of nginx (0.7.67) with mod_zip 1.1.6. I
> believe we have a found a rare bug, I'm trying to figure out of it is
> with mod_zip or with nginx, and if upgrading nginx will potentially
> resolve it.
> 
> The symptom is a worker process getting "stuck" at 100% CPU, leaving
> all connections in CLOSE_WAIT, and servicing no requests. It appears
> that the trigger for this is downloading an archive from mod_zip, but
> we have never been able to reproduce it, only observe it in
> production.

[...]

> I also had the log in debug mode (it is incredibly large ~ 150GB
> uncompressed) and it is completely filled with the following little
> loop:
> 
> 
> 2012/02/17 11:58:45 [debug] 10150#0: *1888 mod_zip: entering
> subrequest body filter
> 2012/02/17 11:58:45 [debug] 10150#0: *1888 http postpone filter
> "/s3/bucket/key" 0000000000000000
> 2012/02/17 11:58:45 [debug] 10150#0: *1888 copy filter: 0 "/s3/bucket/key"
> 2012/02/17 11:58:45 [debug] 10150#0: *1888 pipe write busy: 8192
> 2012/02/17 11:58:45 [debug] 10150#0: *1888 pipe write: out:0000000000000000, f:1
> 2012/02/17 11:58:45 [debug] 10150#0: *1888 http output filter "/s3/bucket/key"
> 2012/02/17 11:58:45 [debug] 10150#0: *1888 copy filter: "/s3/bucket/key"
> 2012/02/17 11:58:45 [debug] 10150#0: *1888 mod_zip: entering
> subrequest body filter
> 2012/02/17 11:58:45 [debug] 10150#0: *1888 http postpone filter
> "/s3/bucket/key" 0000000000000000

[...]

> Am I right in assuming this is a bug in mod_zip, it looks like a
> buffer is never being drained to the client?

Quick look though mod_zip sources suggests it doesn't do anything 
for this request (subrequest).  I would rather think you've hit 
something like this problem:

http://trac.nginx.org/nginx/changeset/4136/nginx

Try upgrading to 1.1.4+/1.0.7+ to see if it helps.

Maxim Dounin