gzip causing high cpu load for streaming traffic

[steveh]

Ok this is weird I noticed tonight high CPU load on two out of three of
our stream servers, which are serving mp4 and flv via the pseudo
streaming modules.

After investigating the differences between the two high CPU boxes and
the one low CPU machine the only notable difference was that on the two
high CPU boxes the config was including our standard gzip options:-
[code]
gzip on;
gzip_static on;
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 6;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript
text/xml application/xml application/xml+rss text/javascript;
# make sure gzip does not lose large gzipped js or css files
gzip_buffers 16 8k;

# Disable gzip for certain browsers.
gzip_disable "MSIE [1-6].(?!.*SV1)";
[/code]

Now as you can see the types is set to text based types so .mp4 and .flv
files really shouldn't be touched, so I looked else where. After much
looking I really couldn't find any significant differences so I added
the include of the above to the low CPU machine. Sure enough the CPU
instantly jumped up with each nginx process using 20-40% cpu instead of
0-2% it was before the reload.

Still not convinced and after double checking the gzip config to confirm
it really shouldn't be going anywhere near the streaming content I added
gzip off and gzip_static off to the *.mp4 and *.flv locations in the
config on the high CPU machine and reloaded. An hour or two later ( time
for the existing users streams to finish ) and the CPU load has already
dropped to the very low values I expect from nginx for the connection
load on the machine e.g. 0 - 2% per nginx process instead of the 40-90%
I was seeing.

Just to confirm I checked the network load and connected users and sure
enough the actual connections and bandwidth is still the same so the
machine is still under the same amount of connection load as before just
with significantly lower CPU.

So what's going on? Have I missed something totally obvious? Or is there
a nasty bug which means when gzip is enabled in this manner it causes
significant processing overhead for items its not even meant to be
processing?

In case its relavent our main http block includes:
[code]
include mime.types;
default_type application/octet-stream;
[/code]

The current streaming bocks, which include the workaround (turning gzip
off) are:-

[code]
# Actual Streaming
location ~ /streaming/(.*\.mp4)$ {
    # Fix strange CPU usage caused by gzip
    gzip off;
    gzip_static off;

    limit_rate_after 10m;
    limit_rate 400k;
    alias /streaming/$1;
    internal;
    mp4;
}

location ~ /streaming/(.*\.flv)$ {
    # Fix strange CPU usage caused by gzip
    gzip off;
    gzip_static off;

    limit_rate_after 10m;
    limit_rate 400k;
    alias /streaming/$1;
    internal;
    flv;
}
[/code]

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,164484,164484#msg-164484