gzip causing high cpu load for streaming traffic

[Maxim Dounin]

Hello!

On Fri, Jan 07, 2011 at 08:39:52PM -0500, steveh wrote:

> Ok this is weird I noticed tonight high CPU load on two out of three of
> our stream servers, which are serving mp4 and flv via the pseudo
> streaming modules.
> 
> After investigating the differences between the two high CPU boxes and
> the one low CPU machine the only notable difference was that on the two
> high CPU boxes the config was including our standard gzip options:-
> [code]
> gzip on;
> gzip_static on;
> gzip_http_version 1.1;
> gzip_vary on;
> gzip_comp_level 6;
> gzip_proxied any;
> gzip_types text/plain text/css application/json application/x-javascript
> text/xml application/xml application/xml+rss text/javascript;
> # make sure gzip does not lose large gzipped js or css files
> gzip_buffers 16 8k;
> 
> # Disable gzip for certain browsers.
> gzip_disable "MSIE [1-6].(?!.*SV1)";
> [/code]
> 
> Now as you can see the types is set to text based types so .mp4 and .flv
> files really shouldn't be touched, so I looked else where. After much
> looking I really couldn't find any significant differences so I added
> the include of the above to the low CPU machine. Sure enough the CPU
> instantly jumped up with each nginx process using 20-40% cpu instead of
> 0-2% it was before the reload.
> 
> Still not convinced and after double checking the gzip config to confirm
> it really shouldn't be going anywhere near the streaming content I added
> gzip off and gzip_static off to the *.mp4 and *.flv locations in the
> config on the high CPU machine and reloaded. An hour or two later ( time
> for the existing users streams to finish ) and the CPU load has already
> dropped to the very low values I expect from nginx for the connection
> load on the machine e.g. 0 - 2% per nginx process instead of the 40-90%
> I was seeing.
> 
> Just to confirm I checked the network load and connected users and sure
> enough the actual connections and bandwidth is still the same so the
> machine is still under the same amount of connection load as before just
> with significantly lower CPU.
> 
> So what's going on? Have I missed something totally obvious? Or is there
> a nasty bug which means when gzip is enabled in this manner it causes
> significant processing overhead for items its not even meant to be
> processing?

Working with "gzip_static on;" implies extra open() syscall per 
request (it work's based on .gz file presense, not gzip_types) and 
isn't recommended unless actually used.

It shouldn't make any difference with flv handler activated though 
(flv disables general content handlers and gzip_static in 
particular).  Most likely the same applies to mp4.

Working with "gzip on;" implies extra check for content-type per 
request (and some minimal checks during response sending).  This 
shouldn't cause much load.

> In case its relavent our main http block includes:
> [code]
> include mime.types;
> default_type application/octet-stream;
> [/code]
> 
> The current streaming bocks, which include the workaround (turning gzip
> off) are:-
> 
> [code]
> # Actual Streaming
> location ~ /streaming/(.*\.mp4)$ {
>     # Fix strange CPU usage caused by gzip
>     gzip off;
>     gzip_static off;
> 
>     limit_rate_after 10m;
>     limit_rate 400k;
>     alias /streaming/$1;
>     internal;
>     mp4;
> }
> 
> location ~ /streaming/(.*\.flv)$ {
>     # Fix strange CPU usage caused by gzip
>     gzip off;
>     gzip_static off;
> 
>     limit_rate_after 10m;
>     limit_rate 400k;
>     alias /streaming/$1;
>     internal;
>     flv;
> }

Could you please check if it's gzip or gzip_static which causes 
high cpu usage?

Also, could you please provide some more details about nginx 
you're using?  I.e. nginx -V output, please.

Maxim Dounin