nginx compression-at-edge (in front of a reverse proxy) compresses some content, not other ?

[Igor Sysoev]

On Tue, Apr 27, 2010 at 09:12:13PM -0400, BenDJ wrote:

> I've an nginx + varnish + apache2 stack.  Nginz server as redirector, ssl handshake, and gzip compression.  Varnish serves as a reverse-proxy, and apache 'just' hosts apps & serves content.
> 
> I'm trying to get gzip compression behaving properly.  I'm getting intermittent results -- some content seems to be gzipped, some not.
> 
> I'm looking for some help figuring out WHERE the problem lies, and what to do to fix it.
> 
> In nginx conf, I've,
> 
> 	[code]
> ...
> 	http {
> 		...
> 		gzip              on;
> 		gzip_http_version 1.0;
> 		gzip_comp_level   9;
> 		gzip_proxied      any;
> 		gzip_buffers      16 8k;
> 		gzip_min_length   0;
> 		gzip_types text/plain text/css text/xml text/javascript application/x-javascript;
> 		gzip_disable "MSIE [1-6].(?!.*SV1)";
> 		gzip_vary         on;
> 	
> 		upstream varnish {
> 			server 127.0.0.1:8090 weight=10 max_fails=3 fail_timeout=15s;
> 		}
> 	
> 		server {
> 			listen                    x.y.z.w:443;
> 			...
> 			location / {
> 				proxy_pass             http://varnish;
> 				proxy_redirect         off;
> 				proxy_set_header       Host $host;
> 				proxy_set_header       X-Real-IP $remote_addr;
> 				proxy_set_header       X-Forwarded-For $proxy_add_x_forwarded_for;
> 				proxy_set_header       X-Client-Verify SUCCESS;
> 				proxy_set_header       X-SSL-Subject $ssl_client_s_dn;
> 				proxy_set_header       X-SSL-Issuer  $ssl_client_i_dn;
> 			}
> 		}
> 		...
> [/code]
> 
> 
> In varnish config, I've a
> 
> [code]
> 	...
> 	if (req.http.Accept-Encoding) {
> 		remove req.http.Accept-Encoding;
> 	}
> 	...
> [/code]
> 
> 
> clause, since the compression is to be done only at the nginx 'edge'.

You may set on it on nginx side:
      proxy_set_header     Accept-Encoding  "";
> 
> Apache's compression (gzip, default, or otherwise), is completely disabled.
> 
> Atm, a check of my test site with YSlow complains that .js's are NOT being compressed.  Checking with LiveHTTPHeaders Firefox plugin, shows, e.g., the .js in question NOT being gzipped, but a .gif *is*, e.g.
> 
> 	[code]
> ...
> 	----------------------------------------------------------
> 
> 	https://my.site.com/main/apostrophePlugin/js/jquery.keycodes-0.2.js
> 
> 	
> 
> 	GET /main/apostrophePlugin/js/jquery.keycodes-0.2.js HTTP/1.1
> 	Host: my.site.com
> 	User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4) Gecko/20100417
> 	Accept: */*
> 	Accept-Language: en-us,en;q=0.5
> 	Accept-Encoding: gzip,deflate
> 	Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
> 	Keep-Alive: 115
> 	Connection: keep-alive
> 	Referer: https://my.site.com/main/
> 	Cookie: SESS6fa8cdc2d7064704bbda0c83e2c2588c=94889db68945e19ed6f666b7e00cdd36; symfony=3KOH8Qk0hV%2C%2CvXLi0PK5YmdenP1
> 	If-Modified-Since: Tue, 27 Apr 2010 18:18:18 GMT
> 	If-None-Match: "362e4-1008-4853810064180"
> 	Authorization: Digest username="admin", realm="AUTH my.site.com", nonce="5GopBDmFBAA=99f7be8796e018dde459a07178393d235366ecd9", uri="/main/apostrophePlugin/js/jquery.keycodes-0.2.js", algorithm=MD5, response="b04cb995cd1f86a67197aab3b5a5dbc9", qop=auth, nc=000001c9, cnonce="9dbeaefee4d57b12"
> 	Cache-Control: max-age=0
> 
> 	
> 
> 	HTTP/1.1 304 Not Modified
> 	Server: nginx/0.8.35
> 	Date: Wed, 28 Apr 2010 00:39:48 GMT
> 	Connection: keep-alive
> 	Etag: "362e4-1008-4853beaefee80"
> 	Expires: Sat, 01 May 2010 00:39:48 GMT
> 	Cache-Control: max-age=259200
> 	Content-Length: 0
> 	X-Varnish: 940462008
> 	Age: 0
> 	Via: 1.1 varnish
> 
> 	----------------------------------------------------------
> 
> 	https://my.site.com/apostrophePlugin/images/a-special-blank.gif
> 
> 	GET /apostrophePlugin/images/a-special-blank.gif HTTP/1.1
> 	Host: my.site.com
> 	User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4) Gecko/20100417
> 	Accept: image/png,image/*;q=0.8,*/*;q=0.5
> 	Accept-Language: en-us,en;q=0.5
> 	Accept-Encoding: gzip,deflate
> 	Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
> 	Keep-Alive: 115
> 	Connection: keep-alive
> 	Referer: https://my.site.com/main/
> 	Cookie: SESS6fa8cdc2d7064704bbda0c83e2c2588c=94889db68945e19ed6f666b7e00cdd36; symfony=3KOH8Qk0hV%2C%2CvXLi0PK5YmdenP1
> 
> 	HTTP/1.1 404 Not Found
> 	Server: nginx/0.8.35
> 	Date: Wed, 28 Apr 2010 00:39:48 GMT
> 	Content-Type: text/html; charset=iso-8859-1
> 	Transfer-Encoding: chunked
> 	Connection: keep-alive
> 	Vary: Accept-Encoding, accept-language,accept-charset
> 	Content-Language: en
> 	X-Varnish: 940462009 940461993
> 	Age: 87
> 	Via: 1.1 varnish
> 	Content-Encoding: gzip
> 	----------------------------------------------------------
> 	...
> [/code]
> 
> 
> where, iiuc, the presence/absence of "Content-Encoding: gzip" defines whether or not the item in question was succesfully gzipped.
> 
> Any ideas why/where the .js is not getting gzipped?

In your examples .js response has 304 code without body, so here is
nothing to gzip, and .gif response has 404 code and it is not image,
but text/html, so it is gzipped.

BTW, why do you use varnish instead of nginx built-in proxy cache ?


-- 
Igor Sysoev
http://sysoev.ru/en/