problem with echo_before when proxying a server which sends gzipped content

[Carsten Germer]

Hi everyone,
currently I'm, trying to configure NGINX as a proxy for JSON from the iTunes API.
It's for a small game, iTunes is slow sometimes and the data for the game is mostly the same for a good length of time, anyway.
The JSON from iTunes is to be padded with the original requests callback parameters. For this there are many good posts out on the net, but I can't seem to get the basic echo_* to work.

I boiled my configuration of nginx down to the point where I just use echo_before or echo_after and proxy_pass.

If I append something with echo_after it works fine in browsers and in jQuery.
If I prepend anything with echo_before the answer can't be read by browsers, "curl --compressed" throws "curl: (23) Error while processing content unencoding: invalid block type".

If I configure Firefox with "about:config" to "network.http.accept-encoding:true" it fixes fixes display in Firefox.

When I look in the network tab of chrome console I see that requesting ".../echo-after/" closes the request after 2Xms.
Requesting ".../echo-before" also gets 200 ok but never arrives fully, is shown as "pending" indefinitely.

My best bet is, that it has something to do with gzip-compressed answer from iTunes but I can't find any solution or even hint for my level of understanding of the inner workings of nginx.

One more info: I tried many different combinations of "gzip on|off" and other directives, basically I indiscriminately tried everything I found mentioned somewhere. But, nothing changed the behavior much as far as I could see, so I stripped it out of the configuration again for readability.

<nginx configuration for testing>
# Proxy iTunes for bug hunting
# test by accessing http://json.musiguess.com/itunes/[raw|echo-after|echo-before]/

location /itunes/raw/ {
  # suppress proxying for testing completely
  proxy_cache off;
  # getting a json containing N current top albums from itunes
  proxy_pass http://itunes.apple.com/de/rss/topalbums/limit=2/explicit=true/json;
}
location /itunes/echo-after/ {
  proxy_cache off;
  proxy_pass http://itunes.apple.com/de/rss/topalbums/limit=2/explicit=true/json;
  # echo something after body
  echo_after_body ");";
}
location /itunes/echo-before/ {
  proxy_cache off;
  # echo something before body
  echo_before_body -n "abc(";
  proxy_pass http://itunes.apple.com/de/rss/topalbums/limit=2/explicit=true/json;
}
location /itunes/echo-beforeNafter/ {
  proxy_cache off;
  echo_before_body -n "abc(";
  proxy_pass http://itunes.apple.com/de/rss/topalbums/limit=2/explicit=true/json;
  echo_after_body ");";
}
</nginx configuration for testing>

You are very welcome to access http://json.musiguess.com/itunes/... if it may help you assess my problem.


<info about my installation>
~# nginx -V
nginx version: nginx/1.5.10
built by gcc 4.7.2 (Debian 4.7.2-5)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --add-module=/var/cache/nginx/nginx-1.5.10/src/echo --with-mail --with-mail_ssl_module --with-file-aio --with-http_spdy_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' --with-ld-opt=-Wl,-z,relro --with-ipv6

Echo v0.51
</info about my installation>

Any help is greatly appreciated!
Cheers /Carsten

---
Carsten Germer
Creative Director
intolabs GmbH
http://www.intolabs.net/