Problem with Nginx returning correct content, but with 404
austinmills
nginx-forum at nginx.us
Thu Jan 20 19:23:17 MSK 2011
Hello,
We have a setup where we have requests coming into a cache server which
is in front of an app server. The requests hit nginx on the cache
server, which makes the request to a process running on the cache
server. That process will return either the requested content (if in the
cache) or a 404. In the case of a 404, nginx on the cache server should
then turn around and make the same request to the app server. The app
server serves the content, and then nginx on the app server should turn
around and return the response to the client.
The problem we are seeing is that this works fine for requests of the
form /admin/fan_pages, but when requesting css or png content (like
/stylesheets/common.css , or /images/admin/fan_pages/refresh.png ), it
is returning the correct content, but with a 404 code. In this case, the
local process on the cache server is returning a 404, the nginx on the
cache server is requesting the page from the app server (which returns
the page with a 200), but then the cache server is turning around and
serving the correct content to the client with a 404.
Anybody have any ideas on how/why this is happening? I'll include config
files and output below. If there's a simpler or better setup to
accomplish what we're trying to do, I'm happy to try that as well.
Thanks,
--Austin
ngrep output (.185 is the cache server, .165 is the app server):
#
T 24.227.153.188:54091 -> 10.102.38.185:80 [AP]
GET /stylesheets/common.css HTTP/1.1.
Host: test5.involver.com.
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8.
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.
Accept-Language: en-us,en;q=0.5.
Accept-Encoding: gzip,deflate.
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7.
Keep-Alive: 115.
Connection: keep-alive.
Cookie:
involver-auth=eyJlbWFpbCI6ImF1c3Rpbi5taWxsc0BpbnZvbHZlci5jb20iLCJzaWduYXR1%0AcmUiOiJLTDA5TmZvMG9QR3V2OUl3QTFkekFWOG1rU04wdkU1d2UyMzBvSjA3%0ATGhvUzJSTDNQOU5tdjRwM09FOFRcblFlUzUvWkFScmxXT29UeldIOVlONTJa%0AUEFzVzdDOXlQdUI1UUZDZURZR3dLZUgzT2RTMVpKbm96ZTY0Q1xuYjk2QUhU%0Aa25FVGdpWmRHaFA4eVJkaE13SDB3YmVkS25MejNGeC9Wd2I4WDhYSDNVM1Yw%0AMkl3RUwwRjNDXG5BS3h3ekFTMlhINHU4b0RXUVM1VkorZ2c4eTFUS2V0UCtw%0AbUVZbXVwN01pcUlXa3BDRnBBSVN2WU5TeXZcbmQ0ekhXWWhDSUE5Ulo4UXY.
.
####
T 10.102.38.185:42920 -> 10.102.38.165:80 [AP]
GET /stylesheets/common.css HTTP/1.0.
X-Real-IP: 24.227.153.188.
X-Forwarded-For: 24.227.153.188.
Host: test5.involver.com.
Connection: close.
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8.
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.
Accept-Language: en-us,en;q=0.5.
Accept-Encoding: gzip,deflate.
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7.
Cookie:
involver-auth=eyJlbWFpbCI6ImF1c3Rpbi5taWxsc0BpbnZvbHZlci5jb20iLCJzaWduYXR1%0AcmUiOiJLTDA5TmZvMG9QR3V2OUl3QTFkekFWOG1rU04wdkU1d2UyMzBvSjA3%0ATGhvUzJSTDNQOU5tdjRwM09FOFRcblFlUzUvWkFScmxXT29UeldIOVlONTJa%0AUEFzVzdDOXlQdUI1UUZDZURZR3dLZUgzT2RTMVpKbm96ZTY0Q1xuYjk2QUhU%0Aa25FVGdpWmRHaFA4eVJkaE13SDB3YmVkS25MejNGeC9Wd2I4WDhYSDNVM1Yw%0AMkl3RUwwRjNDXG5BS3h3ekFTMlhINHU4b0RXUVM1VkorZ2c4eTFUS2V0UCtw%0AbUVZbXVwN01pcUlXa3BDRnBBSVN2WU5TeXZcbmQ0ekhXWWhDSUE5Ulo4UXY.
.
##
T 10.102.38.165:80 -> 10.102.38.185:42920 [A]
HTTP/1.1 200 OK.
Server: nginx/0.7.65.
Date: Thu, 20 Jan 2011 01:14:02 GMT.
Content-Type: text/css.
Last-Modified: Wed, 19 Jan 2011 22:20:44 GMT.
Connection: close.
Expires: Sun, 17 Jan 2021 01:14:02 GMT.
Cache-Control: max-age=315360000.
Content-Encoding: gzip.
.
...........ZYo.H.~..E#~.n`..iIF..$.......b.. at h.-.0E
$.c.........{...
............?............}~.~.....`..\.y......?.._|?Z.N.z.>...$.=....I...%.....U.....U..D.:a5_..~.nE.....e...V...V.L....rs...YQ.....d6...D.Z......Z...D.E.....y..
..........)e.....7....]yD.tV[^.......1*J.=?#6{q&`.d.G..s...^_...2+x]-2..q.~...t.K.....4!.vbQ.u&`.1I.;.%.../...E......0I.m......o.....4..]u...!....+.....<{..$.Ws6.Av.Q<....UE.&......J)M\..?y7._..#k.m]. at ..H.:.P.G.j...X_Q..+.v+.....10Z..-.y.......xy}./..+.........Eo.-.<....Q..P....\...&..
....."......$.9<.I.O.H?......?.)..Ed..Ncp.....].U.E...47..:.t......r~.....x..i....X..h...E.]......d..gh3uLo:....](...(".c.1..:y.=T...i'...3iW.E.Q(.XB.....PC..o.W..b..t..[....6\Q.io....H...D.......Q../..M
.5k.p at S.}.o<..[.h!k.$.Q4.'...wiR.I....C....2....Z....L..a.......Fs..B...hHo8....c.d....HxK9...'{...%}....h..e.z.>.................?._...^.,(.c
..X2..t.....P.Q.s).............N]..$.1.?.. at ....,s....Z....1X...M...pF..d.,......a...y.....l.P...R.'(h.
......Ek........N..G...O$..#...k..L.t...4=..E^1r.&..SK).....[].X..*j(.U..V.V1H..gQ..u.Nk..m.7&...L.M.....|K.H....,8..I8...J.....y.F....0..T.4.
x.,!....b.O..&.....q.9.f.u...E..<......8...8.A......B&@.&../...........].....OX.~.+nR.z,n..*.....{
....{S......]L........`.......LoG..[3..3.W|.f...t]......}/..X..5.P...Q.w.b.P..Ix.e(.h.T.0.......y.....l[..J...a2..>.R.{%e
.,.2.. at ....ad....[..5.../p...;..Uu.....Y....t4.D._.K..'.U....R.z.*..+...Kp.....L"......"...tb...0.uB..p$+. at D5.{..+Uk..*....M.?[4..D....S.^~x..a
..J.F+.i'...=L2._.........X...._.....2..]..k...o(.lp....\....\)...*....T.......L.O..z...Z9.......RP/I.&......(...q.7[.Pb!..7.>...<./..'.P....c"...A..c
.sf8..........d.MA.;`....H..6.ut2K.. at ....g7....P..6.3p..Z..."....l....}V60.x.y.#...)y.P..X...C}.t...9....7..b..)9.T.....m...^7.%..2!~L..1R....%...:uBt.3.... at ..B.....&.d....r:.....`.A..'..B.y.......|........F...b....n. at .../.y.O.$v.)....u*B..P......f..x8nD.6...T..z.L.{.j...|..#...b......M.u...|)td........G...B.$.wp>6 at q6.8.0.P&j..].l.w..A....+....%N......h.... at .x.C.%i.....:Y].._.l......./,..Q..Lj....|+)...........7.-R.....(..].C......D$..pq.c at .K..........
..2..k....$e.P.... .v..r....U.sx...]...$$$
.4.&.........G.....-...o1...G<....[...~i.....xp.ar.....n...|....W......B+...z.+c5
....B.N...>.R...zX.....~.s..J..............0..r.6.5....?L>...........V4&...Z.,uWWWNVl.l7:.0..}Jv....:....=...H..>...2p.z..(..W.:......D..X.I'.a../H....(9.o..6[..G..9..p.kq._.
....&..$C.r...2y:!.d].....`....Y.|.D.~d..........S.K.....>}.....V.m..%...<..-wE....'
?.#1...1NZ.p.........'=.{....Ic^......4...E5.....lWf.x}.n.JT...{..iU-...y...5[.+a...2...J.....?I..//....\*e4a.i...I/9.^....NR.......i.M...i3}:...
##
T 10.102.38.165:80 -> 10.102.38.185:42920 [AP]
.?A...H....Q....?z.'.....6....L..]9......)...a_.:]B.X......K..RO.eQ......7.ko.$R.C..P.8....,.$..... at .[.E$L...q.e.%8...cY.....c#.
.K..~!'....Y.-.v....}^...t.7.N1.t;.x.vq..(^..=..E|..../..R.;.......l<.....0/.+t.....q_.?}....J../.56.A..`...Ad..2g..........0.9.2...=..&..X......4..1....u.......QhqIQ='.. at 7z..0....+..
###
T 10.102.38.185:80 -> 24.227.153.188:54091 [A]
HTTP/1.1 404 Not Found.
Server: nginx/0.7.65.
Date: Thu, 20 Jan 2011 01:14:02 GMT.
Content-Type: text/css.
Transfer-Encoding: chunked.
Connection: keep-alive.
Last-Modified: Wed, 19 Jan 2011 22:20:44 GMT.
Expires: Sun, 17 Jan 2021 01:14:02 GMT.
Cache-Control: max-age=315360000.
Content-Encoding: gzip.
.
b86.
...........ZYo.H.~..E#~.n`..iIF..$.......b.. at h.-.0E
$.c.........{...
............?............}~.~.....`..\.y......?.._|?Z.N.z.>...$.=....I...%.....U.....U..D.:a5_..~.nE.....e...V...V.L....rs...YQ.....d6...D.Z......Z...D.E.....y..
..........)e.....7....]yD.tV[^.......1*J.=?#6{q&`.d.G..s...^_...2+x]-2..q.~...t.K.....4!.vbQ.u&`.1I.;.%.../...E......0I.m......o.....4..]u...!....+.....<{..$.Ws6.Av.Q<....UE.&......J)M\..?y7._..#k.m]. at ..H.:.P.G.j...X_Q..+.v+.....10Z..-.y.......xy}./..+.........Eo.-.<....Q..P....\...&..
....."......$.9<.I.O.H?......?.)..Ed..Ncp.....].U.E...47..:.t......r~.....x..i....X..h...E.]......d..gh3uLo:....](...(".c.1..:y.=T...i'...3iW.E.Q(.XB.....PC..o.W..b..t..[....6\Q.io....H...D.......Q../..M
.5k.p at S.}.o<..[.h!k.$.Q4.'...wiR.I....C....2....Z....L..a.......Fs..B...hHo8....c.d....HxK9...'{...%}....h..e.z.>.................?._...^.,(.c
..X2..t.....P.Q.s).............N]..$.1.?.. at ....,s....Z....1X...M...pF..d.,......a...y.....l.P...R.'(h.
......Ek........N..G...O$..#...k..L.t...4=..E^1r.&..SK).....[].X..*j(.U..V.V1H..gQ..u.Nk..m.7&...L.M.....|K.H....,8..I8...J.....y.F....0..T.4.
x.,!....b.O..&.....q.9.f.u...E..<......8...8.A......B&@.&../...........].....OX.~.+nR.z,n..*.....{
....{S......]L........`.......LoG..[3..3.W|.f...t]......}/..X..5.P...Q.w.b.P..Ix.e(.h.T.0.......y.....l[..J...a2..>.R.{%e
.,.2.. at ....ad....[..5.../p...;..Uu.....Y....t4.D._.K..'.U....R.z.*..+...Kp.....L"......"...tb...0.uB..p$+. at D5.{..+Uk..*....M.?[4..D....S.^~x..a
..J.F+.i'...=L2._.........X...._.....2..]..k...o(.lp....\....\)...*....T.......L.O..z...Z9.......RP/I.&......(...q.7[.Pb!..7.>...<./..'.P....c"...A..c
.sf8..........d.MA.;`....H..6.ut2K.. at ....g7....P..6.3p..Z..."....l....}V60.x.y.#...)y.P..X...C}.t...9....7..b..)9.T.....m...^7.%..2!~L..1R....%...:uBt.3.... at ..B.....&.d....r:.....`.A..'..B.y.......|........F...b....n. at .../.y.O.$v.)....u*B..P......f..x8nD.6...T..z.L.{.j...|..#...b......M.u...|)td........G...B.$.wp>6 at q6.8.0.P&j..].l.w..A....+....%N......h.... at .x.C.%i.....:Y].._.l......./,..Q..Lj....|+)...........7.-R.....(..].C......D$..pq.c at .K..........
..2..k....$e.P.... .v..r....U.sx...]...$$$
.4.&.........G.....-...o1...G<....[...~i.....xp.ar.....n...|....W......B+...z.+c5
....B.N...>.R...zX.....~.s..J..............0..r.6.5....?L>...........V4&...Z.,uWWWNVl.l7:.0..}Jv....:....=...H..>...2p.z..(..W.:......D..X.I'.a../H....(9.o..6[..G..9..p.kq._.
....&..$C.r...2y:!.d].....`....Y.|.D.~d..........S.K.....>}.....V
#
T 10.102.38.185:80 -> 24.227.153.188:54091 [AP]
.m..%...<..-wE....'
?.#1...1NZ.p.........'=.{....Ic^......4...E5.....lWf.x}.n.JT...{..iU-...y...5[.+a...2...J.....?I..//....\*e4a.i...I/9.^....NR.......i.M...i3}:....?A...H....Q....?z.'.....6....L..]9......)...a_.:]B.X......K..RO.eQ......7.ko.$R.C..P.8....,.$..... at .[.E$L...q.e.%8...cY.....c#.
.K..~!'....Y.-.v....}^...t.7.N1.t;.x.vq..(^..=..E|..../..R.;.......l<.....0/.+t.....q_.?}....J../.56.A..`...Ad..2g..........0.9.2...=..&..X......4..1....u.......QhqIQ='.. at 7z..0....+...
####
T 10.102.38.185:80 -> 24.227.153.188:54091 [AP]
0.
.
######################################################
nginx.conf:
user involver;
worker_processes 4;
pid /var/run/nginx.pid;
worker_rlimit_nofile 10240;
events {
worker_connections 8192;
use epoll;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$http_referer"
'
'"$http_user_agent" "$http_x_forwarded_for"';
# main access log
access_log /data/nginx/log/nginx_access.log main;
# main error log
error_log /data/nginx/log/nginx_error.log debug;
sendfile on;
tcp_nopush on;
include /etc/nginx/apps/*.conf;
}
##########################################
apps/involver-cache-server.conf (test5.involver.com is the app server):
upstream switching_backend {
server localhost:81 weight=100;
server test5.involver.com;
}
upstream rails_backend {
server test5.involver.com;
}
upstream cache_backend {
server localhost:8190;
server localhost:8191;
server localhost:8192;
server localhost:8193;
}
server {
listen 80 default;
server_name _;
root /data/apps/involver-cache-server/current/public;
# Fail fast with a 404 for server pages that rails backend will never
serve
location ~* \.(php|jsp|asp|aspx|cfm|pl|py|tpl)$ { return 404; }
include /etc/nginx/common/app.conf;
include /etc/nginx/common/proxy.conf;
location = /file_uploads {
client_max_body_size 200m;
proxy_pass http://rails_backend;
break;
}
location ~* /rest/features/(.*)/content_items/(.*)/images {
proxy_pass http://rails_backend;
break;
}
location /assets/audio_uploads { return 404; }
location /assets {
proxy_pass http://rails_backend;
break;
}
location /cache {
proxy_pass http://rails_backend;
break;
}
location / {
proxy_next_upstream http_404 http_500 http_502 http_503 http_504
error timeout;
proxy_pass http://switching_backend;
}
}
# OFFLOADED SSL
server {
listen 88;
server_name _;
root /data/apps/involver-cache-server/current/public;
# Fail fast with a 404 for server pages that rails backend will never
serve
location ~* \.(php|jsp|asp|aspx|cfm|pl|py|tpl)$ { return 404; }
include /etc/nginx/common/app.conf;
include /etc/nginx/common/proxy.conf;
proxy_set_header X-FORWARDED_PROTO https;
location ~* /rest/features/(.*)/content_items/(.*)/images {
proxy_pass http://rails_backend;
break;
}
location /assets/audio_uploads { return 404; }
location /assets {
proxy_pass http://rails_backend;
break;
}
location /cache {
proxy_pass http://rails_backend;
break;
}
location / {
proxy_next_upstream http_404 http_500 http_502 http_503 http_504
error timeout;
proxy_pass http://switching_backend;
}
}
# CACHING HANDLER
server {
listen 81;
server_name _;
root /data/apps/involver-cache-server/current/public;
include /etc/nginx/common/proxy.conf;
location / {
proxy_pass http://cache_backend;
}
}
#############################################
common/app.conf:
# Max size for file uploads
client_max_body_size 100M;
# This allows people to use images and css in their maintenance.html
file
if ($request_filename ~* \.(css|jpg|gif|png)$) {
break;
}
# Rewrite all the requests to the maintenance.html page if it exists.
if (-f $document_root/system/maintenance.html) {
return 503;
}
error_page 404 /404.html;
error_page 500 502 504 /500.html;
error_page 503 @503;
location @503 {
rewrite ^(.*)$ /system/maintenance.html break;
}
##################################################
common/proxy.conf:
index index.html index.htm;
# needed to forward user's IP address to rails
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_max_temp_file_size 0;
server_name_in_redirect off;
# set Expire header on assets: see
http://developer.yahoo.com/performance/rules.html#expires
location ~ ^/(images|javascripts|stylesheets)/ {
expires 10y;
}
# serve any existing file
if (-f $request_filename) {
break;
}
# serve any standard Rails page cache file with .html extension
if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}
Posted at Nginx Forum: http://forum.nginx.org/read.php?2,167927,167927#msg-167927
More information about the nginx
mailing list