Clean-URL rewrite rule with nested "location" and alias directive

[Ben Johnson]

On 11/23/2013 3:47 PM, Francis Daly wrote:
> On Sat, Nov 23, 2013 at 01:11:46PM -0500, Ben Johnson wrote:
>> On 11/23/2013 12:36 PM, Ben Johnson wrote:
> 
> Hi there,
> 
>>> It's bizarre. At some point while meddling with the configuration,
>>> requests for /stage/ began causing the browser to download index.php
>>> (and I can open the file and see the PHP code). And no matter what I
>>> change in the configuration, this behavior persists.
> 
> I tend to test with "curl", because it avoids any chance of a browser
> cache being involved.
> 

That's very good advice, and yielded a somewhat surprising result when I
added your test string (from your previous reply, down below) to my
configuration:

location = /stage/ { return 200 "Just checking...\n"; }

When I request the /stage/ URL via curl, *from* the web-server on which
this problem is occurring, I see our test string, "Just checking...". I
am using the FQDN when I perform the curl test.

And, when I hit the URL with curl from my local workstation, I see "Just
checking..."

Yet, when I hit the same URL in a web-browser (Chrome in Incognito mode,
Firefox, doesn't seem to matter), from my local workstation, I am
presented with a binary file download for this stale index.php that is
coming from who-knows-where. The MIME-type registers as
application/octet-stream.

I tried hitting the URL in a browser on another machine that is in
another physical location and the result is the same.

When I ping the FQDN from my home workstation, the IP address is as I
expect.

> Any intermediate proxies or caches might still be in the way, of
> course. But the nginx access log and/or debug log should show what nginx
> thinks is happening.
> 

The last time a DNS change was made for this domain was several weeks
ago. But because it may be relevant, that DNS change did directly affect
web services (the change was made as part of a server migration -- this
problem is on the new server).

>>> I went so far as to rename the "stage" directory to "staging", and
>>> changed all references in nginx's configuration accordingly. Yet, for
>>> some reason, requests for /stage/ still download the index.php file! I'm
>>> not even sure where this index.php file is coming from, given that nginx
>>> should have absolutely no knowledge of this file's new/current location.
> 
> The usual reason for this kind of thing is that the nginx.conf that you
> are writing and the nginx.conf that nginx is reading are not the same.
> 

I tried intentionally throwing bad syntax into the nginx config file for
this vhost and restarting nginx, and indeed nginx complains about the
problem and won't restart, which is expected.

This seems to confirm that the configuration file that I am editing is
indeed effective.

>>> The weirdest part is that the downloaded file is *not* the file that
>>> exists at /index.php (the "main site" index file). I confirmed this by
>>> adding some commented PHP code at the bottom of /index.php, and the
>>> comments do not appear in the downloaded file.
> 
> The other likely reason is that the web server you are talking to and
> the web server you think you are talking to are not the same.
> 
> (Either a different machine entirely, or else a different server{}
> block in the same config file.)
> 

I would be surprised if I am still hitting the "old" server, away from
which I migrated over three weeks ago. Every machine from which I "ping"
the FQDN returns the expected IP address (including the machine on which
this is happening).

>>> Hell, I even tried deleting the entire "staging" directory and this
>>> still happens! I've restarted nginx, php5-fpm, etc. and nothing changes.
> 
> Possibly you did a "nginx -s reload", but an error in the conf file
> meant that nginx didn't stop using the older file, or something like that?
> 

I'm using my OS-specific command, "service nginx reload" (which uses
/etc/init.d/nginx) to test configuration changes. Restarting with
"service nginx restart" doesn't seem to make any difference.
Historically, "reload" has always been sufficient to render my changes
effective.

No errors appear in the log, unless I intentionally insert invalid
syntax into the configuration.

>> I removed everything related to this effort (setting-up the staging
>> site) from nginx's configuration, and deleted all of the staging site
>> files from the filesystem, yet nginx is still serving an "index.php"
>> file whenever I request /stage/! How is this possible? This "index.php"
>> is indeed "mine"; it contains the code from my application. But, as I
>> said, if I add some random, commented PHP code to the bottom of the
>> "real index file" at /index.php, it's not present in the downloaded
>> file. I just don't see where this file could be coming from at this point.
> 
> Add something like
> 
>   location = /stage/ { return 200 "Just checking...\n"; }
> 
> and restart nginx. If you don't see that response for that request,
> you're not using that conf file.
> 

When I hit the URL with curl, the request is indeed logged to the
vhost's access log:

"GET /stage/ HTTP/1.1" 200 17 "-" "curl/7.22.0 (i686-pc-linux-gnu)
libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"

When I hit the URL with the browser, the request is logged, too:

"GET /stage/ HTTP/1.1" 200 17 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36"

Given that the requests look identical, excepting the user-agent, I'm
left wondering why the browser downloads some stale index.php, yet curl
displays our test message.

Again, I've tried hitting the URL from different computers in different
locations, using different browsers, and the behavior is consistent.
This implies that the problem is with the server, not the user-agents
with which I'm testing.

> If that much does work, then either read the debug log, or trace through
> nginx.conf to see what should happen for this request -- server-level
> rewrite module directives, plus the one location that the request
> "/stage/" should be handled in, are the most likely sources.
> 

So, I just went to comment-out the test location and see what curl
returns in that scenario, given that a browser is clearly unreliable,
and now we're back to normal! All I did was comment-out the test line in
the vhost's config file and "service nginx reload"!

>> Thanks for any insight as to what could be happening here...
> 
> Make sure nginx is really completely stopped.
> 
> Then start it again with a known config file.
> 
> And look at the responses to a few requests, and see which are not what
> you expect.
> 
> I'm afraid I'm reduced to generalities with the information available.
> 
> Good luck with it,
> 
> 	f
> 

In summary, all I did today was comment-out one line of the config and
reload nginx, and now all is back to normal. Needless to say, I had
tried all of this and much more yesterday, and nothing I tried had any
effect.

I see no explanation for this behavior other than nginx caching its
configuration in some fashion. Maybe after 20-something hours and a
reload, it "let go" of whatever cached version it was using.

If this ever happens again, what is the first avenue to explore? There
has to be a better way to troubleshoot this.

Maybe one of the developers can comment as to how nginx stores its
configuration while running, and how to prevent this issue going forward
(or at least fix it whenever it happens).

Thanks again for your excellent suggestions and sound troubleshooting
logic, Francis.

Now, back to the original problem... :P

-Ben