Allow internal redirect to URI x, but deny external request for x?

[J. Lewis Muir]

On 09/04, Jürgen Wagner (DVT) wrote:
> This is the effect you get by having the HTTP equivalent of a symbolic link
> in the NGINX (visible to the browser), not in the file system (which is
> opaque to users). The file system link will (over time) serve different
> contents under the same URL, so in fact, addressing changes with every
> deployment. The suggested approach keeps URL addressing constant and just
> changes the entry pointer on a new deployment.
> 
> I agree that this is not the solution that first comes to ones mind, but it
> does solve a number of nasty versioning issues we have run into over time.
> Your mileage may vary :-)

Thank you for the further explanation!  Indeed it seems like a
compelling solution!

What about web search engine indexing; do you do anything to avoid
search engines indexing the versioned URLs?  I suppose that if you only
publish the unversioned entry-point URLs, search engines will respect
that?  (Maybe wishful thinking.)  Or will they follow a 307 redirect and
index those URLs?

For example, it would seem undesirable to do a web search for "my-app"
and get a list of, say, the "index.php" for each version (e.g.,
"/my-app/releases/1.0.0/index.php", "/my-app/releases/1.0.1/index.php",
"/my-app/releases/1.0.2/index.php", etc.).

So, perhaps you use a "/robots.txt" to exclude "/my-app/releases/"?

DuckDuckGo seems to respect "/robots.txt" for controlling what gets indexed

  https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/

But Google says "/robots.txt" is not for keeping a web page out of their
index

  https://support.google.com/webmasters/answer/6062608

and that you should use a "noindex" directive instead.

So maybe you use both a "/robots.txt" and the robots meta tag with
content="noindex" in the served resources or perhaps "X-Robots-Tag:
noindex" in the HTTP header response?

Regards,

Lewis