Allow internal redirect to URI x, but deny external request for x?

[j94305]

Robots exclusion is generally quite unreliable. Exclusions based on user
agents are also not really reliable. You can try all of the options for
robots exclusion and may still get undesired crawlers on your site.

The only way you can keep robots out is to require authentication for those
parts you don't want to have crawled.

--j.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285463,285599#msg-285599