Prevent direct access to files but allow download from site

lists lists at
Thu Mar 12 00:45:02 UTC 2020

You could make it harder to pass around the URL if it is dynamic. That is make the url session related. 

You can do a search on "uncrawlable" and then exactly the opposite of what they suggest. That is most people want to be crawled, so their advice is backwards. 

One thing to watch out for is Google dorking. Even if Google doesn't crawl, the html/javascript/etc leave clues to file locations. 

When I am on a server I will copy what is needed to block wget, curl, nutch, screamingfrog, etc. I have a number of Nginx maps I use to block and flag troublemakers. 

  Original Message  

From: abbot at
Sent: March 11, 2020 3:57 PM
To: nginx at
Reply-to: nginx at
Subject: Re: Prevent direct access to files but allow download from site

* MAXMAXarena:

> I want to be able to download a file from the site's html tag [...]
> But do not allow direct access and download, using the browser or
> other tools such as curl or wget.

Public access and restricted access are mutually exclusive. It also
makes nearly no difference what utility is used to access a URL pointing
to the text file you gave as an example, because they'll all send a HTTP
GET request, which is what the web server expects. Attempting to limit
access based on a User-Agent header or similar, to identify the client
type, is easily circumvented.

nginx mailing list
nginx at

More information about the nginx mailing list