Question regarding $invalid_referer

[J Carter]

Hello,

On Tue, 5 Mar 2024 13:07:53 -0800
"lists at lazygranch.com" <lists at lazygranch.com> wrote:

> I am presently using a scheme like this to prevent scraping documents. 
> ************************************
>    location /images/ {
>           valid_referers none blocked  www.example.com example.com forums.othersite.com ;
> # you can tell the browser that it can only download content from the domains you explicitly allow
> #           if ($invalid_referer) {
> #             return 403;
>             if ($invalid_referer) {
>               return 302 $scheme://www.example.com;
> ***************************************
> I commented out some old code which just sends an error message. I
> pulled that from the nginx website. I later added the code which sends
> the user to the top level of the website. 
> 
> It works but the results really aren't user friendly. What I rather do
> is if I find an invalid_referer to some document, I would like to
> redirect the request to the html page that has my link to the document. 
> 
> I am relatively sure I will need to hand code the redirection for every
> file, but plan on only doing this for pdfs. Probably 20 files.
> 
> Here is a google referral I pulled from the log file
> 
> *********************************************
> 302 172.0.0.0 - - [05/Mar/2024:20:18:52 +0000] "GET /images/ttr/0701crash.pdf HTTP/2.0" 145 "https://www.google.com/" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Mobile Safari/537.36" "-"
> **********************************************
> So I would need to map /images/ttr/0701crash.pdf to the referring page
> on the website.
> _______________________________________________

There is really a question in your email :) however, you could use the
SSI module[1] to auto generate the referring page with the link
dynamically if don't already have that.

[1] https://nginx.org/en/docs/http/ngx_http_ssi_module.html

In terms of doing the mapping to some static set of referring pages if
you already have those, that will depend upon what path scheme you plan
for those in relation to original files.

A sensible way would be to make
the referring pages's path related to pdf name, (something like
/referring/0701crash).

In nginx when you do redirect, you can do those mappings
dynamically using regex captures. Something like this using nested
locations:

location /images {
        ...
	location ~/(.+).pdf {
		if ($invalid_referer) {
                	return 302 $scheme://www.example.com/referring/${1};
		}
	}
}