403 errors with standard browser, pages work with GUI browser

[Francis Daly]

On Wed, Jul 29, 2015 at 05:00:40PM +0200, Thorsten Glaser wrote:

Hi there,

> I’ve run across several websites where I get a 403 error when
> I connect to them with my standard webbrowser (lynx), yet the
> very same URIs work with the GUI browser fallback (Firef*x).

> All of these sites is common that they have an nginx error
> message, so this must be connected to your software somehow.

That suggests that they run nginx as their web server.

There is unlikely to be any other connection.

> To add insult to injury, this appears to be dependent on
> the User-Agent HTTP header: “Mozilla/5.0 (compatible)”
> works, “Lynx/2.8.9dev.6” works, “Lynx/2.8.9dev.6 libwww-FM/2.14”
> doesn’t, nor “Mozilla/5.0 (compatible) libwww-FM/2.14”.
> 
> It appears that there is some blacklist on some component
> of my webbrowser. Please kindly remove it.

I suspect that they have configured their instance of their web server
to deny access to user agents which include the string "libwww".

I suspect that your best option for accessing the content is to change
your web browser user agent string to omit "libwww" (and whatever other
parts the server owner has chosen to block); at least until you contact
the owner and convince them to change their configuration.

Good luck with it,

	f
-- 
Francis Daly        francis at daoine.org