403 forbidden with lynx www browser

[Francis Daly]

On Mon, Nov 23, 2015 at 10:23:20AM +1100, Sylvain BERTRAND wrote:
> On Sun, Nov 22, 2015 at 05:25:11PM +0000, Francis Daly wrote:

Hi there,

> > Probably they have configured their servers to deny any request that
> > includes "libwww" in the User-Agent.
> 
> I'm about to block sending the user agent for good and all www sites.
> (or I will send a custom one).

Possibly something like "lynx, but with a non-default user agent string"
will get their attention. (Probably not, though.)

> > But unless the nginx.org (or possibly nginx.com) web sites are among
> > the sites that block you, there's probably not much that readers of this
> > list can do about it.
> 
> That's why I'm posting here: *Only nginx* www sites does block lynx. Something
> is not right there: a default aggressive blocking policy from nginx?

There is no default policy in stock nginx, or in any distributed nginx
that I'm aware of.

My suspicion is that these sites that run nginx also run some back-end
framework, and that that framework may have some suggested configuration
for nginx which blocks access based on user-agent matching.

If there is any hint in the response headers what the full server-side
suite is, that may be a good place to start looking for a more general
future solution.

Cheers,

	f
-- 
Francis Daly        francis at daoine.org