Issue with NGINX as reverse proxy for grpc service

[Víctor Enríquez]

Hi Sergey,

First thanks for your reply. 

What I don't really understand is, shouldn't nginx be more strict by
default with the requests that are passed to grpc backends? I have been
reading a little bit about the GRPC protocol, and it's supposed to just
use POST requests (I might be wrong about this though). Shouldn't any
other kind of request be filtered by nginx directly (at least if it is
obvious that they are malformed)?

I am going to try to reproduce this with a docker compose file and the
standard hello grpc example, to see if it's a failure on our backend
implementation or if I can reproduce it with other grpc backends.

Other than this issue, the backend is working fine if the request is
not created with curl (the clients that are supposed to use the backend
are working fine), so I guess there is some checking on the golang's
grpc implementation to check if the request is valid, and it returns
that value RST_STREAM(INTERNAL_ERROR) (but I think we don't have
control over it AFAIK, again I might be wrong). 

To me ,without being an expert, it sounds like there is something
wrong, either on the nginx side or on the grpc implementation side of
our backend, but we are not returning that error code directly.
(Perhaps the golang's grpc implementation should return other error
code that doesn't make nginx believe that the upstream is dead when it
receives a malformed request).

Again thanks for your help.

On Fri, 2020-08-07 at 19:28 +0300, Sergey Kandaurov wrote:
> > On 7 Aug 2020, at 17:18, Víctor Enríquez <victor at bitonic.nl> wrote:
> > 
> > Hi,
> > 
> > So we have a service exposing a grpc interface under a certain
> > location
> > and we are using nginx in front of it. The config looks like the
> > following:
> > 
> > upstream grpcservers {
> >  server fqdn:port;
> >  server fqdn:port;
> > }
> > 
> > ...
> > 
> > server {
> >  listen port ssl http2;
> >  client_max_body_size 15m;
> >  server_name fqdn;
> > 
> >  ssl_certificate /etc/certs/server.crt;
> >  ssl_certificate_key /etc/certs/server.key;
> > 
> >  location /my.location. {
> >    grpc_set_header X-Ip-Address $remote_addr;
> >    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> >    grpc_ssl_certificate /etc/ssl/mtls-client.crt;
> >    grpc_ssl_certificate_key /etc/ssl/mtls-client.key;
> >    grpc_pass grpcs://grpcservers;
> >    ...
> >  }
> > 
> >  # Error responses
> >  include conf.d/errors.grpc_conf; # gRPC-compliant error responses
> >  default_type application/grpc;   # Ensure gRPC for all error
> > responses
> > 
> > } //End of the server directive
> > 
> > Now we just realized that each time we do a GET / to that specific
> > port
> > under that specific location using curl --http2, the request is
> > forwarded to the backend in such a way that it makes nginx believe
> > that
> > the backend has crashed, allowing anyone to DDoS this particular
> > service by just repeteadly sending GET / request to the endpoint.
> > 
> > I am seeing the following messages in the logs:
> > 
> > 020/08/07 13:02:37 [error] 1100#1100: *199 upstream rejected
> > request
> > with error 2 while reading response header from upstream, client:
> > X.X.X.X, server: fqdn1, request: "POST
> > /my.location.magic.API/GetMagic
> > HTTP/2.0", upstream: "grpcs://Z.Z.Z.Z:PORT", host: "fqdn1:PORT"
> 
> "error 2" means that backend responded with
> RST_STREAM(INTERNAL_ERROR),
> that is, effectively rejected processing request.
> You may want to consult with backend error log to find out the
> reason.
>