Issue with NGINX as reverse proxy for grpc service

[Sergey Kandaurov]

> On 7 Aug 2020, at 17:18, Víctor Enríquez <victor at bitonic.nl> wrote:
> 
> Hi,
> 
> So we have a service exposing a grpc interface under a certain location
> and we are using nginx in front of it. The config looks like the
> following:
> 
> upstream grpcservers {
>  server fqdn:port;
>  server fqdn:port;
> }
> 
> ...
> 
> server {
>  listen port ssl http2;
>  client_max_body_size 15m;
>  server_name fqdn;
> 
>  ssl_certificate /etc/certs/server.crt;
>  ssl_certificate_key /etc/certs/server.key;
> 
>  location /my.location. {
>    grpc_set_header X-Ip-Address $remote_addr;
>    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>    grpc_ssl_certificate /etc/ssl/mtls-client.crt;
>    grpc_ssl_certificate_key /etc/ssl/mtls-client.key;
>    grpc_pass grpcs://grpcservers;
>    ...
>  }
> 
>  # Error responses
>  include conf.d/errors.grpc_conf; # gRPC-compliant error responses
>  default_type application/grpc;   # Ensure gRPC for all error
> responses
> 
> } //End of the server directive
> 
> Now we just realized that each time we do a GET / to that specific port
> under that specific location using curl --http2, the request is
> forwarded to the backend in such a way that it makes nginx believe that
> the backend has crashed, allowing anyone to DDoS this particular
> service by just repeteadly sending GET / request to the endpoint.
> 
> I am seeing the following messages in the logs:
> 
> 020/08/07 13:02:37 [error] 1100#1100: *199 upstream rejected request
> with error 2 while reading response header from upstream, client:
> X.X.X.X, server: fqdn1, request: "POST /my.location.magic.API/GetMagic
> HTTP/2.0", upstream: "grpcs://Z.Z.Z.Z:PORT", host: "fqdn1:PORT"

"error 2" means that backend responded with RST_STREAM(INTERNAL_ERROR),
that is, effectively rejected processing request.
You may want to consult with backend error log to find out the reason.

-- 
Sergey Kandaurov