Issue with NGINX as reverse proxy for grpc service

[Víctor Enríquez]

Hi,

So we have a service exposing a grpc interface under a certain location
and we are using nginx in front of it. The config looks like the
following:

upstream grpcservers {
  server fqdn:port;
  server fqdn:port;
}

...

server {
  listen port ssl http2;
  client_max_body_size 15m;
  server_name fqdn;

  ssl_certificate /etc/certs/server.crt;
  ssl_certificate_key /etc/certs/server.key;

  location /my.location. {
    grpc_set_header X-Ip-Address $remote_addr;
    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    grpc_ssl_certificate /etc/ssl/mtls-client.crt;
    grpc_ssl_certificate_key /etc/ssl/mtls-client.key;
    grpc_pass grpcs://grpcservers;
    ...
  }

  # Error responses
  include conf.d/errors.grpc_conf; # gRPC-compliant error responses
  default_type application/grpc;   # Ensure gRPC for all error
responses

} //End of the server directive

Now we just realized that each time we do a GET / to that specific port
under that specific location using curl --http2, the request is
forwarded to the backend in such a way that it makes nginx believe that
the backend has crashed, allowing anyone to DDoS this particular
service by just repeteadly sending GET / request to the endpoint.

I am seeing the following messages in the logs:

020/08/07 13:02:37 [error] 1100#1100: *199 upstream rejected request
with error 2 while reading response header from upstream, client:
X.X.X.X, server: fqdn1, request: "POST /my.location.magic.API/GetMagic
HTTP/2.0", upstream: "grpcs://Z.Z.Z.Z:PORT", host: "fqdn1:PORT"

Eventually after the 2 upstream servers are marked as failed we get the
following message:

020/08/07 11:07:05 [error] 1100#1100: *96 no live upstreams while
connecting to upstream, client: X.X.X.X, server: fqdn1, request: "POST
/my.location.magic.API/GetMagic HTTP/2.0", upstream:
"grpcs://grpcservers", host: "fqdn1:PORT"

until the servers are marked as valid again. And the cycle repeats.

I am not an expert on HTTP/2 or gRPC, but it seems like nginx is unable
to negotiate the connection with the backend for those particular
requests created with curl and ends marking the backend as failed when
in fact, it is not failing. Any ideas about how can I further debug
this issue?

Thanks in advance.