gRPC reverse proxy closed tcp connection after 1000 rpc calls

jetchars nginx-forum at forum.nginx.org
Wed Mar 6 14:20:03 UTC 2019


Hey, genius

I've followed the official user guide to create a gRPC reverse proxy, config
as follow:

```
upstream grpcservers {
    server 10.90.62.50:60080 weight=3;
    server 10.90.62.51:60080 weight=3;
    server 10.90.62.52:60080 weight=3;
    keepalive 2000;
    keepalive_timeout 120;
    keepalive_requests 100000;
}

server {
    listen       8051 http2;
    error_log /home/nginx/log/s4_mongo_error.log;
    access_log /home/nginx/log/s4_mongo_access.log;
    grpc_socket_keepalive on;
    location / {
        grpc_pass grpc://grpcservers;
        error_page 502 = /error502grpc;
    }

    location = /error502grpc {
        internal;
        default_type application/grpc;
        add_header grpc-status 14;
        add_header grpc-message "unavailable";
        return 204;
    }
}
```

- nginx keepalive with upstream servers successfully
- but when keepalive with gRPC client, after 1000 requests been processed,
nginx will close the tcp connection, because I can find `TIME_WAIT` on the
nginx side.
- then gRPC client will report tens of thousands of `TransientFailure` at
same time

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283297,283297#msg-283297



More information about the nginx mailing list