nginx on high load (upstream prematurely closed connection while reading response header from upstream)

[muzi]

Dear Guys,

I am facing strange issue, during load test and peak load of more than 3k
concurrent users, get below errors in nginx logs continuously.

2016/05/18 11:23:28 [error] 15510#0: *6853 upstream prematurely closed
connection while reading response header from upstream, client: x.x.x.x,
server:abc.com, request: "GET / HTTP/1.1", upstream:
"fastcgi://127.0.0.1:9000", host: "10.50.x.x"

And beside of above error frequently, below error come gradually

2016/05/18 11:23:28 [error] 15510#0: *7431 recv() failed (104: Connection
reset by peer) while reading response header from upstream, client: x.x.x.x,
server: abc.com, request: "POST / HTTP/1.1", upstream:
"fastcgi://127.0.0.1:9000", host: "10.50.x.x", referrer: "http://10.50.x.x"

But with out peak load, no error is coming and during load besides of above
errors continues coming site is working fine and i see 200 response code in
access logs, but still wondering why above errors coming only in peak load
or during load testing. Below is the require info. I already tune up the
sysctl settings for high throughput

nginx version: nginx/1.8.1
php-fpm /php version : 5.4.45 

echo 0 > /proc/sys/net/ipv4/tcp_ecn
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
echo 20 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 102400 > /proc/sys/net/ipv4/tcp_max_syn_backlog
echo 102400 > /proc/sys/net/core/somaxconn
echo 102400 > /proc/sys/net/core/netdev_max_backlog
echo 5242888 > /proc/sys/net/netfilter/nf_conntrack_max
echo 400000 > /proc/sys/net/ipv4/tcp_max_tw_buckets
echo 655300 > /proc/sys/vm/max_map_count
echo 11000 65000 > /proc/sys/net/ipv4/ip_local_port_range

Kindly please help & advise.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,266978,266978#msg-266978