Nginx spending high CPU in perf tests (vs HAProxy)

[shadyabhi]

Two scenarios:- 

* When upstream supports higher aggregate QPS than what Nginx can support on
a host, nginx has better CPU performance than HAProxy. (Nginx == 700% CPU,
HAProxy = 1200% CPU)

* When upstream capacity is limited, and my benchmarking tool sends QPS
higher than what the upstream supports, nginx is showing unbelievably high
CPU usage when compared to HAProxy. (Nginx == 400% CPU, HAProxy = 60% CPU)

Hence, there's something about Nginx where it shines as the throughput
increases but at low throughput, it has higher CPU usage than HAProxy. Any
pointers on what behavior is causing this? My guess is, Nginx is enqueuing a
lot more TCP streams than the upstream can handle, which ultimately causes
higher CPU than HAProxy. 

NGINX CONFIG (v1.15)
```
user root;
worker_processes auto;
daemon off;                                                                 
                                                                            
                                                                            
     error_log ./error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}
                                                                            
                                                                            
                                                                            
     stream {                                                               
                                                                            
                                                                            
              server_traffic_status_zone;
    log_format basic '$proxy_protocol_addr - $remote_user [$time_local] '   
                                                                            
                                                                            
            
    access_log ./access.log basic;

    upstream xproxy {
        server 10.100.10.1:12270;
    }
    server {
        listen              80;

        proxy_pass          xproxy;
        proxy_protocol      on;
    }
}
```

Thank you.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288935,288935#msg-288935