I don't think it would matter that much, especially as the reason for load balancing is probably that you are serving not static pages, which usually need some processing time anyway. What would be interesting though is if you could add a max connections per IP address in the upstream server section.