Slow Time to first byte

[miralize]

Currently running into a host of problems with a collections of sites that
I'm running off a Digital Ocean LEMP stack. There is a > 1s TTFB delay &
things are in general are sluggish

Ubuntu 12.04
4 Core CPU
8GB RAM

3 sites on the one server, but only one gets about 30k hits a day. Very
image heavy, but the images are hosted on S3.

A gist of my nginx, site-enabled conf & www.conf (php5-fpm config)

https://gist.github.com/miralize/b026c1f7ae824396f9af

Running nginx/1.1.19 & Php 5.5 with opcache enabled today (one of the
billion things that I've tried to resolve this by)

Any help would be greatly appreciated and anything other info that might
help solve it, just ask!

Memory limit needed to be increased to get through an image transform task.
Is it too high at 2048MB?

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,254876,254876#msg-254876