Disable NGINX caching 304 Responses from Origin Server

[Ryan Barclay]

We have a pretty simple setup with NGINX sitting on the front and a 
backend server (on a separate physical server) that provides the content.

Nginx then caches content based on the EXPIRES and Cache-Control headers 
set by the origin server.

We noticed that NGINX was not issuing 304 headers to images that were 
not in the local NGINX cache when the If-Modified-Since header was sent 
by the client. Instead, it would issue a 200 with the full data file.

To fix this, we applied:
proxy_set_header If-Modified-Since $http_if_modified_since

So then the If-Modified-Since header was passed to the backend and of 
course, it returned correctly with the 304 header - great.

But what we noticed was that NGINX would cache this 304 response and 
deliver future responses as 304 to clients even without the 
If-Modified-Since header.

How can we disable caching of 304 responses and fix this issue?

Thank you for your help, suggestions, and tips in advance.