Question: http video streaming and nginx ram usage

[Francis Daly]

On Tue, Mar 08, 2011 at 12:15:23AM -0500, birdy wrote:

Hi there,

The short answer is: I don't know the answer to your question.

But read on anyway...

> My Nginx 0.7.64 runs on a small embedded linux device under OpenWRT -
> the device has only 13 MB of
> free ram when nginx is loaded. The user's http requests get forwarded to
> nginx on the OpenWRT box, and nginx
> reverse proxies the requests on to the video server over LAN.
> 
> The job of nginx is to prevent people on the Internet from changing the
> settings of the video server via the http interface - only viewing of
> the streams should be allowed! Therefore, nginx acts as a reverse-proxy
> and forwards only the allowed URL on to the server. So the viewers of
> the stream cannot change the video server's settings via the web
> interface, because my nginx config does not forward the "banned" URLs.

Note that location{}s do not match ?query_strings. So some of your
location blocks are not used at all. If you enable the debug log, you'll
see exactly which location is used for each request.

You may be allowing through more URLs than you
think. /Simple/home.htm?DO_EVIL, for example, will be proxied.

> This video server transfers the images to the viewing person's web
> browser in one long http response in real-time.
> 
> As I wrote, In my setup, I reverse-proxy only the allowed URLs (i.e. the
> ones to "get the video") on to the video server with nginx. This works
> fine, but nginx seems to buffer the http data that it forwards locally.

According to the notes at
http://wiki.nginx.org/HttpProxyModule#proxy_buffering that shouldn't
happen. So if it does, it's a bug. (Or a documentation/version error.)

Can you get logs, from nginx or the system, showing the memory use? That
might allow you...

> So while viewing the video stream, nginx consumes more and more RAM
> until the little linux device crashes. 

...to kill and restart nginx before the system falls over. (Or, if it
shows something other than nginx eating the ram, then you'll know what
else to fix.)

> However, if I stop viewing the stream after 1 or 2 minutes, before RAM
> runs out, the memory gets freed again.
> I believe it frees the ram because the request is finished.

> Has anyone got a hint for me what I could do to stop nginx from keeping
> the long request in RAM (this is what I think what happens, at least)?
> 
> Any help would be greatly appreciated!

As you already have,

   proxy_buffering off;

should be enough. So there's something else going on.

I'd suggest to try simplifying the nginx config: get rid of

location /GetStatus.cgi?JsVar=sStatus
location /Simple/home.htm?IMG
location /Simple/home.htm?ATV

since they are (almost certainly) never matched (unless you actually
make a request like "/Simple/home.htm%3fATV").

Then take all of the repeated identical proxy_* directives and put them
in the server block, outside all location{}s. And then remove them from
the individual location{} blocks, leaving just proxy_pass there.

That should shrink the file size quite a bit, which might make it easier
for someone else to analyse the problem.

Good luck with it,

	f
-- 
Francis Daly        francis at daoine.org