Proxy uploading

[Igor Sysoev]

On Thu, Jul 29, 2010 at 11:45:50AM +0200, Tim Child wrote:

> Hi,
> 
> Currently using Nginx 0.7.65-1 on Ubuntu 10.04. I need to have my application upload files to another backend, so I thought I could use proxy pass, and have a certain URL be proxied to another machine (proxy_pass http://upstreamserver.com:8080/API/import) . The URL would be http//portalvm/API/upload/ that I upload to.
> 
> Otherwise proxy to Apache running on 127.0.0.1:8000.
> 
> What is happening is that it is indeed proxying but I am getting a 404 error from the upstreamserver.com even though the URL that I am using looks correct. The logs (error.log in Debug mode) are saying:
> 
> http proxy header:
> "POST /API/import/raw?transferid=unique321&name=p158cr86gf1i34t099c1gkn1vh51.tmp HTTP/1.0^M
> Authorization: Basic base64string^M
> Host: p upstreamserver.com:8080^M

I see strange host name: "p upstreamserver.com:8080".

> Connection: close^M
> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8^M
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8^M
> Accept-Language: en,en-us;q=0.7,sv;q=0.3^M
> Accept-Encoding: gzip,deflate^M
> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7^M
> Content-Type: application/octet-stream^M
> RunAs: username^M
> INDEX: 0^M
> Referer: http://portalvm/my/uploadpage/^M
> Content-Length: 2809716^M
> Cookie: sessionid=09520e8dfd27d9ee86781b928ed20689^M
> Pragma: no-cache^M
> Cache-Control: no-cache^M
> ^M
> "
> 
> 
> Then there are a lot of logs streaming the file to the upstream server such as:
> 
> http upstream request: "/API/import/raw?transferid=unique321&name=p158csbronq6718k215au1ctp1ulo1.tmp"
> 2010/07/28 17:57:39 [debug] 3950#0: *4 http upstream send request handler
> 2010/07/28 17:57:39 [debug] 3950#0: *4 http upstream send request
> 2010/07/28 17:57:39 [debug] 3950#0: *4 read: 9, 0000000000C9B640, 8192, 0
> 2010/07/28 17:57:39 [debug] 3950#0: *4 chain writer buf fl:0 s:725
> 2010/07/28 17:57:39 [debug] 3950#0: *4 chain writer buf fl:0 s:345
> 2010/07/28 17:57:39 [debug] 3950#0: *4 chain writer buf fl:0 s:8192
> 2010/07/28 17:57:39 [debug] 3950#0: *4 chain writer in: 0000000000C9AD88
> 2010/07/28 17:57:39 [debug] 3950#0: *4 writev: 9262
> 2010/07/28 17:57:39 [debug] 3950#0: *4 chain writer out: 0000000000000000
> 
> Then at the end::
> 
> 2010/07/28 17:58:09 [debug] 3950#0: *4 http upstream process header
> 2010/07/28 17:58:09 [debug] 3950#0: *4 malloc: 0000000000CECF20:4096
> 2010/07/28 17:58:09 [debug] 3950#0: *4 recv: fd:15 260 of 4096
> 2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy status 404 "404 Not Found"
> 2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: "X-Powered-By: Servlet/2.5"
> 2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: "Server: Sun GlassFish Enterprise Server v2.1.1"
> 2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: "Content-Type: text/plain"
> 2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: "Content-Length: 57"
> 2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: "Date: Wed, 28 Jul 2010 15:58:09 GMT"
> 2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: "Connection: close"
> 2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header done
> 2010/07/28 17:58:09 [debug] 3950#0: *4 malloc: 0000000000CEDF30:4096
> 2010/07/28 17:58:09 [debug] 3950#0: *4 HTTP/1.1 404 Not Found^M
> Server: nginx/0.7.65^M
> Date: Wed, 28 Jul 2010 15:58:09 GMT^M
> Content-Type: text/plain^M
> Transfer-Encoding: chunked^M
> Connection: keep-alive^M
> X-Powered-By: Servlet/2.5^M
> Content-Encoding: gzip^M
> 
> 
> 
> What I can't understand is the headers look correct, and so does the HTTP upstream request URL. In fact if I try and use the same headers and URL in a util that lets me post to the server it creates an empty file.
> 
> Any idea on why I am getting a 404?
> 
> Thanks,
> 
> Tim.
> 
> 
> 
> 
> In my nginx.conf I have this (base64string - obviously changed):
> 
> http {
>     include /etc/nginx/mime.types;
>     gzip  on;
>     gzip_comp_level 2;
>     gzip_proxied any;
>     gzip_disable msie6;
>     gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;
> 
>     upstream portalvm {
>         server 127.0.0.1:8000;
>     }
> 
>     server {
>         listen 80;
>                 
>         location ~ ^/(favicon.ico|robots.txt|sitemap.xml)$ {
>             alias /opt/media/$1;
>             expires 30d;        
>         }

It's better to write this as three locations:

         location = /favicon.ico {
             root /opt/media;
             expires 30d;        
         }

         location = /robots.txt {
             root /opt/media;
             expires 30d;        
         }

         location = /sitemap.xml {
             root /opt/media;
             expires 30d;        
         }

>         location /sitemedia {
>             alias /opt/media/;
>             expires 30d;            
>         }
>         location /API/import {
>            error_log /var/log/nginx/error.log debug;
>            client_max_body_size 1000M; 
>            proxy_pass http://upstreamserver.com:8080/API/import;

These directives hide/pass header from upstream to client:

>            proxy_hide_header Referer;
>            proxy_hide_header Cookie;
>            proxy_pass_header Content-Length;

>            proxy_set_header Authorization "Basic base64string";
>         }
>         location / {
>             proxy_pass http://portalvm;
>             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;      
>             add_header X-Handled-By $upstream_addr;      
>         }

I do not understand, what you want to proxy:
"/API/upload/" or "/API/import/" ?


-- 
Igor Sysoev
http://sysoev.ru/en/