Sharing data when download the same object from upstream

[Alex Garzão]

Hello Wandenberg,

Thanks for your reply.

Using proxy_cache_lock, when the second request arrive, it will wait
until the object is complete in the cache (or until
proxy_cache_lock_timeout expires). But, in many cases, my upstream has
a really slow link and NGINX needs more than 30 minutes to download
the object. In practice, probably I will see a lote of parallel
downloads from the same object.

Someone has other idea? Or I'm wrong about proxy_cache_lock?

Regards.
--
Alex Garzão
Projetista de Software
Azion Technologies
alex.garzao (at) azion.com


On Mon, Aug 26, 2013 at 6:58 PM, Wandenberg Peixoto
<wandenberg at gmail.com> wrote:
> Try to use the proxy_cache_lock configuration, I think this is what you are
> looking for.
> Don't forget to configure the proxy_cache_lock_timeout to your use case.
>
> On Aug 26, 2013 6:54 PM, "Alex Garzão" <alex.garzao at azion.com> wrote:
>>
>> Hello guys,
>>
>> This is my first post to nginx-devel.
>>
>> First of all, I would like to congratulate NGINX developers. NGINX is
>> an amazing project :-)
>>
>> Well, I'm using NGINX as a proxy server, with cache enabled. I noted
>> that, when two (or more) users trying to download the same object, in
>> parallel, and the object isn't in the cache, NGINX download them from
>> the upstream. In this case, NGINX creates one connection to upstream
>> (per request) and download them to temp files. Ok, this works, but, in
>> some situations, in one server, we saw more than 70 parallel downloads
>> to the same object (in this case, an object with more than 200 MB).
>>
>> If possible, I would like some insights about how can I avoid this
>> situation. I looked to see if it's just a configuration, but I didn't
>> find nothing.
>>
>> IMHO, I think the best approach is share the temp file. If possible, I
>> would like to known your opinions about this approach.
>>
>> I looked at the code in ngx_http_upstream.c and ngx_http_proxy.c, and
>> I'm trying to fix the code to share the temp. I think that I need to
>> do the following tasks:
>>
>> 1) Register the current downloads from upstreams. Probably I can
>> address this with a rbtree, where each node has the unique object id
>> and a list with downstreams (requests?) waiting for data from the
>> temp.
>>
>> 2) Disassociate the read from upstream from the write to downstream.
>> Today, in the ngx_event_pipe function, NGINX reads from upstream,
>> writes to temp, and writes to downstream. But, as I can have N
>> downstreams waiting data from the same upstream, probably I need to
>> move the write to downstream to another place. The only way I think is
>> implementing a polling event, but I know that this is incorrect
>> because NGINX is event based, and polling waste a lote of CPU.
>>
>> 3) When I know that there more data in temp to be sent, which function
>> I must use? ngx_http_output_filter?
>>
>> Suggestions will welcome :-)
>>
>> Thanks people!
>>
>> --
>> Alex Garzão
>> Projetista de Software
>> Azion Technologies
>> alex.garzao (at) azion.com
>>
>> _______________________________________________
>> nginx-devel mailing list
>> nginx-devel at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel