Sharing data when download the same object from upstream

[Alex Garzão]

Hello Anatoli,

Thanks for your reply. I will appreciate (a lot) your help :-)

I'm trying to fix the code with the following requirements in mind:

1) We were upstreams/downstreams with good (and bad) links; in
general, upstream speed is more than downstream speed but, in some
situations, the downstream speed is a lot more quickly than the
upstream speed;

2) I'm trying to disassociate the upstream speed from the downstream
speed. The first request (request that already will connect in the
upstream) download data to temp file, but no longer sends data to
downstream. I disabled this because, in my understand, if the first
request has a slow downstream, all others downstreams will wait data
to be sent to this slow downstream.

My first doubt is: Need I worry about downstream/upstream speed?

Well, I will try to explain what I did in the code:

1) I created a rbtree (currrent_downloads) that keeps the current
downloads (one rbtree per upstream). Each node keeps the first request
(request that already will connect into upstream) and a list
(download_info_list) that will keep two fields: (a) request waiting
data from the temp file and (b) file offset already sent from the temp
file (last_offset);

2) In ngx_http_upstream_init_request(), when the object isn't in the
cache, before connect into upstream, I check if the object is in
rbtree (current_downloads);

3) When the object isn't in current_downloads, I add a node that
contains the first request (equal to current request) and I add the
current request into the download_info_list. Beyond that, I create a
timer event (polling) that will check all requests in
download_info_list and verify if there are data in temp file that
already not sent to the downstream. I create one timer event per
object [1].

4) When the object is in current_downloads, I add the request into
download_info_list and finalize ngx_http_upstream_init_request() (I
just return without execute ngx_http_upstream_finalize_request());

5) I have disabled (in ngx_event_pipe) the code that sends data to
downstream (requirement 2);

6) In the polling event, I get the current temp file offset
(first_request->upstream->pipe->temp_file->offset) and I check in the
download_info_list if this is > than last_offset. If true, I send more
data to downstream with the ngx_http_upstream_cache_send_partial (code
bellow);

7) In the polling event, when pipe->upstream_done ||
pipe->upstream_eof || pipe->upstream_error, and all data were sent to
downstream, I execute ngx_http_upstream_finalize_request for all
requests;

8) I added a bit flag (first_download_request) in ngx_http_request_t
struct to avoid request to be finished before all requests were
completed. In ngx_http_upstream_finalize_request() I check this flag.
But, in really, I don't have sure if is necessary avoid this
situation...


Bellow you can see the ngx_http_upstream_cache_send_partial code:


/////////////
static ngx_int_t
ngx_http_upstream_cache_send_partial(ngx_http_request_t *r,
ngx_temp_file_t *file, off_t offset, off_t bytes, unsigned last_buf)
{
    ngx_buf_t         *b;
    ngx_chain_t        out;
    ngx_http_cache_t  *c;

    c = r->cache;

    /* we need to allocate all before the header would be sent */

    b = ngx_pcalloc(r->pool, sizeof(ngx_buf_t));
    if (b == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    b->file = ngx_pcalloc(r->pool, sizeof(ngx_file_t));
    if (b->file == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    /* FIX: need to run ngx_http_send_header(r) once... */

    b->file_pos = offset;
    b->file_last = bytes;

    b->in_file = 1;
    b->last_buf = last_buf;
    b->last_in_chain = 1;

    b->file->fd = file->file.fd;
    b->file->name = file->file.name;
    b->file->log = r->connection->log;

    out.buf = b;
    out.next = NULL;

    return ngx_http_output_filter(r, &out);
}
////////////

My second doubt is: Could I just fix ngx_event_pipe to send to all
requests (instead of to send to one request)? And, if true,
ngx_http_output_filter can be used to send a big chunk at first time
(300 MB or more) and little chunks after that?

Thanks in advance for your attention :-)

[1] I know that "polling event" is a bad approach with NGINX, but I
don't know how to fix this. For example, the upstream download can be
very quickly, and is possible that I need send data to downstream in
little chunks. Upstream (in NGINX) is socket event based, but, when
download from upstream finished, which event can I expect?

Regards.
--
Alex Garzão
Projetista de Software
Azion Technologies
alex.garzao (at) azion.com