<div dir="ltr"><div class="gmail_default" style="font-size:small;color:rgb(51,51,153)">One way or another, even if an external script is called, PHP will need to wait for the scripts completion, making the parallelization impossible or at least useless (since, to wait for a return code of an external script is still blocking).<br>
<br></div><div class="gmail_default" style="font-size:small;color:rgb(51,51,153)">I am not trying to find a workaround, I need to know how the fastcgi_reload_timeout works (if I understood it properly), if I properly disabled PHP buffering for my example case and how eventually to control those timeouts.<br>
</div><div class="gmail_default" style="font-size:small;color:rgb(51,51,153)">I'd like to address the central problem here, not closing my eyes on it.<br></div></div><div class="gmail_extra"><br clear="all"><div><font size="1"><span style="color:rgb(102,102,102)">---<br>
</span><b><span style="color:rgb(102,102,102)">B. R.</span></b><span style="color:rgb(102,102,102)"></span></font></div>
<br><br><div class="gmail_quote">On Sun, May 26, 2013 at 10:24 PM, Steve Holdoway <span dir="ltr"><<a href="mailto:steve@greengecko.co.nz" target="_blank">steve@greengecko.co.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Surely, you're still serialising the transfer with a loop?<br>
<div class="HOEnZb"><div class="h5"><br>
On Sun, 2013-05-26 at 22:11 -0400, B.R. wrote:<br>
> Thanks for your answer.<br>
><br>
> I didn't go into specifics because my problem doesn't rely at the<br>
> application-level logic.<br>
><br>
> What you describe is what my script does already.<br>
><br>
><br>
> However in this particular case I have 16 files weighting each a few<br>
> MB which need to be transfered back at once.<br>
><br>
><br>
> PHP allocates 30s for each loop turn (far enough to copy the file +<br>
> echo some output message about successes/failed completion).<br>
><br>
> Nginx cuts the execution avec fastcgi_read_timeout time even with my<br>
> efforts to cut down any buffering on PHP side (thus forcing the output<br>
> to be sent to Nginx to reinitialize the timeout counter).<br>
><br>
> That Nginx action is the center of my attention right now. How can I<br>
> get read of it in a scalable fashion (ie no fastcgi_read_time =<br>
> 9999999) ?<br>
> ---<br>
> B. R.<br>
><br>
><br>
><br>
><br>
> On Sun, May 26, 2013 at 9:46 PM, Steve Holdoway<br>
> <<a href="mailto:steve@greengecko.co.nz">steve@greengecko.co.nz</a>> wrote:<br>
> Write a script that lists the remote files, then checks for<br>
> the<br>
> existence of the file locally, and copy it if it doesn't<br>
> exist? That way<br>
> no internal loop is used - use a different exit code to note<br>
> whether<br>
> there was one copied, or there were none ready.<br>
><br>
> That way you scale for a single file transfer. There's nothing<br>
> to be<br>
> gained from looping internally - well performance-wise that<br>
> is.<br>
><br>
> Steve<br>
><br>
> On Sun, 2013-05-26 at 21:31 -0400, B.R. wrote:<br>
> > No ideas?<br>
> ><br>
> > ---<br>
> > B. R.<br>
> ><br>
> ><br>
> > On Sat, May 25, 2013 at 1:01 PM, B.R.<br>
> <<a href="mailto:reallfqq-nginx@yahoo.fr">reallfqq-nginx@yahoo.fr</a>> wrote:<br>
> > Hello,<br>
> ><br>
> ><br>
> > I am trying to understand how fastcgi_read_timout<br>
> works in<br>
> > Nginx.<br>
> ><br>
> ><br>
> > Here is what I wanna do:<br>
> ><br>
> > I list files (few MB each) on a distant place which<br>
> I copy one<br>
> > by one (loop) on the local disk through PHP.<br>
> ><br>
> > I do not know the amount of files I need to copy,<br>
> thus I do<br>
> > not know the total amount of time I need for the<br>
> script to<br>
> > finish its execution. What I know is that I can<br>
> ensure is a<br>
> > processing time limit per file.<br>
> ><br>
> > I would like my script not to be forcefully<br>
> interrupted by<br>
> > either sides (PHP or Nginx) before completion.<br>
> ><br>
> ><br>
> ><br>
> > What I did so far:<br>
> ><br>
> > - PHP has a 'max_execution_time' of 30s (default?).<br>
> In the<br>
> > loop copying files, I use the set_time_limit()<br>
> procedure to<br>
> > reinitialize the limit before each file copy, hence<br>
> each file<br>
> > processing has 30s to go: way enough!<br>
> ><br>
> ><br>
> > - The problem seems to lie on the Nginx side, with<br>
> the<br>
> > 'fastcgi_read_timeout' configuration entry.<br>
> ><br>
> > I can't ensure what maximum time I need, and I would<br>
> like not<br>
> > to use way-off values such as 2 weeks or 1 year<br>
> there. ;o)<br>
> ><br>
> > What I understood from the documentation is that the<br>
> timeout<br>
> > is reinitialized after a successful read: am I<br>
> right?<br>
> ><br>
> ><br>
> > The challenge is now to cut any buffering occurring<br>
> on the PHP<br>
> > side and let Nginx manage it (since the buffering<br>
> will occur<br>
> > after content is being read from the backend). Here<br>
> is what I<br>
> > did:<br>
> ><br>
> > * PHP's zlib.output_compression is deactivated by<br>
> default in<br>
> > PHP<br>
> ><br>
> > * I deactivated PHP's output_buffering (default is<br>
> 4096 bytes)<br>
> ><br>
> > * I am using the PHP flush() procedure at the end of<br>
> each<br>
> > iteration of the copying loop, after a message is<br>
> written to<br>
> > the output<br>
> ><br>
> ><br>
> ><br>
> > Current state:<br>
> ><br>
> > * The script seems to still be cut after the<br>
> expiration of the<br>
> > 'fastcgi_read_timout' limit (confirmed by the error<br>
> log entry<br>
> > 'upstream timed out (110: Connection timed out)<br>
> while reading<br>
> > upstream')<br>
> ><br>
> > * The PHP loop is entered several times since<br>
> multiple files<br>
> > have been copied<br>
> ><br>
> > * The output sent to the browser is cut before any<br>
> output from<br>
> > the loop appears<br>
> ><br>
> ><br>
> > It seems that there is still some unwanted buffering<br>
> on the<br>
> > PHP side.<br>
> ><br>
> > I also note that the PHP's flush() procedure doesn't<br>
> seem to<br>
> > work since the output in the browser doesn't contain<br>
> any<br>
> > message written after eahc file copy.<br>
> ><br>
> ><br>
> > Am I misunderstanding something about Nginx here<br>
> (especially<br>
> > about the 'fastcgi_read_timeout' directive)?<br>
> ><br>
> > Have you any intel/piece of advice on hte matter?<br>
> ><br>
> > Thanks,<br>
> ><br>
> > ---<br>
> > B. R.<br>
> ><br>
> ><br>
><br>
> > _______________________________________________<br>
> > nginx mailing list<br>
> > <a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
> > <a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
><br>
> --<br>
> Steve Holdoway BSc(Hons) MNZCS <<a href="mailto:steve@greengecko.co.nz">steve@greengecko.co.nz</a>><br>
> <a href="http://www.greengecko.co.nz" target="_blank">http://www.greengecko.co.nz</a><br>
> MSN: <a href="mailto:steve@greengecko.co.nz">steve@greengecko.co.nz</a><br>
> Skype: sholdowa<br>
><br>
> _______________________________________________<br>
> nginx mailing list<br>
> <a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
> <a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
><br>
><br>
> _______________________________________________<br>
> nginx mailing list<br>
> <a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
> <a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
<br>
--<br>
Steve Holdoway BSc(Hons) MNZCS <<a href="mailto:steve@greengecko.co.nz">steve@greengecko.co.nz</a>><br>
<a href="http://www.greengecko.co.nz" target="_blank">http://www.greengecko.co.nz</a><br>
MSN: <a href="mailto:steve@greengecko.co.nz">steve@greengecko.co.nz</a><br>
Skype: sholdowa<br>
<br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</div></div></blockquote></div><br></div>