Running multiple processes in PHP

Sometimes you need multiple commands to run in parallel to save up script processing time for repetitive tasks.
In the “Building a Video sharing site” project that i will present here soon, I needed a script to run multiple video processing jobs. PHP wasn’t meant to support multitasking, but with a few tricks we can emulate a multitasking environment.

On my research there were a couple of solutions that, althought they might seemed viable, had some weakness:

  • Using curl to open multiple instances of the script at once
    The problem was that the script was supposed to be accessible via the Web, and when trying to access external commands, this is a big security issue.
  • Using pcntl functions
    This could have been the best approach, but it required recompiling the PHP version with –enable-pcntl option, and many of us don’t have access to recompile PHP on the server. Beside that, it only works on linux systems, so it’s platform dependent.
  • Finally i found a nice class that simulates multi-threading and after some customisation i managed to set the multi tasking the way i wanted:

Threads.php

<?php
class Thread {
	var $pref ; // process reference
	var $pipes; // stdio
	var $buffer; // output buffer
	var $output;
	var $error;
	var $timeout;
	var $start_time;
 
 
	function Thread() {
		$this->pref = 0;
		$this->buffer = "";
		$this->pipes = (array)NULL;
		$this->output = "";
		$this->error="";
 
		$this->start_time = time();
		$this->timeout = 0;
	}
 
	function Create ($command) {
		$t = new Thread;
		$descriptor = array (0 => array ("pipe", "r"), 1 => array ("pipe", "w"), 2 => array ("pipe", "w"));
		//Open the resource to execute $command
		$t->pref = proc_open($command,$descriptor,$t->pipes);
		//Set STDOUT and STDERR to non-blocking 
		stream_set_blocking ($t->pipes[1], 0);
		stream_set_blocking ($t->pipes[2], 0);
		return $t;
	}
 
	//See if the command is still active
	function isActive () {
		$this->buffer .= $this->listen();
		$f = stream_get_meta_data ($this->pipes[1]);
		return !$f["eof"];
	}
 
	//Close the process
	function close () {
		$r = proc_close ($this->pref);
		$this->pref = NULL;
		return $r;
	}
 
	//Send a message to the command running
	function tell ($thought) {
		fwrite ($this->pipes[0], $thought);
	}
 
	//Get the command output produced so far
	function listen () {
		$buffer = $this->buffer;
		$this->buffer = "";
		while ($r = fgets ($this->pipes[1], 1024)) {
			$buffer .= $r;
			$this->output.=$r;
		}
		return $buffer;
	}
 
	//Get the status of the current runing process
	function getStatus(){
		return proc_get_status($this->pref);
	}
 
	//See if the command is taking too long to run (more than $this->timeout seconds)
	function isBusy(){
		return ($this->start_time>0) && ($this->start_time+$this->timeout<time());
	}
 
	//What command wrote to STDERR
	function getError () {
		$buffer = "";
		while ($r = fgets ($this->pipes[2], 1024)) {
			$buffer .= $r;
		}
		return $buffer;
	}
}
 
 
//Wrapper for Thread class
class Multithread{
	var $output;
	var $error;
	var $thread;
	var $commands = array();
 
	function __construct($commands){
		$this->commands = $commands;
 
		foreach ($this->commands as $key=>$command){
			$this->thread[$key]=Thread::create($command);
		}
	}
 
 
	function run(){
		$commands = $this->commands;
		//Cycle through commands
		while (count($commands)>0){
			foreach ($commands as $key=>$command){
				//Get the output and the errors
				$this->output[$key].=$this->thread[$key]->listen();
				$this->error[$key].=$this->thread[$key]->getError();
				//Check if command is still active
				if ($this->thread[$key]->isActive()){
					$this->output[$key].=$this->thread[$key]->listen();
					//Check if command is busy
					if ($this->thread[$key]->isBusy()){
						$this->thread[$key]->close();
						unset($commands[$key]);
					}
				} else {
					//Close the command and free resources
					$this->thread[$key]->close();
					unset($commands[$key]);
				}
			}
		}
		return $this->output;
	}
}
?>

I call the class using the following code:
Example.php

<?php
set_time_limit(0);
include "threads.php";
$commands = array('ffmpeg -i '.$inputFile[0].' '.$outputFile[0].' 2>&1','ffmpeg -i '.$inputFile[0].' '.$outputFile[0].' 2>&1');
$threads = new Multithread($commands);
$threads->run();
foreach ($threads->commands as $key=>$command){
	echo "Command ".$command.":<br>";
	echo "Output ".$threads->output[$key]."<br>";
	echo "Error ".$threads->error[$key]."<br><br>";
}
?>

Let’s talk about what the script does:

First in Create method it defines some descriptors to use with proc_open. The standard descriptors for shell commands are:
0 for stdin (what is read by the process)
1 for stdout (what is the output of the runing process)
2 for stderror (errors that the process throw).
You can customize your settings here, for example you can choose to log all the errors in a file like this:

2 => array("file", $path_to_error_log_file, "a")

By setting stream_set_blocking($t->pipes[1],0), and stream_set_blocking($t->pipes[2],0) we  tell  php to not wait for the process to finnish with the output and error reporting, before runing another command. This is an important step, in achieving multitasking.
The current state of the runing process is achieved using isActive method, that will read from the stdout (stream_get_meta_data($this->pipes[1])) to see if the process is still active or not.
The command output will be generated by the $t->listen method. Aditionaly, you can capture the errors by calling getError method.
Close method will close the command strem and free the resources associated.
isBusy method will check to see if the $timeout time was passed. I recomend setting timeout high, if you know that your script take longer time to load. If timeout is set to 0 it will not timeout any script.

In the attached file you will see an example implementation of this technique. 3 scripts (one.php, two.php, three.php) will write data in a log file with one second delay between writings, to see how the multitasking is achieved, err.php will simulate a script producing an error.
Have fun experimenting with the script, and if you have any questions, just leave a comment here.

Download code for this post

No related posts.

37 Responses to “Running multiple processes in PHP”

  1. masterb says:

    Great script, works perfect!

  2. Nickolas Wood says:

    Hi,
    I have modified your class a little bit and fixed a couple bugs for you; figured I would contribute back. First off, line 107 and 108 of your threads.php file return undefined reference notices when ini_set(‘display_errors’, ‘On’); and error_reporting(E_ALL); are used. Secondly, the thread timeout of 0 on line 20 of the threads.php file (with no dynamic way of changing it) causes weird behavior when you start checking each thread for null output and resubmitting the command to a new multithread object on such null output. If a thread takes longer then 0 seconds to complete, the isbusy routine kills it. Most commands take longer then 0 seconds to return anything so I kept getting null output. This threw me for a loop for awhile.

    These three problems have been fixed in the below code. I have also made a couple of minor changes to better handle output. Look for lines with #MODIFIED in them, those I changed. I hope someone else finds it useful.

    Great work on your multithread class by the way. It helped me alot.

    START CODE BLOCK:

    pref = 0;
    $this->buffer = “”;
    $this->pipes = (array)NULL;
    $this->output = “”;
    $this->error=”";

    $this->start_time = time();
    $this->timeout = $timeout; #MODIFIED, used $timeout
    }

    function Create ($command, $timeout) { #MODIFIED, added $timeout
    $t = new Thread($timeout); #MODIFIED, used timeout
    $descriptor = array (0 => array (“pipe”, “r”), 1 => array (“pipe”, “w”), 2 => array (“pipe”, “w”));
    //Open the resource to execute $command
    $t->pref = proc_open($command,$descriptor,$t->pipes);
    //Set STDOUT and STDERR to non-blocking
    stream_set_blocking ($t->pipes[1], 0);
    stream_set_blocking ($t->pipes[2], 0);
    return $t;
    }

    //See if the command is still active
    function isActive () {
    $this->buffer .= $this->listen();
    $f = stream_get_meta_data ($this->pipes[1]);
    return !$f["eof"];
    }

    //Close the process
    function close () {
    $r = proc_close ($this->pref);
    $this->pref = NULL;
    return $r;
    }

    //Send a message to the command running
    function tell ($thought) {
    fwrite ($this->pipes[0], $thought);
    }

    //Get the command output produced so far
    function listen () {
    $buffer = $this->buffer;
    $this->buffer = “”;
    while ($r = stream_get_contents($this->pipes[1])) { #MODIFIED, replaced fgets with stream_get_contents
    #MODIFIED, removed $this->output appending (unneeded)
    $buffer .= $r;
    }
    return $buffer;
    }

    //Get the status of the current runing process
    function getStatus(){
    return proc_get_status($this->pref);
    }

    //See if the command is taking too long to run (more than $this->timeout seconds)
    function isBusy(){
    return ($this->start_time>0) && ($this->start_time+$this->timeoutpipes[2])) { #MODIFIED, replaced fgets with stream_get_contents
    $buffer .= $r;
    }
    return $buffer;
    }
    }

    //Wrapper for Thread class
    class Multithread{
    var $output;
    var $error;
    var $thread;
    var $commands = array();

    function __construct($commands, $timeout){ #MODIFED, added $timeout
    $this->commands = $commands;

    foreach ($this->commands as $key=>$command){
    $this->thread[$key]=Thread::create($command, $timeout); #MODIFIED, used $timeout
    $this->output[$key] = “”; #MODIFIED, added this line to remove undefined reference notices
    $this->error[$key] = “”; #MODIFIED, added this line to remove undefined reference notices
    }
    }

    function run(){
    $commands = $this->commands;
    //Cycle through commands
    while (count($commands)>0){
    foreach ($commands as $key=>$command){
    //Check if command is still active
    if ($this->thread[$key]->isActive()){
    //Get the output and the errors
    $this->output[$key].=$this->thread[$key]->listen(); #I removed this line on my server as I don’t need it, I want all output or nothing. I could see where it is useful however
    $this->error[$key].=$this->thread[$key]->getError(); #I removed this line on my server as I don’t need it, I want all output or nothing. I could see where it is useful however
    //Check if command is busy
    if ($this->thread[$key]->isBusy()){
    #MODIFIED, removed $this-output (unneeded, if we kill a process then its output is unusable and there is no need to update it further)
    $this->thread[$key]->close();
    unset($commands[$key]);
    }
    } else {
    //Close the command, gather results and free resources #MODIFIED, added, gather results comment
    $this->output[$key].=$this->thread[$key]->listen(); #MODIFIED, added this line (want to make sure we have all output before we close a thread)
    $this->error[$key].=$this->thread[$key]->getError(); #MODIFIED, added this line (want to make sure we have all output before we close a thread)
    $this->thread[$key]->close();
    unset($commands[$key]);
    }
    }
    }
    return $this->output;
    }
    }
    ?>

    END CODE BLOCK:

    I know, the formatting sucks but I couldn’t find a nice way of doing it. Now, I call this class like so:

    START CODE BLOCK:

    $threads = new Multithread($command_list_exec, $execution_time);
    $threads->run();

    END CODE BLOCK:

    As I mentioned earlier, I check each process’s output against several strpos instances for known conditions. The most common problem I saw was no output at all. If this was the case then it was killed before anything had a chance to be reported and it needs to be run again.

    It is vitally important that set_time_limit($execution_time); is used in php pages that call the multithread class so that a rogue process doesn’t cripple a with endless thread creations/kills.

    Hope this helps.

    Thanks again!!

  3. Nickolas Wood says:

    I apologize, I missed a couple lines of code at the very top; at the first START CODE BLOCK. This codes should be inserted there:

    START CODE BLOCK:

    END CODE BLOCK:

  4. Nickolas Wood says:

    Umm, let try this again removing the initial php tag:

    START CODE BLOCK:

    class Thread {
    var $pref ; // process reference
    var $pipes; // stdio
    var $buffer; // output buffer
    var $output;
    var $error;
    var $timeout;
    var $start_time;

    function Thread($timeout) {
    $this->

    END CODE BLOCK:

    This will be my last attempt so that I don’t over post here.

  5. Silviu says:

    Hello. I’ve tried your code and I get the following error. Please help :)

    “Error ‘tasks’ is not recognized as an internal or external command, operable program or batch file.”

  6. carlos Pimentel says:

    Hi Nickolas Wood,
    could you please post the code that you’ve created in pastebin.com or similar website ?
    This code isn’t working for me, sometime I receive the output from the process some time I don’t and I almost crazy debugging this…
    Thanks !

  7. bee says:

    Hey cool threads.php class, Works great. I got a question anyway, what is the maximum number of scripts that can be multitasked?

    Thanks,
    bee.

  8. …Trackback: More Informations on that topic…

    [...]Excellent weblog right here! Additionally your site quite a bit up very fast![...]…

  9. Hi there! This is my 1st comment here so I just wanted to
    give a quick shout out and say I genuinely enjoy reading through your articles.

    Can you suggest any other blogs/websites/forums that cover
    the same subjects? Many thanks!

  10. Extraordinaires Fendi Artiste Totes

    Si vous décidez de diffuser à la suite de l’artère Santee de Shedd Angeles et de plus récentes Canal Avenue de York, vous remarquerez sosie sacs à sa disposition sur la route. Faux bacs pourraient très bien être à l’aise grâce à l’utilisation de sa mousseline électronique, longchamp taschen reparatur bon achat adressage et logos d’art par ailleurs mal orthographiés sont definitily un élément. S’il vous arrive d’apercevoir le fait, comment ça va raboter un voyage à choisir? Le problème est généralement que un autre sac Louis Vuitton est sans doute trop cher et ne devrait pas finir par être fourni en utilisant un individu normal. Un seul coup d’oeil longchamp taschen ratingen dans le coût ainsi que les espoirs d’une personne devrait recevoir péri. Les vrais Vuitton sacs à main Louis viennent également dans quelques modèles assez, couleurs ainsi que le dimensionnement de Il accueille à peu près tous la dépendance unique sur absolument gratuit themes.Memorable vous faire des cheveux qui va générer son homonyme longchamp taschen roseau incroyablement, les styles de couleurs chics cependant animés présentant aujourd’hui de chic must-haves. Pour être honnête, je revêts tout et tous les analystes. Ou Prenons votre Hobo Hysteria trouvé dans insignes beige typique néanmoins couplée ces temps à une autre à mourir pour le matériel de gemme.

    Hampton rapide pause mode Cette écharpe Impress Demi profite à une poche ou un sac externe particulier, filez premier securer, marque écharpe tissu de tissu par voie de daim et de plus digne cuir des ménages. Cette demi unique de 200 $ s’applique. Vous y trouverez des sacs à main fabuleux supplémentaires sur le net ou peut-être en général le store.Although ou même regardé méthodes pour longchamp taschen rechnung économiser votre argent en collectant des sacs à main mode reconnus? Semble être extrêmement difficile? Il n’est tout simplement pas! La fonction sac hermes birkin est particulier avec le Jane Birkin idée présentateur. confortablement personnalisable earn sac Birkin ou Kelly felix étuis de transport à l’intérieur par rapport à consulter mois. vous pourriez favoriser une nuance spécifique de couleur, chamois, des semaines Diamètre intérieur, le travail intérieur de diamètre thus.The dames ensemble de sac à main est considérée comme l’un bon nombre de gros sacs à main imités en ces temps modernes. Une personne spéciale avancé connecté avec des sacs à main abordables vous devez savoir pour investir dans un sac à main réelle est connu pour devenir cette volonté d’argent et plus de style. Ces jours-ci nous allons parler de la meilleure façon préféré ne pas mentionner souhaitable ce genre de sélection est.

    extraordinaires Fendi Artiste emballages, les emballages et même les portefeuilles et sacs à main étonnants Fendi sacs à main, pochettes personnalisées abordables ainsi que des portefeuilles, Une touche longchamps taschen rot Fendi Le passé historique Fendi est sûrement un designer italien de votre maison la plus courante que pour les embrayages de “baguette”. Qu’il a été publié quand il s’agit de 1925 comme une peau sans parler détaillant d’alligator dans la capitale, à l’époque actuelle est vraiment un des produits de l’extravagance internationales et le nom de marque de solutions achetés à la suite de LVMH.Some très paresseux, tout en étant l’âge âgé et longchamp le pliage for sale in the philippines donc chic. Je ne suis pas en général dans des cas extrêmement organisés, que je préfère généralement une récession plus pour vraiment mes propres backpags. Et c’est aussi sans doute précisément pourquoi mon groupe est excité au sujet de Proenza Schouler PS1 sacs à main! Vous pouvez aussi augmenter le risque pour les couleurs selon vos normes. Il ya beaucoup de couleurs pour prendre le sélectionner. Certaines options comprennent les fruits, violet, gris, noir coloré, lumineux, écologique, bronzage léger, en plus grisâtre.

  11. アディダス ジェレミー スコット 格安販売http://www.back2iraq.com/アディダス-ジェレミー-スコット-c-85.html

  12. click here says:

    Acrílico: contracción, sabor picante, ceniza negro duro quebradizo o esfera hueca.

  13. basket basse says:

    Taoka is really a excellent coach, but has made errors in judging his players and also the opposition. He nonetheless has an eye for expertise although, as he recognized Uozomi’s possible and gave him reassurance when Uozomi’s abilities were weak and he was considering of quitting. Taoka also recognized that when Sendoh and Fukuda joined the group they would grow to be crucial gamers.
    basket basse http://www.brianjordanmusic.co.uk/nike-cortez.asp?id=basket-basse

  14. Essentially the most reliable device professionals have for motivating call center agents could be the a single product that is certainly most frequently mentioned as missing when agents are surveyed. That device is recognition. Agents need to be acknowledged once they do a thing very effectively, of course, but it is not just a make a difference of acknowledging major performers..
    louis vuitton speedy graffiti http://www.dfi-bpi.com/lv/louis-vuitton-speedy-graffiti.html

  15. Team developing is a crucial work inside the continuing growth of any organization, no matter if brand name new or firmly established. A company’s employees are its perfect resources, and uniting them to feel, act, and function being a team is very important in almost any sector. There’s lots of completely different ways to make a team: interactive routines, skill growth workshops, and last although not minimum, only finding the time to acquire to know a single one other on an expert basis.
    louis vuitton keepall bags http://www.strouddistrictcouncil.co.uk/lv/louis-vuitton-keepall-bags.html

  16. The 2 lived together briefly in his mansion in La, and continued a warm friendship even right after she still left his home. For his psychological, energetic performance, Sal was once again nominated for an Oscar. And again, Sal lost, the award instead planning to Peter Ustinov for his function in Spartacus.

  17. Thanks a bunch for sharing this with all of us you really understand what you are talking about! Bookmarked. Please also visit my site =). We will have a hyperlink change arrangement between us!
    prada bowler bag http://www.carris.com/CFIDE/classes/prada.php?p=prada-bowler-bag.html

  18. Simply want to say your article is as surprising. The clarity in your post is just cool and i can assume you’re an expert on this subject. Well with your permission allow me to grab your RSS feed to keep updated with forthcoming post. Thanks a million and please carry on the enjoyable work.
    prada shoes 2012 http://www.caesarhouse.com/Foto/prada.asp?p=prada-shoes-2012.html

  19. I am really delighted to glance at this weblog posts which consists of tons of helpful data, thanks for providing these kinds of data.
    prada mens sandals http://www.carris.com/CFIDE/classes/prada.php?p=prada-mens-sandals.html

  20. Hello there, You’ve done an excellent job. I’ll certainly digg it and personally suggest to my friends. I’m confident they will be benefited from this website.
    spaccio prada firenze http://www.caesarhouse.com/Foto/prada.asp?p=spaccio-prada-firenze.html

  21. cheap pradas says:

    You made some good points there. I did a search on the subject matter and found most people will go along with with your blog.
    cheap pradas http://www.carris.com/CFIDE/classes/prada.php?p=cheap-pradas.html

  22. Wow, marvelous blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your website is magnificent, let alone the content!
    prada fur bag http://www.carris.com/CFIDE/classes/prada.php?p=prada-fur-bag.html

  23. I’m still learning from you, while I’m trying to achieve my goals. I definitely love reading everything that is written on your site.Keep the aarticles coming. I liked it!
    mulberry bags sale http://www.carris.com/CFIDE/componentutils/mulberry.php?p=mulberry-bags-sale.html

  24. You really make it seem so easy with your presentation but I find this topic to be actually something that I think I would never understand. It seems too complex and extremely broad for me. I am looking forward for your next post, I will try to get the hang of it!
    mulberry handbags http://www.carris.com/CFIDE/componentutils/mulberry.php?p=mulberry-handbags.html

  25. dhxmhhbrc says:

    Running multiple processes in PHP | PHP Code
    dhxmhhbrc http://www.go235wl8i13zl2i1r31ib884gou5u90us.org/
    [url=http://www.go235wl8i13zl2i1r31ib884gou5u90us.org/]udhxmhhbrc[/url]
    adhxmhhbrc

  26. Within this approach, the thrower kicks one leg backward in direction of the throwing region and attempts to shift the excess weight in direction of the front with the circle. This approach is easier to learn as well as the mentor teaches this shot put approach to the new throwers for the reason that they’re able to conveniently learn it. To get a major aggressive edge, excellent gliding methods are crucial and within this approach, maintain the shot inside the suitable hand below your chin as well as your suitable foot should really be pointed inside the reverse path with the shot put course..
    christian louboutin leopard sneakers http://www.elecro.co.uk/christianlouboutin.asp
    christian louboutin leopard sneakers

  27. I do trust all the concepts you’ve offered for your post. They’re very convincing and can certainly work. Still, the posts are too quick for newbies. Could you please extend them a little from next time? Thanks for the post.

  28. I have been exploring for a little bit for any high-quality articles or blog posts in this sort of house . Exploring in Yahoo I eventually stumbled upon this site. Studying this information So iˇm satisfied to convey that I have an incredibly excellent uncanny feeling I found out just what I needed. I so much for sure will make sure to donˇt fail to remember this web site and provides it a look on a constant basis.

  29. un lecteur de flux rss. Il s’agit without doute du as well abouti des lecteurs flux actuellement sur l’AppStore. Disponible au prix de 3.99.10 drags grandes tailles qui ont du flair jusqu
    ray ban prescription sunglasses with logo http://www.referenceline.com/rayban.asp?p=162
    ray ban prescription sunglasses with logo

  30. dsceyqiq.dpef, Buy Dapoxetine, shIpeaN, [url=http://www.dapoxetine247.com/]Dapoxetine[/url], ntYqbiJ, http://www.dapoxetine247.com/ Www dapoxetine, ojVVtqC.

Leave a Reply