Multithreaded PHP

[Rob]

Whilst working on a new webmaster tool (which I will unleash to you lot in the future) I ran into an issue...

The tool performs dozens of checks on a web page and then returns a report.

The problem was that the speed was very very slow indeed and I needed a way to speed it up.

What I needed was a way to handle these checks in a multithreaded way (or at least a pseudo multithreaded way) so I rang up the only person I knew who could offer a suggestion (Graham Ellis of Well House Consultants)

He suggested a workaround and whilst it isnt true multithreading, it added severe speed to my application.

Here are the two core files which make it all happen.

This first file is the main php file you hit in your browser and for the sake of this example, needs a url passed in the querystring. The url is then passed to the 6 scripts by way of a command line parameter (yes, you read that right)

Code: Select all

<?php
 
$url = $_GET['url'];
$scripts = array(
   "script1.php",
    "script2.php",
    "script3.php",
    "script4.php",
    "script5.php",
    "script6.php");
 
# Set off parallel contacts to each of the scripts
 
for ($p=0; $p<count($scripts); $p++) {
    $fh[$p] = popen("/usr/local/bin/php ".
    "/home/yoursite/public_html/" . $scripts[$p] ." " .
    $url,"r");
    }
 
########################################
# DO AS MUCH PROCESSING IN HERE AS YOU LIKE
########################################
# your scripts here
# code here is processed whilst the above script calls are being processed
#
#
#
#
 
# We're done with our work - read back the results from the scripts
 
for ($p=0; $p<count($scripts); $p++) {
    $st = fgets($fh[$p]);
    $rs .=  $scripts[$p] . " : " . $st ."<br />";
    }
 
print $rs;
?>

What the above does is fire off the 6 scripts in parallel, passing $url to each script as a command line parameter.
It executes each script by executing the php binary itself, so the processes themselves live outside of apache.
The instant the scripts are fired, they work in the background whilst control is passed back to php to continue.
After we've done other intensive processing (whatever code you insert above) we can finally get all the scripts return values.

For this example, here is the code for one of the scripts that is called - note how it reads the url we passed from the command line

Code: Select all

<?
$url = $argv[1];
// code goes here
// in my webmaster tool, each of these scripts does a function
// such as check pageRank, alexa rank, backlinks, etc.
print "this is the returned value";
 ?>

The above method is best used on very intensive tasks that typically take a long time.

From my own tests, it's best to split up around 50% of the work. With that 50%, farm it out to a few different scripts and execute them with the above method. The other 50% can be processing within the main script before you finally read the results back in.

There are probabally better ways to further optimise and maybe better ways to split up the work load, but all of that experimentation I shall leave up to you.

Obviously this method can only be used in certain situations like mine... it wont make any differnce to scripts that are processor intensive, but for those which are not - this will reall add some speed if your script is suitable for splitting up in this way.

Be sure to post here with any info you gather.

Happy Coding

Digg this article

[grahame]

Many thanks for the "plug", Rob ... the work you've done and shown me is fascinating - I'll be coming back and running it against a sampling of my pages and it should be a great help to show us how we can improve our placement.

I have used similar techniques (though through Tcl/Expect or Python) in the past to check which systems are available on a network. It was a similar case to yours in that I wanted to check up on some 20 hosts, but couldn't afford to wait for serial ping timeouts. So I "paralleled" them. To this day, I have a Tcl page on our main web sites that, in a single click, tells me about some 10 different servers on which we look after sites ... and I can check from anywhere in the world.

One caution for anyone using this technique. I believe that pipes have a 4k buffer in them, so your popen'd scripts could come to a temporary stop if they generate a lot of output and haven't been "reaped".

[Ryan Fait]

That's cool! I can seriously use that in a project I'm working on!

[Tim356]

If it works as you say it does, then I will also be able to use it in a current project. Thanks guys!

[ropu]

hi, i also needed some multithreading/parallel task.
i had a different approach, using WebService like interface and sockets.

I have the main file, socket_cli_test.php, the one called by the user in the Browser.

That files calls internally n other ones via socket connection (HTTP request) and then grabs the answers. Ive also implemented time out, because i needed the whole process to finish before 3 seconds.

here is the code:
socket_cli_test.php

PHP: Select all

<?php
/*
 * Created on 31/01/2007
 *
 * Coded by: Ropu
 *  - Buenos Aires, Argentina  - z-tests
 */
error_reporting(E_ALL);
$cant_sockets = 9;
$timeout = 2.9;

// Write
$sockets = array();
$URI = '/~ropu/tests/socket_srv_test.php';
$Host =  'localhost';
$port = 80;

echo "start\n<xmp>";

list($start_m, $start_s) = explode(' ', microtime());
$start = $start_m + $start_s;

// Create a new socket
for($i=0; $i<$cant_sockets; $i++) {
    $sockets[$i] = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
}

// Connect to destination address
for( $i = 0; $i < $cant_sockets; $i++ ) {
    socket_connect($sockets[$i], $Host, $port);
//    socket_set_nonblock($sockets[$i]);
    socket_set_option($sockets[$i], SOL_SOCKET, SO_RCVTIMEO, array("sec" => 0, "usec" => 500000));
}

for($i=0; $i<$cant_sockets; $i++) {
    $ReqBody = $i;
    $ReqHeader =     "POST $URI HTTP/1.1\n".
                            "Host: $Host\n".
                            "User-Agent: Ropu Soft\n".
//                            "Authorization: Basic ".base64_encode($merid.':'.$merkey) . "\n".
                            "Content-Type: application/xml\n".
                            "Connection: close\n". // Important HEADER!!! to avoid delays when doing do{}while(reading)! 
                            "Content-Length:  ". strlen($ReqBody) ."\n\n". // without this one, post wont work
                            "$ReqBody\n";
    socket_write($sockets[$i], $ReqHeader);
}

$mid = $start;
while($mid-$start < $timeout){
    $new_socks = $sockets;
    // Calculate the remaing time before timeOut
    $working = $timeout - ($mid-$start);
    $rsec = (int)($working);
    $rusec = (int)(($working - $rsec) * 1000000);
    
    // Watch for activity in any socket
    $num_changed_sockets = socket_select($new_socks, $write = NULL, $except = NULL, $rsec , $rusec);
    for($i=0; $i<count($new_socks); $i++) {
        // Read all info in the socket
        do {
            echo $read = socket_read($new_socks[$i], 2048);
        }while(!empty($read));
//        echo "sock: ". $i . " => " .socket_strerror(socket_last_error($new_socks[$i])) ."\n";
    }
    // Discard used sockets
    $sockets = array_diff($sockets, $new_socks);
    // If all sockets are read, don't wait for the timeOut.
    if(count($sockets) < 1)
        break;
    // Calculate actual time
    list($mid_m, $mid_s) = explode(' ', microtime());
    $mid = $mid_m + $mid_s;
}

list($end_m, $end_s) = explode(' ', microtime());
$end = $end_m + $end_s;
echo "\n\nTardó: ". ($end-$start) ." segs"; 

?>

socket_srv_test.php

PHP: Select all

<?php
/*
 * Created on 31/01/2007
 *
 * Coded by: Ropu
 *  - Buenos Aires, Argentina  - z-tests
 */

// set_time_limit(1);
//print_r($_POST);
//print_r($HTTP_RAW_POST_DATA); echo "\n";
$_POST['id'] = $HTTP_RAW_POST_DATA;
 $id = rand(100, 100000);
 $message_log = fopen('test.log', "a");
 fwrite($message_log, sprintf("\n\rIn: %s:- Id: %d %d\n",date("D M j G:i:s T Y"), $id, $_POST['id'] ));
 echo "In: ".date("D M j G:i:s T Y") . " Id: $id " . $_POST['id'] ."\n";
 
 $delay = rand(1,5);
// if($_POST['id'] == 0)
//     $delay = $_POST['id'] + 2;
//$delay = 2;
 sleep($delay);

 fwrite($message_log, sprintf("\n\rOut: %s:- Id: %d %d Delay: %d\n",date("D M j G:i:s T Y"), $id,  $_POST['id'], $delay));
 echo "\nOut: ".date("D M j G:i:s T Y") ." Id: $id " . $_POST['id'] ." Delay: $delay\n\n";
 echo "--------------------------------------------------\n\n";
 //print_r($_SERVER);
 
?>

Feel free to send some feedback and suggestions.

hope it helps,

ropu

[Ryan Fait]

Thanks for sharing!

[matseth]

Hello
can anyone help me??
I am doing the code that Rob giving but there no output it only print out the name of the file.

Is this code supported in window xp because i want to do php multithread.

What i know that php multithread and forking is not supported in window platform and if supported it can only be used in CLI am i right.

plzz someone help me i am stuck to do my project

[Accurax]

Grahame... i think its true to say we all love you ... :wub: :wub: