流方式实现多线程采集有关问题，请高手分析上

程序员文章站 2022-06-16 13:43:21

...

流方式实现多线程采集问题，请高手分析下
采集内容速度慢，我一直很头大，最近在研究多线程采集，下面贴出比较代码，有两个问题，一是获取的结果长度有点不一致；二是效率是不是还不够高？大伙帮忙分析，测试！

PHP code


';
}
$timeEnd = microtimeFloat();
echo sprintf("Spend time: %s second(s)\n", $timeEnd - $timeStart),'
';
$timeStart = microtimeFloat();
$timeout = 30;
$status = array();
$retdata = array();
$sockets = array();
$userAgent = $_SERVER['HTTP_USER_AGENT'];
foreach($urls as $id => $url) {
    $tmp = parse_url($url);
    $host = $tmp['host'];
    $path = isset($tmp['path'])?$tmp['path']:'/';
    empty($tmp['query']) or $path .= '?' . $tmp['query'];
    if (empty($tmp['port'])) {
        $port = $tmp['scheme'] == 'https' ? 443 : 80;
    } else $port = $tmp['port'];
    $fp = stream_socket_client("$host:$port", $errno, $errstr, 30);
    if (!$fp) {
        $status[$id] = "failed, $errno $errstr";
    } else {
        $status[$id] = "in progress";
        $retdata[$id] = '';
        $sockets[$id] = $fp;
        fwrite($fp, "GET $path HTTP/1.1\r\nHost: $host\r\nUser-Agent: $userAgent\r\nConnection: Close\r\n\r\n");
    }
}
// Now, wait for the results to come back in

while (count($sockets)) {
    $read = $write = $sockets;
    //This is the magic function - explained below
    if (stream_select($read, $write = null, $e = null, $timeout)) {
        //readable sockets either have data for us, or are failed connection attempts
        foreach ($read as $r) {
            $id = array_search($r, $sockets);
            $data = fread($r, 8192);
            if (strlen($data) == 0) {
                if ($status[$id] == "in progress") {
                    $status[$id] = "failed to connect";
                }
                fclose($r);
                unset($sockets[$id]);
            } else {
                $retdata[$id] .= $data;
            }
        }
    }
}
foreach($retdata as $data){
    $data = trim(substr($data, strpos($data, "\r\n\r\n") + 4));
    echo strlen($data),'
';
}
$timeEnd = microtimeFloat();
echo sprintf("Spend time: %s second(s)\n", $timeEnd - $timeStart);
?>

------解决方案--------------------
你可以尝试 curl_multi_.... 并发执行
这样可尽可能的减少 php 指令，至于楼上两位说的问题。绝不是php所能解决的

------解决方案--------------------
当然，file_get_contents()是阻塞型的，所以如果是执行多个抓取任务，当然会慢。
而socket_*(), fsockopen(), stream_*()都是非阻塞的。
------解决方案--------------------
慢到什么程度?

试下加上这个:

$context = stream_context_create(array('http' => array('header'=>'Connection: close')));
file_get_contents(".....",false,$context);

声明：本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系admin@php.cn核实处理。