欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  后端开发

PHPCrawl爬虫库抓取酷狗歌单

程序员文章站 2022-06-14 22:33:20
...
本人看了网络爬虫相关的视频后,蠢蠢欲动,也想爬点什么。最近Facebook上表情包大战很激烈,就想着把所有表情包都爬下来,却一时没有找到合适的VPN,因此只好仿照视频爬歌单,把酷狗最近一月精选歌曲和简单介绍抓取到本地。代码写得有点乱,自己不是很满意,并不想放上来丢人现眼。不过转念一想,这好歹是自己第一次爬虫,记录一下人生中的某个“第一次”有何不可?于是...就有了如下不堪入目的代码~~~(ps.我是直接增、删、改PHPCrawl爬虫库中给的example.php文件。由于抓取的数据量较小,所以没有考虑多进程什么的,不过我看了一下PHPCrawl的文档,发现PHPCrawl库已经把我能想到的功能都封装好了,实现起来很方便,YY着下次有时间就用它爬一些“大数据”,再用可视化工具进行数据分析,想想还有点小激动呢~)

header("Content-type:text/html;charset=utf-8");
// It may take a whils to crawl a site ...
set_time_limit(10000);
include("libs/PHPCrawler.class.php");
class MyCrawler extends PHPCrawler {
function handleDocumentInfo($DocInfo) {
// Just detect linebreak for output ("\n" in CLI-mode, otherwise "
").
if (PHP_SAPI == "cli") $lb = "\n";
else $lb = "
";

$url = $DocInfo->url;
$pat = "/http:\/\/www\.kugou\.com\/yy\/special\/single\/\d+\.html/";
if(preg_match($pat,$url) > 0){
$this->parseSonglist($DocInfo);
}
flush();
}

public function parseSonglist($DocInfo){
$content = $DocInfo->content;
$songlistArr = array();
$songlistArr['raw_url'] = $DocInfo->url;
//解析歌曲介绍
$matches = array();
$pat = "/名称:([^(
$ret = preg_match($pat,$content,$matches);
if($ret>0){
$songlistArr['title'] = $matches[1];
}else{
$songlistArr['title'] = '';
}
//解析歌曲
$pat = "/ $matches = array();
preg_match_all($pat,$content,$matches);
$songlistArr['songs'] = array();
for($i = 0;$i $song_title = $matches[1][$i];
array_push($songlistArr['songs'],array('title'=>$song_title));
}
echo "

";
print_r($songlistArr);
echo "
";
}
}
$crawler = new MyCrawler();
// URL to crawl
$start_url="http://www.kugou.com/yy/special/index/1-0-2.html";
$crawler->setURL($start_url);
// Only receive content of files with content-type "text/html"
$crawler->addContentTypeReceiveRule("#text/html#");
//链接扩展
$crawler->addURLFollowRule("#http://www\.kugou\.com/yy/special/single/\d+\.html$# i");
$crawler->addURLFollowRule("#http://www.kugou\.com/yy/special/index/\d+-\d+-2\.html$# i");
// Store and send cookie-data like a browser does
$crawler->enableCookieHandling(true);
// Set the traffic-limit to 1 MB(1000 * 1024) (in bytes,
// for testing we dont want to "suck" the whole site)
//爬取大小无限制
$crawler->setTrafficLimit(0);
// Thats enough, now here we go
$crawler->go();
// At the end, after the process is finished, we print a short
// report (see method getProcessReport() for more information)
$report = $crawler->getProcessReport();
if (PHP_SAPI == "cli") $lb = "\n";
else $lb = "
";

echo "Summary:".$lb;
echo "Links followed: ".$report->links_followed.$lb;
echo "Documents received: ".$report->files_received.$lb;
echo "Bytes received: ".$report->bytes_received." bytes".$lb;
echo "Process runtime: ".$report->process_runtime." sec".$lb;
?>

以上就介绍了PHPCrawl爬虫库抓取酷狗歌单,包括了方面的内容,希望对PHP教程有兴趣的朋友有所帮助。