htmlunit和爬虫问题

最近写了个爬虫项目，暂时放在github上。https://github.com/shenbaise/goodcrawler

 

对于爬虫最难的问题应该是javascript和ajax的处理。现在很多网站使用大量ajax，普通爬虫无法获取js生成的

内容。

 

目前大体上有2中方式来解决这个问题。

一是使用htmlunit之类的所谓handless browser。htmlunit对js支持相当不错。只是适用起来不很方便。想要知道什么时候js执行完了比较困难。

官网 FQA对这个问题的说明。

 

 

Nothing happens when using HtmlUnit with AJAX, although page works correctly in browsers. What's wrong?

The main thread using HtmlUnit may be finishing execution before allowing background threads to run. You have a couple of options:

1. webClient.setAjaxController(new NicelyResynchronizingAjaxController()); will tell your WebClient instance to re-synchronize asynchronous XHR.
2. webClient.waitForBackgroundJavaScript(10000); or webClient.waitForBackgroundJavaScriptStartingBefore(10000); just after getting the page and before manipulating it.
3. Explicitly wait for a condition that is expected be fulfilled when your JavaScript runs, e.g.

           //try 20 times to wait .5 second each for filling the page.
           for (int i = 0; i < 20; i++) {
               if (condition_to_happen_after_js_execution) {
                   break;
               }
               synchronized (page) {
                   page.wait(500);
               }
           }

我的测试：

public static void testYouku() throws Exception {
		String url = "http://v.youku.com/v_show/id_XNDc2MDkzMTIw.html";
		String xurl = "http://v.youku.com/v_vpofficiallistv5/id_119023280_showid_271942_page_2?__rt=1&__ro=listitem_page2";
		// String a = "<a page=\"2\">178-101</a>";
		// String url="http://www.baidu.com";
		// 模拟一个浏览器
		final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_17);

		LogFactory.getFactory().setAttribute("org.apache.commons.logging.Log","org.apache.commons.logging.impl.NoOpLog");
		java.util.logging.Logger.getLogger("net.sourceforge.htmlunit").setLevel(java.util.logging.Level.OFF);
		webClient.getOptions().setThrowExceptionOnScriptError(false);
		webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
		// final WebClient webClient=new
		// WebClient(BrowserVersion.FIREFOX_10,"http://myproxyserver",8000);
		// //使用代理
		// final WebClient webClient2=new
		// WebClient(BrowserVersion.INTERNET_EXPLORER_10);
		// 设置webClient的相关参数
		webClient.getOptions().setJavaScriptEnabled(true);
		webClient.getOptions().setActiveXNative(false);
		webClient.getOptions().setCssEnabled(false);
		webClient.getOptions().setThrowExceptionOnScriptError(false);
		webClient.waitForBackgroundJavaScript(600*1000);
		webClient.setAjaxController(new NicelyResynchronizingAjaxController());
		
		webClient.getOptions().setJavaScriptEnabled(true);  
		/*
		webClient.setJavaScriptTimeout(3600*1000);  
		webClient.getOptions().setRedirectEnabled(true);  
		webClient.getOptions().setThrowExceptionOnScriptError(true);  
		webClient.getOptions().setThrowExceptionOnFailingStatusCode(true);  
		webClient.getOptions().setTimeout(3600*1000);  
		webClient.waitForBackgroundJavaScript(600*1000);  
		*/
//		webClient.waitForBackgroundJavaScript(600*1000);
		webClient.setAjaxController(new NicelyResynchronizingAjaxController()); 
		
		// 模拟浏览器打开一个目标网址
		final HtmlPage page = webClient.getPage(url);
//		该方法在getPage()方法之后调用才能生效
		webClient.waitForBackgroundJavaScript(1000*3);
		webClient.setJavaScriptTimeout(0);
//		Thread.sleep(1000 *3L);
//		String js = "javascript:checkShowFollow('271942','2');";
//		ScriptResult sr = page.executeJavaScript(js);
//		HtmlPage newPage = (HtmlPage) sr.getNewPage();
//		System.out.println("new page.asText=" + newPage.asText());
//		System.out.println("page.asText=" + page.asText());
//		System.out.println("page.getUrl=" + page.getUrl());
		List links = (List) page.getByXPath ("//*[@id=\"groups_tab\"]/div[1]/ul/li[1]/a");
		if(null!=links){
			System.out.println(links.size());
			HtmlAnchor link = (HtmlAnchor) links.get(0);
			System.out.println(link.asXml());
			HtmlPage p = link.click();
			webClient.waitForBackgroundJavaScript(1000*3L);
//			webClient.waitForBackgroundJavaScriptStartingBefore(1000L);
//			Thread.sleep(3000L);
			System.out.println(p.asText());
		}
	}

 youku上面超过100集会拆分多个列表展示，全部是js生成的。

 

上面代码执行的成功率不是很高。

 

2013.12.05 19:25:58 Re-synchronized call to http://v.youku.com/v_vpofficiallistv5/id_119023280_showid_271942_page_2?__rt=1&__ro=listitem_page2

2013.12.05 19:26:00 Expected content type of 'application/javascript' or 'application/ecmascript' for remotely loaded JavaScript element at 'http://html.atm.youku.com/html?p=74,75,558,747,753,850,101618,616,101564&ct=h&cs=2224|2210|2209&td=0&s=271942&v=119023280&u=83980375&k=%E7%BD%91%E7%90%83%E7%8E%8B%E5%AD%90|&sid=1386242750138i0r&tt=网球王子 001&pu=http://v.youku.com/v_show/id_XNDc2MDkzMTIw.html&ref=', but got 'text/html'.

 

其实只要能拿到Re-synchronized call to后面的URL后直接使用httpclient就可以了。但是htmlunit没有提供相关方法。debug了一下源码还是没找到什么时候这个URL被生成的（这个URL肯定是js函数中生成并异步调用的）。

 

继续研究。。。

 

第二种方法是使用selenium，jdic直接调用浏览器。

 

这两种方式都太慢。相对来说第一种方式效率比第二种要好一些。