Python使用Beautiful Soup包编写爬虫时的一些关键点
程序员文章站
2022-06-20 09:44:30
1.善于利用soup节点的parent属性
比如对于已经得到了如下html代码:
1.善于利用soup节点的parent属性
比如对于已经得到了如下html代码:
<td style="padding-left:0" width="60%"><label>November</label> <input type="Hidden" id="cboMonth1" name="cboMonth1" value="11"> </td><td style="padding-right:0;" width="40%"> <label>2012</label> <input type="Hidden" id="cboYear1" name="cboYear1" value="2012"> </td>
的soup变量eachMonthHeader了。
想要提取其中的
Month的label的值:November
和Year的label的值:2012
最简单,也是最省事的办法是,直接搜两个label,然后肯定会找到这两个label,然后分别对应着Month和Year的label,然后获得对应的string即可:
foundTwoLabel = eachMonthHeader.findAll("label"); print "foundTwoLabel=",foundTwoLabel; monthLabel = foundTwoLabel[0]; yearLabel = foundTwoLabel[1]; monthStr = monthLabel.string; yearStr = yearLabel.string; print "monthStr=",monthStr; # monthStr= November print "yearStr=",yearStr; # yearStr= 2012
但是很明显,这样的逻辑性很不好,而且万一处理多个这样的soup变量,而且两者的顺便颠倒了,那么结果也就错误了。
此时,可以考虑利用soup变量的parent属性,从一个soup变量本身,获得其上一级的soup变量。
示例代码如下:
# <td style="padding-left:0" width="60%"><label>November</label> # <input type="Hidden" id="cboMonth1" name="cboMonth1" value="11"> # </td><td style="padding-right:0;" width="40%"> # <label>2012</label> # <input type="Hidden" id="cboYear1" name="cboYear1" value="2012"> # </td> foundCboMonth = eachMonthHeader.find("input", {"id":re.compile("cboMonth\d+")}); #print "foundCboMonth=",foundCboMonth; tdMonth = foundCboMonth.parent; #print "tdMonth=",tdMonth; tdMonthLabel = tdMonth.label; #print "tdMonthLabel=",tdMonthLabel; monthStr = tdMonthLabel.string; print "monthStr=",monthStr; foundCboYear = eachMonthHeader.find("input", {"id":re.compile("cboYear\d+")}); #print "foundCboYear=",foundCboYear; tdYear = foundCboYear.parent; #print "tdYear=",tdYear; tdYearLabel = tdYear.label; #print "tdYearLabel=",tdYearLabel; yearStr = tdYearLabel.string; print "yearStr=",yearStr;
我们再来看一个例子:
from BeautifulSoup import BeautifulSoup doc = ['<html><head><title>Page title</title></head>', '<body><p id="firstpara" align="center">This is paragraph <b>one</b>.', '<p id="secondpara" align="blah">This is paragraph <b>two</b>.', '</html>'] soup = BeautifulSoup(''.join(doc)) print soup.prettify() # <html> # <head> # <title> # Page title # </title> # </head> # <body> # <p id="firstpara" align="center"> # This is paragraph # <b> # one # </b> # . # </p> # <p id="secondpara" align="blah"> # This is paragraph # <b> # two # </b> # . # </p> # </body> # </html>
这个例子中,<HEAD> Tag的parent是<HTML> Tag. <HTML> Tag 的parent是BeautifulSoup 剖析对象自己。 剖析对象的parent是None. 利用parent,你可以向前遍历剖析树。
soup.head.parent.name # u'html' soup.head.parent.parent.__class__.__name__ # 'BeautifulSoup' soup.parent == None # True
2.当解析非UTF-8或ASCII编码类型的HTML时,需要指定对应的字符编码
当html为ASCII或UTF-8编码时,可以不指定html字符编码,便可正确解析html为对应的soup:
#这里respHtml是ASCII或UTF-8编码,此时可以不指定编码类型,即可正确解析出对应的soup soup = BeautifulSoup(respHtml);
当html为其他类型编码,比如GB2312的话,则需要指定相应的字符编码,BeautifulSoup才能正确解析出对应的soup:
比如:
#此处respHtml是GB2312编码的,所以要指定该编码类型,BeautifulSoup才能解析出对应的soup htmlCharset = "GB2312"; soup = BeautifulSoup(respHtml, fromEncoding=htmlCharset);