HTMLParser模块详解

使用HTMLParser模块解析HTML页面

    HTMLParser是python用来解析html和xhtml文件格式的模块。它可以分析出html里面的标签、数据等等，是一种处理html的简便途径。HTMLParser采用的是一种事件驱动的模式，当HTMLParser找到一个特定的标记时，它会去调用一个用户定义的函数，以此来通知程序处理。它主要的回调函数的命名都是以handler_开头的，都HTMLParser的成员函数。当我们使用时，就从HTMLParser派生出新的类，然后重新定义这几个以handler_开头的函数即可。和在htmllib中的解析器不同，这个解析器并不是基于sgmllib模块的SGML解析器。
 htmllib模块和sgmllib模块从Python2.6开始不鼓励使用，3.0以后的版本中被移除~~~
HTMLParser
class HTMLParser.HTMLParser
The HTMLParser class is instantiated without arguments.
HTMLParser类不需要参数进行实例化。
An HTMLParser instance is fed HTML data and calls handler functions when tags begin and end. The HTMLParser class is meant to be overridden by the user to provide a desired behavior.
一个HTMLParser实例传入HTML数据并且当传入开始和结束的tags参数时调用handler函数。HTMLParser类通过被用户重写方法来提供所需要的行为。
Unlike the parser in htmllib, this parser does not check that end tags match start tags or call the end-tag handler for elements which are closed implicitly by closing an outer element.
与htmllib中的解析器不同的是，这个解析器不检查结尾标签与开始标签的匹配或者对由关闭外部元素表明是关闭的元素调用结束标签handler
An exception is defined as well:
异常也被定义了：
exception HTMLParser.HTMLParseError
Exception raised by the HTMLParser class when it encounters an error while parsing. This exception provides three attributes: msg is a brief message explaining the error, lineno is the number of the line on which the broken construct was detected, and offset is the number of characters into the line at which the construct starts.
当遇到解析遇到错误时，该类将产生一个异常。该异常提供了三个属性：msg是用来解释错误的消息，lineno是检测到打断构造的行数，offset则是该行产生该构造的字符数。
HTMLParser instances have the following methods:
HTMLParser实例有以下方法：
HTMLParser.reset()
Reset the instance. Loses all unprocessed data. This is called implicitly at instantiation time.
重置该实例。失去所有未处理的数据。这个在实例化对象时被隐含地调用。
HTMLParser.feed(data)
Feed some text to the parser. It is processed insofar as it consists of complete elements; incomplete data is buffered until more data is fed or close() is called.
提供一些文本给解析器。在由完整元素组成的限度内进行处理，不完整的数据被缓冲直到更多的数据提供或者close()被调用。
HTMLParser.close()
Force processing of all buffered data as if it were followed by an end-of-file mark. This method may be redefined by a derived class to define additional processing at the end of the input, but the redefined version should always call theHTMLParser base class method close().
强制将所有的缓冲数据按跟在结束标记的数据一样进行处理。该方法可以通过派生类定义对输入结尾的额外处理来进行重定义，但是重定义的版本应该总是调用HTMLParser基类方法close()
HTMLParser.getpos()
Return current line number and offset.
返回当前行数和位移值。
HTMLParser.get_starttag_text()
Return the text of the most recently opened start tag. This should not normally be needed for structured processing, but may be useful in dealing with HTML “as deployed” or for re-generating input with minimal changes (whitespace between attributes can be preserved, etc.).
返回文本最近的开放标签。
HTMLParser.handle_starttag(tag, attrs)
This method is called to handle the start of a tag. It is intended to be overridden by a derived class; the base class implementation does nothing.
The tag argument is the name of the tag converted to lower case. The attrs argument is a list of (name, value) pairs containing the attributes found inside the tag’s <> brackets. The name will be translated to lower case, and quotes in the value have been removed, and character and entity references have been replaced. For instance, for the tag <A HREF="http://www.cwi.nl/">, this method would be called as handle_starttag('a', [('href', 'http://www.cwi.nl/')]).
该方法用来处理开始标签。其目的是被派生类重写；基类什么也不实现。tag参数是转换成小写的标签名称。attrs参数是一个(name，value)对包含了在标签<>中得属性。name将会转换成小写，并且value中得引号会被引出，并且字符串和实体引用将会被替代。
例如，对于标签<A HREF="http://www.cwi.nl/">，该方法将会调用为handle_starttag('a', [('href', 'http://www.cwi.nl/')])
Changed in version 2.6: All entity references from htmlentitydefs are now replaced in the attribute values.
2.6的版本变动：所有来自htmlentitydefs的实体引用现在在属性值中被替代了。
HTMLParser.handle_startendtag(tag, attrs)
Similar to handle_starttag(), but called when the parser encounters an XHTML-style empty tag (<a .../>). This method may be overridden by subclasses which require this particular lexical information; the default implementation simple calls handle_starttag() and handle_endtag().
类似于handle_starttag()，不过用来处理遇到XHTML风格的空标签(<a .../>)。
HTMLParser.handle_endtag(tag)
This method is called to handle the end tag of an element. It is intended to be overridden by a derived class; the base class implementation does nothing. The tag argument is the name of the tag converted to lower case.
该方法用来处理元素的结束标签。
HTMLParser.handle_data(data)
This method is called to process arbitrary data. It is intended to be overridden by a derived class; the base class implementation does nothing.
该方法用来处理任意的数据。
HTMLParser.handle_charref(name)
This method is called to process a character reference of the form &#ref;. It is intended to be overridden by a derived class; the base class implementation does nothing.
该方法用来处理&#ref;形式的字符引用。
HTMLParser.handle_entityref(name)
This method is called to process a general entity reference of the form &name; where name is an general entity reference. It is intended to be overridden by a derived class; the base class implementation does nothing.
该方法用来处理形式&name;的一般实体引用，参数name是一般的实体引用。
HTMLParser.handle_comment(data)
This method is called when a comment is encountered. The comment argument is a string containing the text between the --and -- delimiters, but not the delimiters themselves. For example, the comment <!--text--> will cause this method to be called with the argument 'text'. It is intended to be overridden by a derived class; the base class implementation does nothing.
该方法用来处理遇到的评论。
HTMLParser.handle_decl(decl)
Method called when an SGML doctype declaration is read by the parser. The decl parameter will be the entire contents of the declaration inside the <!...> markup. It is intended to be overridden by a derived class; the base class implementation does nothing.
当解析器读到一个SGML的doctype声明该方法被调用。
HTMLParser.unknown_decl(data)
Method called when an unrecognized SGML declaration is read by the parser. The data parameter will be the entire contents of the declaration inside the <!...> markup. It is sometimes useful to be overridden by a derived class; the base class implementation throws an HTMLParseError.
 当解析器读到一个未被识别SGML声明时将调用该方法。
HTMLParser.handle_pi(data)
Method called when a processing instruction is encountered. The data parameter will contain the entire processing instruction. For example, for the processing instruction <?proc color='red'>, this method would be called as handle_pi("proccolor='red'"). It is intended to be overridden by a derived class; the base class implementation does nothing.
当遇到一个处理指令时将调用该方法。
Note     The HTMLParser class uses the SGML syntactic rules for processing instructions. An XHTML processing instruction using the trailing '?' will cause the '?' to be included in data.
实际运用
以网络爬虫的抓取url为例，我们需要抓取网易首页上的所有链接。首先得介绍一点HTML的知识，以下内容来自于w3cshool
什么是HTML 超链接
超链接可以是一个字，一个词，或者一组词，也可以是一幅图像，您可以点击这些内容来跳转到新的文档或者当前文档中的某个部分。
当您把鼠标指针移动到网页中的某个链接上时，箭头会变为一只小手。
我们通过使用 <a> 标签在 HTML 中创建链接。
有两种使用 <a> 标签的方式：
通过使用 href 属性 - 创建指向另一个文档的链接
通过使用 name 属性 - 创建文档内的书签
HTML 链接语法
链接的 HTML 代码很简单。它类似这样：
<a href="url">Link text</a>
href 属性规定链接的目标。
开始标签和结束标签之间的文字被作为超级链接来显示。
实例
<a href="http://www.w3school.com.cn/">Visit W3School</a>
编写代码
      从上面我们得知链接在起始标签<a>中，href属性指向我们需要解析的链接。那么重写handle_startag()方法来实现这个目的。
 
# -*- coding: utf-8 -*-
"""
Created on Tue Aug 30 09:46:45 2011

@author: Nupta
"""
import urllib2
import HTMLParser

class MyParser(HTMLParser.HTMLParser):
      
    def handle_starttag(self, tag, attrs):
        if tag == 'a':
            for name,value in attrs:
                if name == 'href' and value.startswith('http'):
                    print value

if __name__ == '__main__':
    url=raw_input(u'输入地址：'.encode('cp936'))
    f=urllib2.urlopen(url).read()
    my=MyParser()
    try:
        my.feed(content)
    except HTMLParser.HTMLParseError,e:
        print e
问题分析
      输出地链接很多，先省略大部分，注意看最后一行：
asp?bianhao=0102000102300012">http://www.hd315.gov.cn/beian/view.asp?bianhao=0102000102300012
http://www.itrust.org.cn/yz/pjwx.asp?wm=2012043533
http://www.bj.cyberpolice.cn/index.htm
malformed start tag, at line 3339, column 44
 
      在读取的html文件中，第3339行的第44列读到一个有缺陷的开始标签，发生HTMLParseError异常。从给出的信息来看也就是html文件中的第3338行的43个元素。因为前面使用的是read()方法，这里我们需要使用readlines()把html文件读入一个列表中。
    print f[3338][34:67]
      看看结果就明白为什么了：
<a href=\'http://mail.163.com/\'
      这两个转义字符导致了解析器的解析异常，要是不知道如何判断html代码的正确性，请点击W3C的传送门，输入代码即可获得分析结果：
   Line 1, Column 9: an attribute value must be a literal unless it contains only name characters
 <a href=\'http://mail.163.com/\'
You have used a character that is not considered a "name character" in an attribute value. Which characters are considered "name characters" varies between the different document types, but a good rule of thumb is that unless the value contains only lower or upper case letters in the range a-z you must put quotation marks around the value. In fact, unless you have extremefile size requirements it is a very very good idea to always put quote marks around your attribute values. It is never wrong to do so, and very often it is absolutely necessary.

 
      从下面126邮箱的登录页面的检测结果可以看到，目前HTML要符合XHTML规范仍有很长的路要走。

 

作者：lmh12506