【爬虫】Scrapy Item

【原文链接】https://doc.scrapy.org/en/latest/topics/items.html

 

Items

The main goal in scraping is to extract structured data from unstructured sources, typically, web pages. Scrapy spiders can return the extracted data as Python dicts. While convenient and familiar, Python dicts lack structure: 很容易犯拼写错误 in a field name 或返回不一致数据, especially in a larger project with many spiders.

为了定义普世的输出数据格式 Scrapy provides the Item class. Item objects are simple 容器 used to 手机爬取到的数据. They provide a dictionary-like API with a convenient syntax for 声明他们可用的 fields.

Various Scrapy 组件使用 Items 提供的额外的信息: exporters look at 声明的 fields 来找出需要 export 的列, 使用 Item fields 元数据可以定制序列化, trackref 追踪 Item 实例来找到内存泄漏 (see Debugging memory leaks with trackref), etc.

Declaring Items

Items are declared using a simple class definition syntax and Field objects. Here is an example:

import scrapy

class Product(scrapy.Item):
    name = scrapy.Field()
    price = scrapy.Field()
    stock = scrapy.Field()
    last_updated = scrapy.Field(serializer=str)

Note

Those familiar with Django will notice that Scrapy Items are declared similar to Django Models, except that Scrapy Items are much simpler as there is no concept of different field types.

Item Fields

Field 对象用来指定每个 field 的元数据. For example, 上面代码中 last_updated field 的序列化函数.

你可以指定每个 field 的任何元数据. Field 对象所能接受的值没有任何限制. 因此, 元数据可使用的键没有说明列表. Field 对象定义的每个键可以被不同的组件使用, 并且只有这些组件 know about it. 你也可以在项目中定义和使用其他任何 Field 键, for your own needs. Field 对象的主要作用是在一个地方提供一种方法来定义所有 field 元数据. Typically, 那些行为表现依赖于每个 field 的组件会使用某些 field 键来配置这些行为表现. 你必须参照文档来查看每个组件都使用哪些元数据键.

It’s important to note that 用来声明 item 的 Field 对象不会一直作为类属性 stay assigned. Instead, 他们可以通过 Item.fields 属性被获取到.

Working with Items

Here are some examples of 普遍的 tasks performed with items, 使用 the Product item declared above. You will notice the API is very similar to the dict API.

Creating items

>>> product = Product(name='Desktop PC', price=1000)
>>> print product
Product(name='Desktop PC', price=1000)

Getting field values

>>> product['name']
Desktop PC
>>> product.get('name')
Desktop PC

>>> product['price']
1000

>>> product['last_updated']
Traceback (most recent call last):
    ...
KeyError: 'last_updated'

>>> product.get('last_updated', 'not set')
not set

>>> product['lala'] # getting unknown field
Traceback (most recent call last):
    ...
KeyError: 'lala'

>>> product.get('lala', 'unknown field')
'unknown field'

>>> 'name' in product  # is name field populated?
True

>>> 'last_updated' in product  # is last_updated populated?
False

>>> 'last_updated' in product.fields  # is last_updated a declared field?
True

>>> 'lala' in product.fields  # is lala a declared field?
False

Setting field values

>>> product['last_updated'] = 'today'
>>> product['last_updated']
today

>>> product['lala'] = 'test' # setting unknown field
Traceback (most recent call last):
    ...
KeyError: 'Product does not support field: lala'

Accessing all populated values

To access all populated values, just use the typical dict API:

>>> product.keys()
['price', 'name']

>>> product.items()
[('price', 1000), ('name', 'Desktop PC')]

Other common tasks

Copying items:

>>> product2 = Product(product)
>>> print product2
Product(name='Desktop PC', price=1000)

>>> product3 = product2.copy()
>>> print product3
Product(name='Desktop PC', price=1000)

Creating dicts from items:

>>> dict(product) # create a dict from all populated values
{'price': 1000, 'name': 'Desktop PC'}

Creating items from dicts:

>>> Product({'name': 'Laptop PC', 'price': 1500})
Product(price=1500, name='Laptop PC')

>>> Product({'name': 'Laptop PC', 'lala': 1500}) # warning: unknown field in dict
Traceback (most recent call last):
    ...
KeyError: 'Product does not support field: lala'

Extending Items

你可以通过声明原本 Item 的一个子类来扩展 Items (来增加更多的 fields 或改变一些 fields 的一些元数据).

For example:

class DiscountedProduct(Product):
    discount_percent = scrapy.Field(serializer=str)
    discount_expiration_date = scrapy.Field()

你也可以通过使用之前的 field 元数据并 change 或 append 值来扩展 field 元数据, like this:

class SpecificProduct(Product):
    name = scrapy.Field(Product.fields['name'], serializer=my_serializer)

这会为 name field 增加或替换 the serializer 元数据键, 但是保留所有之前已经存在的元数据值.

Item objects

class scrapy.item.Item([arg])

根据指定 argument 可选择性地初始化一个新 Item 并返回. Items 复制标准 dict API, 包括它的构造函数. Item 提供的属性中唯一一个增加的是:

fields

这个 Item 的一个包含所有已声明的 fields 的字典, not only those populated. 键和是 field 名称，值是 Item declaration  中使用的 Field 对象.

Field objects

class scrapy.item.Field([arg])

Field 类只是内置的 dict 类的一个别名, 不提供多余的功能或属性. 换句话说, Field 对象是普通的 Python 字典. 会使用另一个基于类属性的类来支持 item declaration syntax.