欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  IT编程

Python:使用Scrapy框架的ImagesPipeline下载图片如何保持原图片名称呢?

程序员文章站 2022-09-14 20:07:38
默认情况下,使用imagepipeline下载图片的时候,图片名称是以图片url的sha1值进行保存的。 如: 图片url:https://www.example.c...
默认情况下,使用imagepipeline下载图片的时候,图片名称是以图片url的sha1值进行保存的。
如:
图片url:https://www.example.com/image.jpg
sha1结果:3afec3b4765f8f0a07b78f98c07b83f013567a0a
则图片名称:3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg
但是,我想要以原来的图片名称进行保存,比如上面例子中的图片保存到本地的话,图片名称就应该是:image.jpg

*上说是可以重写image_key函数,不过我试了下,结果发现不行,重写的image_key函数没被调用。

后面查看了下imagepipeline的:

class imagespipeline(filespipeline):
    """abstract pipeline that implement the image thumbnail generation logic

    """

    media_name = 'image'
    min_width = 0
    min_height = 0
    thumbs = {}
    default_images_urls_field = 'image_urls'
    default_images_result_field = 'images'

    @classmethod
    def from_settings(cls, settings):
        cls.min_width = settings.getint('images_min_width', 0)
        cls.min_height = settings.getint('images_min_height', 0)
        cls.expires = settings.getint('images_expires', 90)
        cls.thumbs = settings.get('images_thumbs', {})
        s3store = cls.store_schemes['s3']
        s3store.aws_access_key_id = settings['aws_access_key_id']
        s3store.aws_secret_access_key = settings['aws_secret_access_key']

        cls.images_urls_field = settings.get('images_urls_field', cls.default_images_urls_field)
        cls.images_result_field = settings.get('images_result_field', cls.default_images_result_field)
        store_uri = settings['images_store']
        return cls(store_uri)

    def file_downloaded(self, response, request, info):
        return self.image_downloaded(response, request, info)

    def image_downloaded(self, response, request, info):
        checksum = none
        for path, image, buf in self.get_images(response, request, info):
            if checksum is none:
                buf.seek(0)
                checksum = md5sum(buf)
            width, height = image.size
            self.store.persist_file(
                path, buf, info,
                meta={'width': width, 'height': height},
                headers={'content-type': 'image/jpeg'})
        return checksum

    def get_images(self, response, request, info):
        path = self.file_path(request, response=response, info=info)
        orig_image = image.open(stringio(response.body))

        width, height = orig_image.size
        if width 
其中,有这么一句话:
imagespipeline.image_key(url) and file_key(url) methods are deprecated, please use file_path(request, response=none, info=none) instead
也就是说,在最新版本的scrapy中(0.22.2),使用file_path代替image_key函数。

因此,我在自定义的imagepipeline类中,重写了file_path函数,代码如下:

__author__ = 'fly'
#coding:utf-8
from scrapy.contrib.pipeline.images import imagespipeline
from scrapy.http import request
from scrapy.exceptions import dropitem

class myimagespipeline(imagespipeline):
    def file_path(self, request, response=none, info=none):
        image_guid = request.url.split('/')[-1]
        return 'full/%s' % (image_guid)

    def get_media_requests(self, item, info):
        for image_url in item['image_urls']:
            yield request(image_url)

    def item_completed(self, results, item, info):
        image_paths = [x['path'] for ok, x in results if ok]
        if not image_paths:
            raise dropitem("item contains no images")
        return item
以上代码主要返回原图片名称+图片后缀。

作者:曾是土木人(https://blog.csdn.net/php_fly)

原文地址:https://blog.csdn.net/php_fly/article/details/19688595