Python:使用Scrapy框架的ImagesPipeline下载图片如何保持原图片名称呢?
程序员文章站
2022-09-14 20:07:38
默认情况下,使用imagepipeline下载图片的时候,图片名称是以图片url的sha1值进行保存的。
如:
图片url:https://www.example.c...
默认情况下,使用imagepipeline下载图片的时候,图片名称是以图片url的sha1值进行保存的。
如:
图片url:
sha1结果:
则图片名称:
但是,我想要以原来的图片名称进行保存,比如上面例子中的图片保存到本地的话,图片名称就应该是:
imagespipeline.image_key(url) and file_key(url) methods are deprecated, please use file_path(request, response=none, info=none) instead
也就是说,在最新版本的scrapy中(0.22.2),使用file_path代替image_key函数。
如:
图片url:
https://www.example.com/image.jpg
sha1结果:
3afec3b4765f8f0a07b78f98c07b83f013567a0a
则图片名称:
3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg
但是,我想要以原来的图片名称进行保存,比如上面例子中的图片保存到本地的话,图片名称就应该是:
image.jpg
*上说是可以重写image_key函数,不过我试了下,结果发现不行,重写的image_key函数没被调用。
后面查看了下imagepipeline的:
class imagespipeline(filespipeline): """abstract pipeline that implement the image thumbnail generation logic """ media_name = 'image' min_width = 0 min_height = 0 thumbs = {} default_images_urls_field = 'image_urls' default_images_result_field = 'images' @classmethod def from_settings(cls, settings): cls.min_width = settings.getint('images_min_width', 0) cls.min_height = settings.getint('images_min_height', 0) cls.expires = settings.getint('images_expires', 90) cls.thumbs = settings.get('images_thumbs', {}) s3store = cls.store_schemes['s3'] s3store.aws_access_key_id = settings['aws_access_key_id'] s3store.aws_secret_access_key = settings['aws_secret_access_key'] cls.images_urls_field = settings.get('images_urls_field', cls.default_images_urls_field) cls.images_result_field = settings.get('images_result_field', cls.default_images_result_field) store_uri = settings['images_store'] return cls(store_uri) def file_downloaded(self, response, request, info): return self.image_downloaded(response, request, info) def image_downloaded(self, response, request, info): checksum = none for path, image, buf in self.get_images(response, request, info): if checksum is none: buf.seek(0) checksum = md5sum(buf) width, height = image.size self.store.persist_file( path, buf, info, meta={'width': width, 'height': height}, headers={'content-type': 'image/jpeg'}) return checksum def get_images(self, response, request, info): path = self.file_path(request, response=response, info=info) orig_image = image.open(stringio(response.body)) width, height = orig_image.size if width其中,有这么一句话:
imagespipeline.image_key(url) and file_key(url) methods are deprecated, please use file_path(request, response=none, info=none) instead
也就是说,在最新版本的scrapy中(0.22.2),使用file_path代替image_key函数。
因此,我在自定义的imagepipeline类中,重写了file_path函数,代码如下:
__author__ = 'fly' #coding:utf-8 from scrapy.contrib.pipeline.images import imagespipeline from scrapy.http import request from scrapy.exceptions import dropitem class myimagespipeline(imagespipeline): def file_path(self, request, response=none, info=none): image_guid = request.url.split('/')[-1] return 'full/%s' % (image_guid) def get_media_requests(self, item, info): for image_url in item['image_urls']: yield request(image_url) def item_completed(self, results, item, info): image_paths = [x['path'] for ok, x in results if ok] if not image_paths: raise dropitem("item contains no images") return item以上代码主要返回原图片名称+图片后缀。
作者:曾是土木人(https://blog.csdn.net/php_fly)
原文地址:https://blog.csdn.net/php_fly/article/details/19688595