brpop阻塞redis消息队列

不使用brpop的时候其实也可以实现redis的消息队列,只是不是阻塞的,目前已知的问题长时间没有任务的话,consumer会出现假死的状态,使用redis3.0版本,听说使用3.2以上的版本不会出现这种假死的问题,目前没有测试:

 def parse_url(self):
        while True:
            url=self.redis.rpop(self.url_detail_list)    
            if not url:
                continue
            url='https://yaohuo.me/'+url
            headers = {
                'accept': 	"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
                'accept-encoding': "gzip, deflate, br",
                'accept-language': "zh-CN,zh;q=0.9",
                'cache-control': "no-cache",
                'pragma': "no-cache",
                'upgrade-insecure-requests': "1",
                'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36",
            }
            try:
                response = self.s.get(url, headers=headers)
            except Exception:
                print traceback.format_exc()
                continue

使用 redis.rpop不会阻塞,这样一直在死循环的找任务,占用cpu,造成不必要的浪费,使用 redis.brpop在没有任务的时候阻塞,设置timeout=0,即是有list中有任务来到的时候就会自动将任务pop出去,由consumer消费掉.修改后的代码:

 def parse_url(self):
        while True:
            url=self.redis.brpop(self.url_detail_list)[1]
            if not url:
                continue
            url='https://yaohuo.me/'+url
            headers = {
                'accept': 	"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
                'accept-encoding': "gzip, deflate, br",
                'accept-language': "zh-CN,zh;q=0.9",
                'cache-control': "no-cache",
                'pragma': "no-cache",
                'upgrade-insecure-requests': "1",
                'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36",
            }
            try:
                response = self.s.get(url, headers=headers)
            except Exception:
                print traceback.format_exc()
                continue```