Queue

queue队列

queue is especially useful in threaded programming when information must be exchanged safely between multiple threads.

class queue.Queue(maxsize=0) #先入先出 

class queue.LifoQueue(maxsize=0) #last in fisrt out 

class queue.PriorityQueue(maxsize=0) #存储数据时可设置优先级的队列

Constructor for a priority queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.

The lowest valued entries are retrieved first (the lowest valued entry is the one returned by sorted(list(entries))[0]). A typical pattern for entries is a tuple in the form: (priority_number, data).

exception queue.Empty

Exception raised when non-blocking get() (or get_nowait()) is called on a Queue object which is empty.

exception queue.Full

Exception raised when non-blocking put() (or put_nowait()) is called on a Queue object which is full.

Queue.qsize()

Queue.empty() #return True if empty  

Queue.full() # return True if full 

Queue.put(item, block=True, timeout=None)

Put item into the queue. If optional args block is true and timeout is None (the default), block if necessary until a free slot is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Full exception if no free slot was available within that time. Otherwise (block is false), put an item on the queue if a free slot is immediately available, else raise the Full exception (timeout is ignored in that case).

Queue.put_nowait(item)

Equivalent to put(item, False).

Queue.get(block=True, timeout=None)

Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Empty exception if no item was available within that time. Otherwise (block is false), return an item if one is immediately available, else raise the Empty exception (timeout is ignored in that case).

Queue.get_nowait()

Equivalent to get(False).

Two methods are offered to support tracking whether enqueued tasks have been fully processed by daemon consumer threads.

Queue.task_done()

Indicate that a formerly enqueued task is complete. Used by queue consumer threads. For each get() used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.

If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put() into the queue).

Raises a ValueError if called more times than there were items placed in the queue.

Queue.join() block直到queue被消费完毕

生产者消费者模型

在并发编程中使用生产者和消费者模式能够解决绝大多数并发问题。该模式通过平衡生产线程和消费线程的工作能力来提高程序的整体处理数据的速度。

为什么要使用生产者和消费者模式

在线程世界里，生产者就是生产数据的线程，消费者就是消费数据的线程。在多线程开发当中，如果生产者处理速度很快，而消费者处理速度很慢，那么生产者就必须等待消费者处理完，才能继续生产数据。同样的道理，如果消费者的处理能力大于生产者，那么消费者就必须等待生产者。为了解决这个问题于是引入了生产者和消费者模式。

什么是生产者消费者模式

生产者消费者模式是通过一个容器来解决生产者和消费者的强耦合问题。生产者和消费者彼此之间不直接通讯，而通过阻塞队列来进行通讯，所以生产者生产完数据之后不用等待消费者处理，直接扔给阻塞队列，消费者不找生产者要数据，而是直接从阻塞队列里取，阻塞队列就相当于一个缓冲区，平衡了生产者和消费者的处理能力。

队列的一些应用

>>>import queue
>>>q = queue.Queue()   #可以加maxsize参数，如加入3，表示最多只能添加3个
>>>q.put("q1")              #有block=True和timeout=None两个参数，如果设置了block，则当队列满，会抛出一串，get()函数同put一样
>>>q.put("q2")
>>>q.put("q3")
>>>q.get()
>>>'q1'
>>>q.get()
>>>'q2'
>>>q.get()
>>>'q3'
>>>q.get()    #此时队列中已经没有数据了，此时我们再get，就阻塞了


import queue
>>>q = queue.Queue(3)
>>>q.put("q3",timeout = 3)
>>>q.put("q3",timeout = 3)
>>>q.put("q3",timeout = 3)
>>>q.put("q3",timeout = 3)    #等待3秒，列表还是满的话就抛出异常
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\python35\lib\queue.py", line 141, in put
    raise Full

import queue
>>>q = queue.Queue(3)
>>>q.put("q3",block = False)
>>>q.put("q3",block = False)
>>>q.put("q3",block = False)
>>>q.put("q3",block = False)       #如果列表满，就直接抛出异常，不阻塞
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\python35\lib\queue.py", line 130, in put
    raise Full
queue.Full

　　

两个生产者消费者的例子

# import queue
#
# q = queue.Queue()
#
# def producer():
#     for i in range(10):
#         q.put("骨头 %s"%i)
#
#     print("开始等待所有的骨头被取走...")
#     q.join()                        #阻塞调用线程，直到队列中所有任务都被处理掉
#     print("所有的骨头被取完了...")
#
# def consumer(n):
#
#     while q.qsize():
#         print("%s 取到"%n,q.get())
#         q.task_done()    #意味着之前入队的一个任务已经完成。由队列的消费者线程调用。每一个get()调用得到一个任务，接下来的task_done()调用告诉队列该任务已经处理完毕。
#
# t =threading.Thread(target=producer)
# c = threading.Thread(target=consumer,args=("lll",))
# t.start()
# c.start()


import time,random
import queue,threading
q = queue.Queue()
def Producer(name):
  count = 0
  while count <20:
    time.sleep(random.randrange(3))
    q.put(count)
    print('Producer %s has produced %s baozi..' %(name, count))
    count +=1
def Consumer(name):
  count = 0
  while count <20:
    time.sleep(random.randrange(4))
    if not q.empty():
        data = q.get()
        print(data)
        print('\033[32;1mConsumer %s has eat %s baozi...\033[0m' %(name, data))
    else:
        print("-----no baozi anymore----")
    count +=1
p1 = threading.Thread(target=Producer, args=('A',))
c1 = threading.Thread(target=Consumer, args=('B',))
p1.start()
c1.start()