戴口罩场景下的口罩检测+人脸识别（百度的PaddleHub+OpenCV）

前言

疫情之下，人脸识别有了更具挑战的应用场景。我有一些想法并实践了一下，如果大家有不错的想法欢迎交流。百度公司目前已经开放了口罩场景下的人脸检测和人脸识别API，但只公布了人脸检测的模型。
链接: link.

软件工具

系统：Ubuntu 18
语言：Python 2/3
PaddlePaddle
PaddleHub
OpenCV

思路

由于戴口罩时的人脸下半部被遮挡，而下半部人脸的特征比上半部人脸的多且明显，提取脸部信息的难度会提高很多。整个流程并不复杂，难点在于识别器的算法选择和调节参数

算法模型选择

人脸检测模型：PadddleHub已经提供的预训练模型(pyramidbox_lite_mobile_mask/pyramidbox_lite_server_mask) 链接: link.
切割人脸图像：1.使用OpenCV直接对人脸图像按比例进行切割。2.使用人脸关键点检测，按点位进行切割.链接：link
OpenCV识别器算法：OpenCV 的识别器提供三种算法：Eigen、Fisher、LBPH。官方文档：link 这里我采用的时LBPH算法，它提取的是局部特征并且空间因素对其干扰小。原理介绍参考：link
了解原理对后续调参很重要

代码

库

# coding=utf-8
import numpy as np
import cv2 as cv
import paddlehub as hub

加载模型、初始化

cap = cv.VideoCapture(0)
if not cap.isOpened():
    print("Cannot open camera")
    exit()
#创建检测器
face_recognizer = cv.face.LBPHFaceRecognizer_create(radius = 2, neighbors = 12, grid_x = 12)
# 加载模型
module = hub.Module(name="pyramidbox_lite_server_mask")
face_landmark = hub.Module(name="face_landmark_localization")
face_landmark.set_face_detector_module(hub.Module(name="ultra_light_fast_generic_face_detector_1mb_320"))
face_recognizer.read('face-mask-rec.xml')

人脸检测

def detect_mask(frame):
	input_dict = {"data": [frame]}
	results = module.face_detection(data=input_dict)
	if results != [] :
	    label = results[0]['data']['label']
        x1 = int(results[0]['data']['left'])
	    y1 = int(results[0]['data']['top'])
	    x2 = int(results[0]['data']['right'])
	    y2 = int(results[0]['data']['bottom'])
	    rect = (x1,y1,x2,y2)
	    return rect, label
	else:
	    return (0, 0, 0, 0), None

图像切割

def cut_mask(frame, rect):
	
    """
    #关键点检测，按点位切割
    results = face_landmark.keypoint_detection(images=[frame], visualization=False)
    if results != []:
    	x1 = int(results[0]['data'][0][1][0])
    	y1 = int(results[0]['data'][0][19][1])
    	x2 = int(results[0]['data'][0][15][0])
    	y2 = int(results[0]['data'][0][15][1])
		h = y2-y1
		y1 = int(y1-(h/2))
    	cut_img = frame[y1:y2, x1:x2]
        if x1 > 0 or y1 > 0:
            return cut_img
        else:
            return None
    else:
        return None
        """
    #按比列切割
    x1 = rect[0]
    y1 = rect[1]
    x2 = rect[2]
    y2 = rect[3]
    h = y2-y1
    w = x2-x1
    y2_0 = y2-int((y2-y1)/1.8) 
    cut_img = frame[y1:y2_0, x1:x2]
    if x1 <= 0 or y1<= 0:
        return None
    else:
		print x1,x2
        return cv.cvtColor(cut_img, cv.COLOR_BGR2GRAY)

准备训练数据

def prepare_training_data():
    faces = []
    labels = []
    label = user_list()
    for num in range(0,71):
        rec, frame = cap.read()
        rect ,mask_label = detect_mask(frame)
        cut_img = cut_mask(frame,rect)
        if cut_img is None:
	   		return None, None	
		#cv.imwrite('1/'+str(num)+'.jpg', half_face)
        #将脸添加到脸部列表并添加相应的标签
        faces.append(cut_img)
        labels.append(label)
    return faces, labels

录入标签信息

def user_list():
    label = 0
    while True:
		name = face_recognizer.getLabelInfo(label)
		if len(name) != 0:
	    	print 'label:', label, 'name:', name
		else:
	    	name =  raw_input('input user'+str(label)+'name:')
	    	face_recognizer.setLabelInfo(label, name)
	    	return label
		label += 1
    return None

预测

def predict(frame, rect):
    eye = cut_mask(frame, rect)
    if eye is not None:
        #预测人脸
        results = face_recognizer.predict(eye)
        #print(results[0])
        #置信度阈值
        if results[1] < 180:
	    	label_text = face_recognizer.getLabelInfo(results[0])
        else:
	    	label_text = 'stranger'
        return label_text
    else:
        return 'not whole face'

画框，显示文本

def draw_rectangle(img, rect):
    (x1, y1, x2, y2) = rect
    cv.rectangle(img, (x1, y1), (x2, y2), (128, 128, 0), 2)
def draw_text(img, text, x, y):
    cv.putText(img, text, (x, y), cv.FONT_HERSHEY_COMPLEX, 1, (0, 0, 255), 2)

主循环

while True:
    ret, frame = cap.read()
    rect, label = detect_mask(frame)
    draw_rectangle(frame, rect)
    draw_text(frame, label, rect[0], rect[1])
    if cv.waitKey(3) == ord('s'):
		print("prepare data")
		faces, labels = prepare_training_data()
		print("training..")
        if faces is None:
            print 'error:lose face'
        else:
	    	face_recognizer.update(faces, np.array(labels))
	    	face_recognizer.write('face-mask-rec.xml')
    	    print("SUCCESS!")
    	 #参数信息
		"""GridX = face_recognizer.getGridX()
		GridY = face_recognizer.getGridX()
		Neighbors = face_recognizer.getNeighbors()
		Radius = face_recognizer.getRadius()
		Threshold = face_recognizer.getThreshold()
		print("GridX:",GridX)
		print("GridY:",GridY)
		print("Neighbors:",Neighbors)
		print("Radius:",Radius)
		print("Threshold:",Threshold)"""
    elif cv.waitKey(3) == ord('q'):
		break
    if face_recognizer != 0:
        name_label = predict(frame, rect)
        # 标出预测的名字
        draw_text(frame, name_label, rect[0], rect[3])
    cv.imshow('frame', frame)
# When everything done, release the capture
cap.release()

注意

预训练模型：face-mask-rec.xml。这是之前自己训练好，然后保存的。第一次运行程序需要稍微改动一点以上代码，创建预训练模型并保存。然后才能正常运行以上完整的代码。

效果展示

总结

调参决定效果好坏，我只是简单地调整一下，并没有进行细致的评估和优化，还有很大的提升空间