SSD 源码分析

http://blog.csdn.net/mydear_11000/article/details/73867041

SSD（SSD: Single Shot MultiBox Detector）是采用单个深度神经网络模型实现目标检测和识别的方法。如图0-1所示，该方法是综合了Faster R-CNN的anchor box和YOLO单个神经网络检测思路（YOLOv2也采用了类似的思路，详见YOLO升级版：YOLOv2和YOLO9000解析），既有Faster R-CNN的准确率又有YOLO的检测速度，可以实现高准确率实时检测。在300*300分辨率，SSD在VOC2007数据集上准确率为74.3%mAP，59FPS；512*512分辨率，SSD获得了超过Fast R-CNN，获得了80%mAP/19fps的结果，如图0-2所示。SSD关键点分为两类：模型结构和训练方法。模型结构包括：多尺度特征图检测网络结构和anchor boxes生成；训练方法包括：ground truth预处理和损失函数。本文解析的是SSD的tensorflow实现源码，来源balancap/SSD-Tensorflow。本文结构如下：

1，多尺度特征图检测网络结构；

2，anchor boxes生成；

3，ground truth预处理；

4，目标函数；

5，总结

图0-1 SSD与MultiBox，Faster R-CNN，YOLO原理（此图来源于作者在eccv2016的PPT）

图0-2 SSD检测速度与精确度。（此图来源于作者在eccv2016的PPT）

1 多尺度特征图检测网络结构

SSD的网络模型如图1-1所示。

图1-1 SSD模型结构。（此图来源于原论文）

模型建立源代码包含于ssd_vgg_300.py中。模型多尺度特征图检测如图1-2所示。模型选择的特征图包括：38×38（block4）,19×19（block7），10×10（block8），5×5（block9），3×3（block10），1×1（block11）。对于每张特征图，生成采用3×3卷积生成 默认框的四个偏移位置和21个类别的置信度。比如block7，默认框（def boxes）数目为6，每个默认框包含4个偏移位置和21个类别置信度（4+21）。因此，block7的最后输出为(19*19)*6*(4+21)。

图1-2 多尺度特征采样（此图来源：知乎专栏）

其中，初始化参数如下：

    """
    Implementation of the SSD VGG-based 300 network.

    The default features layers with 300x300 image input are:
      conv4 ==> 38 x 38
      conv7 ==> 19 x 19
      conv8 ==> 10 x 10
      conv9 ==> 5 x 5
      conv10 ==> 3 x 3
      conv11 ==> 1 x 1
    The default image size used to train this network is 300x300.
    """
    default_params = SSDParams(
        img_shape=(300, 300),#输入尺寸
        num_classes=21,#预测类别20+1=21（20类加背景）
        #获取feature map层
        feat_layers=['block4', 'block7', 'block8', 'block9', 'block10', 'block11'],
        feat_shapes=[(38, 38), (19, 19), (10, 10), (5, 5), (3, 3), (1, 1)],
        # 每个特性层上的anchor大小都不一样， 越靠近输入的层其anchor越小。 
               #  确定第一个与最后一个feature层的anchor大小以后， 处于中间的层的anchor大小则通过线性插值计算而来。例如，假如anchor_size_bounds = [0.2, 0.7]， 有6个feature layer，则每个layer对应的default anchor大小为：[0.2, 0.3, 0.4, 0.5, 0.6, 0.7].
        anchor_size_bounds=[0.15, 0.90],
        #anchor boxes的大小
        anchor_sizes=[(21., 45.),
                      (45., 99.),
                      (99., 153.),
                      (153., 207.),
                      (207., 261.),
                      (261., 315.)],
        #anchor boxes的aspect ratios
        anchor_ratios=[[2, .5],
                       [2, .5, 3, 1./3],
                       [2, .5, 3, 1./3],
                       [2, .5, 3, 1./3],
                       [2, .5],
                       [2, .5]],
        anchor_steps=[8, 16, 32, 64, 100, 300],#anchor的层
        anchor_offset=0.5,#补偿阀值0.5
        normalizations=[20, -1, -1, -1, -1, -1],#该特征层是否正则，大于零即正则；小于零则否
        prior_scaling=[0.1, 0.1, 0.2, 0.2]
        )

建立模型代码如下，作者采用了TensorFlow-Slim（类似于keras的高层库）来建立网络模型，详细内容可以参考TensorFlow-Slim网页。

#建立ssd网络函数
def ssd_net(inputs,
            num_classes=21,
            feat_layers=SSDNet.default_params.feat_layers,
            anchor_sizes=SSDNet.default_params.anchor_sizes,
            anchor_ratios=SSDNet.default_params.anchor_ratios,
            normalizations=SSDNet.default_params.normalizations,
            is_training=True,
            dropout_keep_prob=0.5,
            prediction_fn=slim.softmax,
            reuse=None,
            scope='ssd_300_vgg'):
    """SSD net definition.
    """
    # End_points collect relevant activations for external use.
    #用于收集每一层输出结果
    end_points = {}
    #采用slim建立vgg网络,网络结构参考文章内的结构图
    with tf.variable_scope(scope, 'ssd_300_vgg', [inputs], reuse=reuse):
        # Original VGG-16 blocks.
        net = slim.repeat(inputs, 2, slim.conv2d, 64, [3, 3], scope='conv1')
        end_points['block1'] = net
        net = slim.max_pool2d(net, [2, 2], scope='pool1')
        # Block 2.
        net = slim.repeat(net, 2, slim.conv2d, 128, [3, 3], scope='conv2')
        end_points['block2'] = net
        net = slim.max_pool2d(net, [2, 2], scope='pool2')
        # Block 3.
        net = slim.repeat(net, 3, slim.conv2d, 256, [3, 3], scope='conv3')
        end_points['block3'] = net
        net = slim.max_pool2d(net, [2, 2], scope='pool3')
        # Block 4.
        net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv4')
        end_points['block4'] = net
        net = slim.max_pool2d(net, [2, 2], scope='pool4')
        # Block 5.
        net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv5')
        end_points['block5'] = net
        net = slim.max_pool2d(net, [3, 3], 1, scope='pool5')#max pool

        #外加的SSD层
        # Additional SSD blocks.
        # Block 6: let's dilate the hell out of it!
        #输出shape为19×19×1024
        net = slim.conv2d(net, 1024, [3, 3], rate=6, scope='conv6')
        end_points['block6'] = net
        # Block 7: 1x1 conv. Because the fuck.
        #卷积核为1×1
        net = slim.conv2d(net, 1024, [1, 1], scope='conv7')
        end_points['block7'] = net

        # Block 8/9/10/11: 1x1 and 3x3 convolutions stride 2 (except lasts).
        end_point = 'block8'
        with tf.variable_scope(end_point):
            net = slim.conv2d(net, 256, [1, 1], scope='conv1x1')
            net = slim.conv2d(net, 512, [3, 3], stride=2, scope='conv3x3')
        end_points[end_point] = net
        end_point = 'block9'
        with tf.variable_scope(end_point):
            net = slim.conv2d(net, 128, [1, 1], scope='conv1x1')
            net = slim.conv2d(net, 256, [3, 3], stride=2, scope='conv3x3')
        end_points[end_point] = net
        end_point = 'block10'
        with tf.variable_scope(end_point):
            net = slim.conv2d(net, 128, [1, 1], scope='conv1x1')
            net = slim.conv2d(net, 256, [3, 3], scope='conv3x3', padding='VALID')
        end_points[end_point] = net
        end_point = 'block11'
        with tf.variable_scope(end_point):
            net = slim.conv2d(net, 128, [1, 1], scope='conv1x1')
            net = slim.conv2d(net, 256, [3, 3], scope='conv3x3', padding='VALID')
        end_points[end_point] = net

        # Prediction and localisations layers.
        #预测和定位
        predictions = []
        logits = []
        localisations = []
        for i, layer in enumerate(feat_layers):
            with tf.variable_scope(layer + '_box'):
                #接受特征层的输出，生成类别和位置预测
                p, l = ssd_multibox_layer(end_points[layer],
                                          num_classes,
                                          anchor_sizes[i],
                                          anchor_ratios[i],
                                          normalizations[i])
            #把每一层的预测收集
            predictions.append(prediction_fn(p))#prediction_fn为softmax，预测类别
            logits.append(p)#概率
            localisations.append(l)#预测位置信息

        return predictions, localisations, logits, end_points

2 anchor box生成

对每一张特征图，按照不同的大小（scale） 和长宽比（ratio） 生成生成k个默认框（default boxes），原理图如图2-1所示(此图中，默认框数目k=6，其中5×5的红色点代表特征图，因此：5*5*6 = 150 个boxes)。

每个默认框大小计算公式为：，其中，m为特征图数目，为最底层特征图大小（原论文中值为0.2，代码中为0.15），为最顶层特征图默认框大小（原论文中为0.9,代码中为0.9）。

每个默认框长宽比根据比例值计算，原论文中比例值为，因此，每个默认框的宽为，高为。对于比例为1的默认框，额外添加一个比例为的默认框。最终，每张特征图中的每个点生成6个默认框。每个默认框中心设定为,其中，为第k个特征图尺寸。

图2-1 anchor box生成示意图（此图来源于知乎专栏）

源代码中，默认框生成函数为ssd_anchor_one_layer()，代码如下：

#生成一层的anchor boxes
def ssd_anchor_one_layer(img_shape,#原始图像shape
                         feat_shape,#特征图shape
                         sizes,#预设的box size
                         ratios,#aspect 比例
                         step,#anchor的层
                         offset=0.5,
                         dtype=np.float32):
    """Computer SSD default anchor boxes for one feature layer.

    Determine the relative position grid of the centers, and the relative
    width and height.

    Arguments:
      feat_shape: Feature shape, used for computing relative position grids;
      size: Absolute reference sizes;
      ratios: Ratios to use on these features;
      img_shape: Image shape, used for computing height, width relatively to the
        former;
      offset: Grid offset.

    Return:
      y, x, h, w: Relative x and y grids, and height and width.
    """
    # Compute the position grid: simple way.
    # y, x = np.mgrid[0:feat_shape[0], 0:feat_shape[1]]
    # y = (y.astype(dtype) + offset) / feat_shape[0]
    # x = (x.astype(dtype) + offset) / feat_shape[1]
    # Weird SSD-Caffe computation using steps values...
    
    """
    #测试中，参数如下
    feat_shapes=[(38, 38), (19, 19), (10, 10), (5, 5), (3, 3), (1, 1)]
    anchor_sizes=[(21., 45.),
                      (45., 99.),
                      (99., 153.),
                      (153., 207.),
                      (207., 261.),
                      (261., 315.)]
    anchor_ratios=[[2, .5],
                       [2, .5, 3, 1./3],
                       [2, .5, 3, 1./3],
                       [2, .5, 3, 1./3],
                       [2, .5],
                       [2, .5]]
    anchor_steps=[8, 16, 32, 64, 100, 300]


    offset=0.5

    dtype=np.float32

    feat_shape=feat_shapes[0]
    step=anchor_steps[0]
    """
    #测试中，y和x的shape为（38,38）（38,38）
    #y的值为
    #array([[ 0,  0,  0, ...,  0,  0,  0],
     #  [ 1,  1,  1, ...,  1,  1,  1],
    # [ 2,  2,  2, ...,  2,  2,  2],
    #   ..., 
     #  [35, 35, 35, ..., 35, 35, 35],
    #  [36, 36, 36, ..., 36, 36, 36],
     #  [37, 37, 37, ..., 37, 37, 37]])
    y, x = np.mgrid[0:feat_shape[0], 0:feat_shape[1]]
    #测试中y=(y+0.5)×8/300,x=(x+0.5)×8/300
    y = (y.astype(dtype) + offset) * step / img_shape[0]
    x = (x.astype(dtype) + offset) * step / img_shape[1]

    #扩展维度，维度为（38,38,1）
    # Expand dims to support easy broadcasting.
    y = np.expand_dims(y, axis=-1)
    x = np.expand_dims(x, axis=-1)

    # Compute relative height and width.
    # Tries to follow the original implementation of SSD for the order.
    #数值为2+2
    num_anchors = len(sizes) + len(ratios)
    #shape为（4,）
    h = np.zeros((num_anchors, ), dtype=dtype)
    w = np.zeros((num_anchors, ), dtype=dtype)
    # Add first anchor boxes with ratio=1.
    #测试中，h[0]=21/300,w[0]=21/300?
    h[0] = sizes[0] / img_shape[0]
    w[0] = sizes[0] / img_shape[1]
    di = 1
    if len(sizes) > 1:
        #h[1]=sqrt(21*45)/300
        h[1] = math.sqrt(sizes[0] * sizes[1]) / img_shape[0]
        w[1] = math.sqrt(sizes[0] * sizes[1]) / img_shape[1]
        di += 1
    for i, r in enumerate(ratios):
        h[i+di] = sizes[0] / img_shape[0] / math.sqrt(r)
        w[i+di] = sizes[0] / img_shape[1] * math.sqrt(r)
    #测试中，y和x shape为（38,38,1）
    #h和w的shape为（4,）
    return y, x, h, w

3 ground truth预处理

训练过程中，首先需要将label信息（ground truth box，ground truth category）进行预处理，将其对应到相应的默认框上。根据默认框和ground truth box的jaccard 重叠来寻找对应的默认框。文章中选取了jaccard重叠超过0.5的默认框为正样本，其它为负样本。

源代码ground truth预处理代码位于ssd_common.py文件中，关键代码如下：

#label和bbox编码函数
def tf_ssd_bboxes_encode_layer(labels,#ground truth标签，1D tensor
                               bboxes,#N×4 Tensor（float）
                               anchors_layer,#anchors，为list
                               matching_threshold=0.5,#阀值
                               prior_scaling=[0.1, 0.1, 0.2, 0.2],#缩放
                               dtype=tf.float32):
    """Encode groundtruth labels and bounding boxes using SSD anchors from
    one layer.

    Arguments:
      labels: 1D Tensor(int64) containing groundtruth labels;
      bboxes: Nx4 Tensor(float) with bboxes relative coordinates;
      anchors_layer: Numpy array with layer anchors;
      matching_threshold: Threshold for positive match with groundtruth bboxes;
      prior_scaling: Scaling of encoded coordinates.

    Return:
      (target_labels, target_localizations, target_scores): Target Tensors.
    """
    # Anchors coordinates and volume.
    #获取anchors层
    yref, xref, href, wref = anchors_layer
    ymin = yref - href / 2.
    xmin = xref - wref / 2.
    ymax = yref + href / 2.
    xmax = xref + wref / 2.
    #xmax的shape为((38, 38, 1), (38, 38, 1), (4,), (4,))
(38, 38, 4)
    #体积
    vol_anchors = (xmax - xmin) * (ymax - ymin)

    # Initialize tensors...
    shape = (yref.shape[0], yref.shape[1], href.size)
    feat_labels = tf.zeros(shape, dtype=tf.int64)
    feat_scores = tf.zeros(shape, dtype=dtype)
    #shape为（38,38,4）
    feat_ymin = tf.zeros(shape, dtype=dtype)
    feat_xmin = tf.zeros(shape, dtype=dtype)
    feat_ymax = tf.ones(shape, dtype=dtype)
    feat_xmax = tf.ones(shape, dtype=dtype)

    #计算jaccard重合
    def jaccard_with_anchors(bbox):
        """Compute jaccard score a box and the anchors.
        """
        # Intersection bbox and volume.
        int_ymin = tf.maximum(ymin, bbox[0])
        int_xmin = tf.maximum(xmin, bbox[1])
        int_ymax = tf.minimum(ymax, bbox[2])
        int_xmax = tf.minimum(xmax, bbox[3])
        h = tf.maximum(int_ymax - int_ymin, 0.)
        w = tf.maximum(int_xmax - int_xmin, 0.)

        # Volumes.
        inter_vol = h * w
        union_vol = vol_anchors - inter_vol \
            + (bbox[2] - bbox[0]) * (bbox[3] - bbox[1])
        jaccard = tf.div(inter_vol, union_vol)
        return jaccard
    #条件函数 
    def condition(i, feat_labels, feat_scores,
                  feat_ymin, feat_xmin, feat_ymax, feat_xmax):
        """Condition: check label index.
        """
        #tf.less函数 Returns the truth value of (x < y) element-wise.
        r = tf.less(i, tf.shape(labels))
        return r[0]
    #主体
    def body(i, feat_labels, feat_scores,
             feat_ymin, feat_xmin, feat_ymax, feat_xmax):
        """Body: update feature labels, scores and bboxes.
        Follow the original SSD paper for that purpose:
          - assign values when jaccard > 0.5;
          - only update if beat the score of other bboxes.
        """
        # Jaccard score.
        label = labels[i]
        bbox = bboxes[i]
        scores = jaccard_with_anchors(bbox)#计算jaccard重合值

        # 'Boolean' mask.
        #tf.greater函数返回大于的布尔值
        mask = tf.logical_and(tf.greater(scores, matching_threshold),
                              tf.greater(scores, feat_scores))
        imask = tf.cast(mask, tf.int64)
        fmask = tf.cast(mask, dtype)
        # Update values using mask.
        feat_labels = imask * label + (1 - imask) * feat_labels
        feat_scores = tf.select(mask, scores, feat_scores)

        feat_ymin = fmask * bbox[0] + (1 - fmask) * feat_ymin
        feat_xmin = fmask * bbox[1] + (1 - fmask) * feat_xmin
        feat_ymax = fmask * bbox[2] + (1 - fmask) * feat_ymax
        feat_xmax = fmask * bbox[3] + (1 - fmask) * feat_xmax
        return [i+1, feat_labels, feat_scores,
                feat_ymin, feat_xmin, feat_ymax, feat_xmax]
    # Main loop definition.
    i = 0
    [i, feat_labels, feat_scores,
     feat_ymin, feat_xmin,
     feat_ymax, feat_xmax] = tf.while_loop(condition, body,
                                           [i, feat_labels, feat_scores,
                                            feat_ymin, feat_xmin,
                                            feat_ymax, feat_xmax])
   
    # Transform to center / size.
    #计算补偿后的中心
    feat_cy = (feat_ymax + feat_ymin) / 2.
    feat_cx = (feat_xmax + feat_xmin) / 2.
    feat_h = feat_ymax - feat_ymin
    feat_w = feat_xmax - feat_xmin
    # Encode features.
    feat_cy = (feat_cy - yref) / href / prior_scaling[0]
    feat_cx = (feat_cx - xref) / wref / prior_scaling[1]
    feat_h = tf.log(feat_h / href) / prior_scaling[2]
    feat_w = tf.log(feat_w / wref) / prior_scaling[3]
    # Use SSD ordering: x / y / w / h instead of ours.
    feat_localizations = tf.stack([feat_cx, feat_cy, feat_w, feat_h], axis=-1)
    return feat_labels, feat_localizations, feat_scores


#ground truth编码函数
def tf_ssd_bboxes_encode(labels,#ground truth标签，1D tensor
                         bboxes,#N×4 Tensor（float）
                         anchors,#anchors，为list
                         matching_threshold=0.5,#阀值
                         prior_scaling=[0.1, 0.1, 0.2, 0.2],#缩放
                         dtype=tf.float32,
                         scope='ssd_bboxes_encode'):
    """Encode groundtruth labels and bounding boxes using SSD net anchors.
    Encoding boxes for all feature layers.

    Arguments:
      labels: 1D Tensor(int64) containing groundtruth labels;
      bboxes: Nx4 Tensor(float) with bboxes relative coordinates;
      anchors: List of Numpy array with layer anchors;
      matching_threshold: Threshold for positive match with groundtruth bboxes;
      prior_scaling: Scaling of encoded coordinates.

    Return:
      (target_labels, target_localizations, target_scores):
        Each element is a list of target Tensors.
    """
    with tf.name_scope(scope):
        target_labels = []
        target_localizations = []
        target_scores = []
        for i, anchors_layer in enumerate(anchors):
            with tf.name_scope('bboxes_encode_block_%i' % i):
                #将label和bbox进行编码
                t_labels, t_loc, t_scores = \
                    tf_ssd_bboxes_encode_layer(labels, bboxes, anchors_layer,
                                               matching_threshold, prior_scaling, dtype)
                target_labels.append(t_labels)
                target_localizations.append(t_loc)
                target_scores.append(t_scores)
        return target_labels, target_localizations, target_scores


#编码goundtruth的label和bbox
    def bboxes_encode(self, labels, bboxes, anchors,
                      scope='ssd_bboxes_encode'):
        """Encode labels and bounding boxes.
        """
        return ssd_common.tf_ssd_bboxes_encode(
            labels, bboxes, anchors,
            matching_threshold=0.5,
            prior_scaling=self.params.prior_scaling,
            scope=scope)

4 目标函数

SSD目标函数分为两个部分：对应默认框的位置loss（loc）和类别置信度loss（conf）。定义 为第i个默认框和对应的第j个ground truth box，相应的类别为p。目标函数定义为：

其中，N为匹配的默认框。如果N=0，loss为零。为预测框和ground truth box 的Smooth L1 loss，值通过cross validation设置为1。

定义如下：其中，为预测框，为ground truth。为补偿（regress to offsets）后的默认框（）的中心，为默认框的宽和高。

定义为多累别softmax loss，公式如下：

目标函数定义源码位于ssd_vgg_300.py，注释如下：

# =========================================================================== #
# SSD loss function.
# =========================================================================== #
def ssd_losses(logits, #预测类别
               localisations,#预测位置
               gclasses, #ground truth 类别
               glocalisations, #ground truth 位置
               gscores,#ground truth 分数
               match_threshold=0.5,
               negative_ratio=3.,
               alpha=1.,
               label_smoothing=0.,
               scope='ssd_losses'):
    """Loss functions for training the SSD 300 VGG network.

    This function defines the different loss components of the SSD, and
    adds them to the TF loss collection.

    Arguments:
      logits: (list of) predictions logits Tensors;
      localisations: (list of) localisations Tensors;
      gclasses: (list of) groundtruth labels Tensors;
      glocalisations: (list of) groundtruth localisations Tensors;
      gscores: (list of) groundtruth score Tensors;
    """
    # Some debugging...
    # for i in range(len(gclasses)):
    #     print(localisations[i].get_shape())
    #     print(logits[i].get_shape())
    #     print(gclasses[i].get_shape())
    #     print(glocalisations[i].get_shape())
    #     print()
    with tf.name_scope(scope):
        l_cross = []
        l_loc = []
        for i in range(len(logits)):
            with tf.name_scope('block_%i' % i):
                # Determine weights Tensor.
                pmask = tf.cast(gclasses[i] > 0, logits[i].dtype)
                n_positives = tf.reduce_sum(pmask)#正样本数目
                
                #np.prod函数Return the product of array elements over a given axis
                n_entries = np.prod(gclasses[i].get_shape().as_list())
                # r_positive = n_positives / n_entries
                # Select some random negative entries.
                r_negative = negative_ratio * n_positives / (n_entries - n_positives)#负样本数
                nmask = tf.random_uniform(gclasses[i].get_shape(),
                                          dtype=logits[i].dtype)
                nmask = nmask * (1. - pmask)
                nmask = tf.cast(nmask > 1. - r_negative, logits[i].dtype)

                #cross_entropy loss
                # Add cross-entropy loss.
                with tf.name_scope('cross_entropy'):
                    # Weights Tensor: positive mask + random negative.
                    weights = pmask + nmask
                    loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits[i],
                                                                          gclasses[i])
                    loss = tf.contrib.losses.compute_weighted_loss(loss, weights)
                    l_cross.append(loss)

                #smooth loss
                # Add localization loss: smooth L1, L2, ...
                with tf.name_scope('localization'):
                    # Weights Tensor: positive mask + random negative.
                    weights = alpha * pmask
                    loss = custom_layers.abs_smooth(localisations[i] - glocalisations[i])
                    loss = tf.contrib.losses.compute_weighted_loss(loss, weights)
                    l_loc.append(loss)

        # Total losses in summaries...
        with tf.name_scope('total'):
            tf.summary.scalar('cross_entropy', tf.add_n(l_cross))
            tf.summary.scalar('localization', tf.add_n(l_loc))

5 总结

本文对SSD: Single Shot MultiBox Detector的tensorflow的关键源代码进行了解析。本文采用的源码来自于balancap/SSD-Tensorflow。源码作者写得非常详细，内容较多（其它还包括了图像预处理，多GPU并行训练等许多内容），因此只选取了关键代码进行解析。在看完论文后，再结合关键代码分析，结构就很清晰了。SSD代码实现的关键点为：1，多尺度特征图检测网络结构；2，anchor boxes生成；3，ground truth预处理；4，目标函数。SSD和YOLOv2类似，可以实现高准确率下的实时目标检测，是非常值得研究和改进的目标检测方法。