前言
YoloV5中 loss由正样本和负样本两部分 loss组成，负样本对应着图像的背景，如果负样本远多于正样本，则负样本会淹没正样本的损失，从而降低网络收敛的效率与检测精度。这就是目标检测中常见的 正负样本不均衡问题，解决方案之一是增加正样本数。
Yolo anchor_based 系列使用的 loss公式如下：
公式中：
S S S： S × S S×S S ×S 个网格；
B B B：每个网格产生 B B B 个候选框 anchor box；
1 i , j o b j 1_{i,j}^{obj}1 i ,j o b j ： 如果在 i , j i,j i ,j 处的 box有目标（正样本），其值为1，否则为0；
1 i , j n o o b j 1_{i,j}^{noobj}1 i ,j n o o b j ： 如果在 i , j i,j i ,j 处的 box没有目标（负样本），其值为1，否则为0；
l i , j b o x l^{box}_{i,j}l i ,j b o x : 在 i , j i,j i ,j 处的 box损失函数；
l i , j c l s l^{cls}_{i,j}l i ,j c l s :在 i , j i,j i ,j 处的 cls 损失函数；
l i , j o b j l^{obj}_{i,j}l i ,j o b j :在 i , j i,j i ,j 处的 obj损失函数
; 正样本采样
Yolov5算法使用如下3种方式增加正样本个数：
(1) 跨 anchor 预测
假设一个 GT框落在了某个预测分支的某个网格内，该网格具有3种不同大小 anchor，若 GT可以和这3种 anchor中的多种 anchor匹配，则这些匹配的 anchor都可以来预测该 GT框，即一个 GT框可以使用多种 anchor来预测。
具体方法：
不同于 IOU匹配， yolov5采用基于宽高比例的匹配策略， GT的宽高与 anchors的宽高对应相除得到 ratio1， anchors的宽高与 GT的宽高对应相除得到 ratio2，取 ratio1和 ratio2的最大值作为最后的宽高比，该宽高比和设定阈值（默认为4）比较，小于设定阈值的 anchor则为匹配到的 anchor。
计算例子：
anchor_boxes=torch.tensor([[1.25000, 1.62500],[2.00000, 3.75000],[4.12500, 2.87500]])
gt_box=torch.tensor([5,4])
ratio1=gt_box/anchor_boxes
ratio2=anchor_boxes/gt_box
ratio=torch.max(ratio1, ratio2).max(1)[0]
print(ratio)
anchor_t=4
res=ratio<anchor_t
print(res)
输出：
tensor([4.0000, 2.5000, 1.3913])
tensor([False, True, True])
与 GT相匹配的的 anchor为 anchor 2和 anchor3。
图1 匹配 anchor
(2) 跨 grid 预测
假设一个 GT框落在了某个预测分支的某个网格内，则该网格有左、上、右、下4个邻域网格，根据 GT框的中心位置，将最近的2个邻域网格也作为预测网格，也即一个 GT框可以由3个网格来预测。
计算例子：
GT box中心点处于 grid1中， grid1被选中，为了增加增样本， grid1的上下左右 grid为候选网格，因为 GT中心点更靠近 grid2和 grid3， grid2和 grid3也作为匹配到的网格，根据上步的 anchor匹配结果， GT与 anchor2、 anchor3相匹配，因此 GT在当前层匹配到的正样本有6个，分别为： grid1_anchor2， grid1_anchor3， grid2_anchor2， grid2_anchor3， grid3_anchor2， grid3_anchor3。
图2 匹配 grid
; (3) 跨分支预测
假设一个 GT框可以和2个甚至3个预测分支上的 anchor匹配，则这2个或3个预测分支都可以预测该 GT框，即一个 GT框可以由多个预测分支来预测，重复 anchor匹配和 grid匹配的步骤，可以得到某个 GT匹配到的所有正样本。
debug 代码
上述正样本采样过程在 yolov5通过函数 build_targets实现，为了方便理解上述代码，对每一步进行debug，具体如下：
def build_targets(self, p, targets):
na, nt = self.na, targets.shape[0]
tcls, tbox, indices, anch = [], [], [], []
gain = torch.ones(7, device=targets.device)
ai = torch.arange(na, device=targets.device).float().view(na, 1).repeat(1, nt)
targets = torch.cat((targets.repeat(na, 1, 1), ai[:, :, None]), 2)
g = 0.5
off = torch.tensor([[0, 0],
[1, 0], [0, 1], [-1, 0], [0, -1],
], device=targets.device).float() * g
for i in range(self.nl):
anchors = self.anchors[i]
gain[2:6] = torch.tensor(p[i].shape)[[3, 2, 3, 2]]
t = targets * gain
if nt:
r = t[:, :, 4:6] / anchors[:, None]
j = torch.max(r, 1 / r).max(2)[0] < self.hyp['anchor_t']
t = t[j]
gxy = t[:, 2:4]
gxi = gain[[2, 3]] - gxy
j, k = ((gxy % 1 < g) & (gxy > 1)).T
l, m = ((gxi % 1 < g) & (gxi > 1)).T
j = torch.stack((torch.ones_like(j), j, k, l, m))
t = t.repeat((5, 1, 1))[j]
offsets = (torch.zeros_like(gxy)[None] + off[:, None])[j]
else:
t = targets[0]
offsets = 0
b, c = t[:, :2].long().T
gxy = t[:, 2:4]
gwh = t[:, 4:6]
gij = (gxy - offsets).long()
gi, gj = gij.T
a = t[:, 6].long()
indices.append((b, a, gj.clamp_(0, gain[3] - 1), gi.clamp_(0, gain[2] - 1)))
tbox.append(torch.cat((gxy - gij, gwh), 1))
anch.append(anchors[a])
tcls.append(c)
return tcls, tbox, indices, anch
输入： targets: shape (7，6)，其中7是一个 batch的 bbox数量，6代表 (image_id, class, x, y, w, h)：
tensor([[[0.00000, 2.00000, 0.23471, 0.63425, 0.30464, 0.16110],
[0.00000, 3.00000, 0.08736, 0.64150, 0.11682, 0.21073],
[0.00000, 3.00000, 0.86652, 0.63158, 0.10384, 0.19088],
[0.00000, 2.00000, 0.69091, 0.65983, 0.21684, 0.24738],
[0.00000, 2.00000, 0.95464, 0.64647, 0.09072, 0.13514],
[0.00000, 2.00000, 0.69473, 0.98720, 0.17866, 0.02560],
[0.00000, 3.00000, 0.87034, 0.98758, 0.13743, 0.02484]]], device='cuda:0')
前置处理
ai = torch.arange(na, device=targets.device).float().view(na, 1).repeat(1, nt)
targets = torch.cat((targets.repeat(na, 1, 1), ai[:, :, None]), 2)
g = 0.5
off = torch.tensor([[0, 0],[1, 0], [0, 1], [-1, 0], [0, -1]],
device=targets.device).float() * g
将 target复制3份， shape变成(3,7,7)，7代表 (image_id, class, x, y, w, h, anchor_id)，复制的目的是为了进行 anchor匹配，对应3个 anchor:
tensor([[[0.00000, 2.00000, 0.23471, 0.63425, 0.30464, 0.16110, 0.00000],
[0.00000, 3.00000, 0.08736, 0.64150, 0.11682, 0.21073, 0.00000],
[0.00000, 3.00000, 0.86652, 0.63158, 0.10384, 0.19088, 0.00000],
[0.00000, 2.00000, 0.69091, 0.65983, 0.21684, 0.24738, 0.00000],
[0.00000, 2.00000, 0.95464, 0.64647, 0.09072, 0.13514, 0.00000],
[0.00000, 2.00000, 0.69473, 0.98720, 0.17866, 0.02560, 0.00000],
[0.00000, 3.00000, 0.87034, 0.98758, 0.13743, 0.02484, 0.00000]],
[[0.00000, 2.00000, 0.23471, 0.63425, 0.30464, 0.16110, 1.00000],
[0.00000, 3.00000, 0.08736, 0.64150, 0.11682, 0.21073, 1.00000],
[0.00000, 3.00000, 0.86652, 0.63158, 0.10384, 0.19088, 1.00000],
[0.00000, 2.00000, 0.69091, 0.65983, 0.21684, 0.24738, 1.00000],
[0.00000, 2.00000, 0.95464, 0.64647, 0.09072, 0.13514, 1.00000],
[0.00000, 2.00000, 0.69473, 0.98720, 0.17866, 0.02560, 1.00000],
[0.00000, 3.00000, 0.87034, 0.98758, 0.13743, 0.02484, 1.00000]],
[[0.00000, 2.00000, 0.23471, 0.63425, 0.30464, 0.16110, 2.00000],
[0.00000, 3.00000, 0.08736, 0.64150, 0.11682, 0.21073, 2.00000],
[0.00000, 3.00000, 0.86652, 0.63158, 0.10384, 0.19088, 2.00000],
[0.00000, 2.00000, 0.69091, 0.65983, 0.21684, 0.24738, 2.00000],
[0.00000, 2.00000, 0.95464, 0.64647, 0.09072, 0.13514, 2.00000],
[0.00000, 2.00000, 0.69473, 0.98720, 0.17866, 0.02560, 2.00000],
[0.00000, 3.00000, 0.87034, 0.98758, 0.13743, 0.02484, 2.00000]]], device='cuda:0')
tensor([[ 0.00000, 0.00000],
[ 0.50000, 0.00000],
[ 0.00000, 0.50000],
[-0.50000, 0.00000],
[ 0.00000, -0.50000]], device='cuda:0')
target 坐标映射到当前层的尺度
输入的target坐标值为归一化的值，当前层尺寸为80，target坐标值乘以80则可将target坐标值映射到当前层的尺度：
anchors = self.anchors[i]
gain = torch.ones(7, device=targets.device)
gain[2:6] = torch.tensor(p[i].shape)[[3, 2, 3, 2]]
t = targets * gain
tensor([[1.25000, 1.62500],
[2.00000, 3.75000],
[4.12500, 2.87500]], device='cuda:0')
tensor([ 1., 1., 80., 80., 80., 80., 1.], device='cuda:0')
tensor([[[ 0.00000, 2.00000, 18.77713, 50.73999, 24.37135, 12.88811, 0.00000],
[ 0.00000, 3.00000, 6.98848, 51.32027, 9.34541, 16.85838, 0.00000],
[ 0.00000, 3.00000, 69.32172, 50.52621, 8.30702, 15.27027, 0.00000],
[ 0.00000, 2.00000, 55.27307, 52.78621, 17.34703, 19.79027, 0.00000],
[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 0.00000],
[ 0.00000, 2.00000, 55.57848, 78.97575, 14.29298, 2.04826, 0.00000],
[ 0.00000, 3.00000, 69.62713, 79.00629, 10.99460, 1.98718, 0.00000]],
[[ 0.00000, 2.00000, 18.77713, 50.73999, 24.37135, 12.88811, 1.00000],
[ 0.00000, 3.00000, 6.98848, 51.32027, 9.34541, 16.85838, 1.00000],
[ 0.00000, 3.00000, 69.32172, 50.52621, 8.30702, 15.27027, 1.00000],
[ 0.00000, 2.00000, 55.27307, 52.78621, 17.34703, 19.79027, 1.00000],
[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 1.00000],
[ 0.00000, 2.00000, 55.57848, 78.97575, 14.29298, 2.04826, 1.00000],
[ 0.00000, 3.00000, 69.62713, 79.00629, 10.99460, 1.98718, 1.00000]],
[[ 0.00000, 2.00000, 18.77713, 50.73999, 24.37135, 12.88811, 2.00000],
[ 0.00000, 3.00000, 6.98848, 51.32027, 9.34541, 16.85838, 2.00000],
[ 0.00000, 3.00000, 69.32172, 50.52621, 8.30702, 15.27027, 2.00000],
[ 0.00000, 2.00000, 55.27307, 52.78621, 17.34703, 19.79027, 2.00000],
[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 2.00000],
[ 0.00000, 2.00000, 55.57848, 78.97575, 14.29298, 2.04826, 2.00000],
[ 0.00000, 3.00000, 69.62713, 79.00629, 10.99460, 1.98718, 2.00000]]], device='cuda:0')
匹配 anchor
step1：对每一个 GT框，分别计算它与9种 anchor的宽与宽的比值、高与高的比值；
step2：在宽比值、高比值这2个比值中，取最极端的一个比值，作为 GT框和 anchor的比值，具体实现的伪代码为： max(anchor / GT, GT / anchor )；
step3：得到 GT框和 anchor的比值后，若这个比值小于设定的比值阈值，那么这个 anchor就负责预测 GT框，这个 anchor的预测框就被称为正样本，所有其它的预测框都是负样本。
得到当前层需要检测的 GT以及其对应的 anchor_id。
r = t[:, :, 4:6] / anchors[:, None]
j = torch.max(r, 1 / r).max(2)[0] < self.hyp['anchor_t']
t = t[j]
输出：
tensor([[[19.49708, 7.93114],
[ 7.47632, 10.37439],
[ 6.64562, 9.39709],
[13.87762, 12.17863],
[ 5.80609, 6.65314],
[11.43438, 1.26047],
[ 8.79568, 1.22288]],
[[12.18568, 3.43683],
[ 4.67270, 4.49557],
[ 4.15351, 4.07207],
[ 8.67351, 5.27741],
[ 3.62881, 2.88303],
[ 7.14649, 0.54620],
[ 5.49730, 0.52992]],
[[ 5.90821, 4.48282],
[ 2.26555, 5.86378],
[ 2.01382, 5.31140],
[ 4.20534, 6.88357],
[ 1.75942, 3.76047],
[ 3.46496, 0.71244],
[ 2.66536, 0.69119]]], device='cuda:0')
tensor([[False, False, False, False, False, False, False],
[False, False, False, False, True, False, False],
[False, False, False, False, True, True, True]], device='cuda:0')
tensor([[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 1.00000],
[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 2.00000],
[ 0.00000, 2.00000, 55.57848, 78.97575, 14.29298, 2.04826, 2.00000],
[ 0.00000, 3.00000, 69.62713, 79.00629, 10.99460, 1.98718, 2.00000]], device='cuda:0')
匹配 grid
为了扩展正样本数量，将上下左右的 gird作为候选 grid，更靠近 GT 所在的 grid的则为匹配到的 grid。
gxy = t[:, 2:4]
gxi = gain[[2, 3]] - gxy
j, k = ((gxy % 1 < g) & (gxy > 1)).T
l, m = ((gxi % 1 < g) & (gxi > 1)).T
j = torch.stack((torch.ones_like(j), j, k, l, m))
t = t.repeat((5, 1, 1))[j]
offsets = (torch.zeros_like(gxy)[None] + off[:, None])[j]
输出：
tensor([[76.37107, 51.71729],
[76.37107, 51.71729],
[55.57848, 78.97575],
[69.62713, 79.00629]], device='cuda:0')
tensor([[ 3.62893, 28.28271],
[ 3.62893, 28.28271],
[24.42152, 1.02425],
[10.37287, 0.99371]], device='cuda:0')
j: tensor([ True, True, False, False], device='cuda:0')
k: tensor([False, False, False, True], device='cuda:0')
l: tensor([False, False, True, True], device='cuda:0')
m: tensor([ True, True, True, False], device='cuda:0')
tensor([[ True, True, True, True],
[ True, True, False, False],
[False, False, False, True],
[False, False, True, True],
[ True, True, True, False]], device='cuda:0')
tensor([[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 1.00000],
[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 2.00000],
[ 0.00000, 2.00000, 55.57848, 78.97575, 14.29298, 2.04826, 2.00000],
[ 0.00000, 3.00000, 69.62713, 79.00629, 10.99460, 1.98718, 2.00000],
[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 1.00000],
[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 2.00000],
[ 0.00000, 3.00000, 69.62713, 79.00629, 10.99460, 1.98718, 2.00000],
[ 0.00000, 2.00000, 55.57848, 78.97575, 14.29298, 2.04826, 2.00000],
[ 0.00000, 3.00000, 69.62713, 79.00629, 10.99460, 1.98718, 2.00000],
[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 1.00000],
[ 0.00000, 2.00000, 76.37107, 51.71729, 7.25761, 10.81135, 2.00000],
[ 0.00000, 2.00000, 55.57848, 78.97575, 14.29298, 2.04826, 2.00000]], device='cuda:0')
tensor([[ 0.00000, 0.00000],
[ 0.00000, 0.00000],
[ 0.00000, 0.00000],
[ 0.00000, 0.00000],
[ 0.50000, 0.00000],
[ 0.50000, 0.00000],
[ 0.00000, 0.50000],
[-0.50000, 0.00000],
[-0.50000, 0.00000],
[ 0.00000, -0.50000],
[ 0.00000, -0.50000],
[ 0.00000, -0.50000]], device='cuda:0')
grid坐标
计算匹配到的 grid左上坐标点：
b, c = t[:, :2].long().T
gxy = t[:, 2:4]
gwh = t[:, 4:6]
gij = (gxy - offsets).long()
gi, gj = gij.T
输出：
tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')
tensor([2, 2, 2, 3, 2, 2, 3, 2, 3, 2, 2, 2], device='cuda:0')
tensor([[76.37107, 51.71729],
[76.37107, 51.71729],
[55.57848, 78.97575],
[69.62713, 79.00629],
[76.37107, 51.71729],
[76.37107, 51.71729],
[69.62713, 79.00629],
[55.57848, 78.97575],
[69.62713, 79.00629],
[76.37107, 51.71729],
[76.37107, 51.71729],
[55.57848, 78.97575]], device='cuda:0')
tensor([[ 7.25761, 10.81135],
[ 7.25761, 10.81135],
[14.29298, 2.04826],
[10.99460, 1.98718],
[ 7.25761, 10.81135],
[ 7.25761, 10.81135],
[10.99460, 1.98718],
[14.29298, 2.04826],
[10.99460, 1.98718],
[ 7.25761, 10.81135],
[ 7.25761, 10.81135],
[14.29298, 2.04826]], device='cuda:0')
tensor([[76, 51],
[76, 51],
[55, 78],
[69, 79],
[75, 51],
[75, 51],
[69, 78],
[56, 78],
[70, 79],
[76, 52],
[76, 52],
[55, 79]], device='cuda:0')
tensor([76, 76, 55, 69, 75, 75, 69, 56, 70, 76, 76, 55], device='cuda:0')
tensor([51, 51, 78, 79, 51, 51, 78, 78, 79, 52, 52, 79], device='cuda:0')
结果放到列表中
a = t[:, 6].long()
indices.append((b, a, gj.clamp_(0, gain[3] - 1), gi.clamp_(0, gain[2] - 1)))
tbox.append(torch.cat((gxy - gij, gwh), 1))
anch.append(anchors[a])
tcls.append(c)
输出：
tensor([1, 2, 2, 2, 1, 2, 2, 2, 2, 1, 2, 2], device='cuda:0')
[(tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0'),
tensor([1, 2, 2, 2, 1, 2, 2, 2, 2, 1, 2, 2], device='cuda:0'),
tensor([51, 51, 78, 79, 51, 51, 78, 78, 79, 52, 52, 79], device='cuda:0'),
tensor([76, 76, 55, 69, 75, 75, 69, 56, 70, 76, 76, 55], device='cuda:0'))]
[tensor([[ 3.71071e-01, 7.17293e-01, 7.25761e+00, 1.08114e+01],
[ 3.71071e-01, 7.17293e-01, 7.25761e+00, 1.08114e+01],
[ 5.78480e-01, 9.75746e-01, 1.42930e+01, 2.04826e+00],
[ 6.27129e-01, 6.28662e-03, 1.09946e+01, 1.98718e+00],
[ 1.37107e+00, 7.17293e-01, 7.25761e+00, 1.08114e+01],
[ 1.37107e+00, 7.17293e-01, 7.25761e+00, 1.08114e+01],
[ 6.27129e-01, 1.00629e+00, 1.09946e+01, 1.98718e+00],
[-4.21520e-01, 9.75746e-01, 1.42930e+01, 2.04826e+00],
[-3.72871e-01, 6.28662e-03, 1.09946e+01, 1.98718e+00],
[ 3.71071e-01, -2.82707e-01, 7.25761e+00, 1.08114e+01],
[ 3.71071e-01, -2.82707e-01, 7.25761e+00, 1.08114e+01],
[ 5.78480e-01, -2.42538e-02, 1.42930e+01, 2.04826e+00]], device='cuda:0')]
[tensor([[2.00000, 3.75000],
[4.12500, 2.87500],
[4.12500, 2.87500],
[4.12500, 2.87500],
[2.00000, 3.75000],
[4.12500, 2.87500],
[4.12500, 2.87500],
[4.12500, 2.87500],
[4.12500, 2.87500],
[2.00000, 3.75000],
[4.12500, 2.87500],
[4.12500, 2.87500]], device='cuda:0')]
[tensor([2, 2, 2, 3, 2, 2, 3, 2, 3, 2, 2, 2], device='cuda:0')]
参考
YOLOV5: https://github.com/ultralytics/yolov5
Original: https://blog.csdn.net/weixin_46142822/article/details/123820031
Author: 大林兄
Title: 目标检测: 一文读懂 YOLOV5 Loss 正样本采样
原创文章受到原创版权保护。转载请注明出处：https://www.johngo689.com/703002/
转载文章受原作者版权保护。转载请注明原作者出处！