文章目录
*
– 1. SwinIR 论文
– 2. SWinIR 网络结构
–
+ 2.1 整体框架
+ 2.2 浅层特征提取
+ 2.3 深层特征提取
+ 2.4 图像重建模块
– 3. 主要代码理解
–
+ 3.1 SwinIR
+ 3.2 MLP
+ 3.3 Patch Embedding
+ 3.4 Window Attention
+ 3.5 残差 Swin Transformer 块 (RSTB)
+ 3.6 HQ Image Reconstruction
+ 3.7 一个测试实例
– 4. 参考文献
– 5. 结语
1. SwinIR 论文
主要工作：将 Swin Transformer 在图像恢复中应用，降低参数量的同时取得很好的效果。
论文地址：https://arxiv.org/pdf/2108.10257.pdf
源代码：
[1] https://github.com/JingyunLiang/SwinIR
[2] https://download.csdn.net/download/Wenyuanbo/40284900
自定义完整训练测试代码： https://download.csdn.net/download/Wenyuanbo/78124947
; 2. SWinIR 网络结构
2.1 整体框架
SwinIR 的网络结构主要分为 3 个部分，分别是浅层特征提取模块，深层特征提取模块和高质量图像重建模块。其中前后两个模块都是基于 CNN 的，中间模块则主要使用 SwInTransformer。
; 2.2 浅层特征提取
浅层特征提取只使用一层卷积进行提取。
2.3 深层特征提取
深层特征提取模块由若干个残差 SwInTransformer 块 (RSTB) 和卷积块构成，具体结构如下图。
(1) 首先将来自浅层特征提取模块的特征图分割成多个不重叠的 patch embeddings；
(2) 再通过多个串联的残差 SWin Transformer 块 (RSTB);
(3) 将多个不重叠的 patch embeddings 重新组合成与输入特征图分辨率一样；
(4) 最后通过一个卷积层 (1 层或3 层卷积) 输出；
(5) 在每个 RSTB 中都引入残差连接。
残差 SwInTransformer 块 (RSTB) 中的 STL 就是 SwIn Transformer Layer 的意思，具体结构如下图。
(1) 首先通过一个归一化层 LayerNorm；
(2) 再通过多头自注意力 (Multi-head Self Attention) 模块；
(3) 在多头自注意力结尾引入残差；
(4) 再通过一个归一化层 LayerNorm；
(5) 最后通过一个多层感知机 MLP；
(6) 结尾同样引入残差。
; 2.4 图像重建模块
图像重建模块其实就是卷积+上采样的组合，在这块论文提出 4 种结构。（ 注：这是一个容易改进的地方）
(1) 经典超分 (卷积 + pixelshuffle 上采样 + 卷积)；
(2) 轻量超分 (卷积 + pixelshuffle 上采样)；
(3) 真实图像超分 (卷积 + 卷积插值上采样 + 卷积插值上采样 + 卷积)；
(4) 像去噪和 JPEG 压缩去伪影 （卷积 + 引入残差）。
3. 主要代码理解
关于 SwinIR 中涉及 CNN 的部分代码非常简单，就不在此单独列出，这里主要注释一下其中有关 Swin Transformer 的实现代码。另外针对不构成主要网络结构的部分代码进行了删减，完整代码请移步：
(1) GitHub 链接：https://github.com/JingyunLiang/SwinIR
(2) CSDN 链接：https://download.csdn.net/download/Wenyuanbo/40284900
(3) 详尽注释代码：https://download.csdn.net/download/Wenyuanbo/40284085
3.1 SwinIR
SwinIR 主要由浅层特征提取，深层特征提取和高质量图像重建模块组成，具体原理如前所说，直接欣赏代码吧。
class SwinIR(nn.Module):
r""" SwinIR
基于 Swin Transformer 的图像恢复网络.
输入:
img_size (int | tuple(int)): 输入图像的大小，默认为 64*64.
patch_size (int | tuple(int)): patch 的大小，默认为 1.
in_chans (int): 输入图像的通道数，默认为 3.
embed_dim (int): Patch embedding 的维度，默认为 96.
depths (tuple(int)): Swin Transformer 层的深度.
num_heads (tuple(int)): 在不同层注意力头的个数.
window_size (int): 窗口大小，默认为 7.
mlp_ratio (float): MLP隐藏层特征图通道与嵌入层特征图通道的比，默认为 4.
qkv_bias (bool): 给 query, key, value 添加可学习的偏置，默认为 True.
qk_scale (float): 重写默认的缩放因子，默认为 None.
drop_rate (float): 随机丢弃神经元，丢弃率默认为 0.
attn_drop_rate (float): 注意力权重的丢弃率，默认为 0.
drop_path_rate (float): 深度随机丢弃率，默认为 0.1.
norm_layer (nn.Module): 归一化操作，默认为 nn.LayerNorm.
ape (bool): patch embedding 添加绝对位置 embedding，默认为 False.
patch_norm (bool): 在 patch embedding 后添加归一化操作，默认为 True.
use_checkpoint (bool): 是否使用 checkpointing 来节省显存，默认为 False.
upscale: 放大因子， 2/3/4/8 适合图像超分, 1 适合图像去噪和 JPEG 压缩去伪影
img_range: 灰度值范围， 1 或者 255.
upsampler: 图像重建方法的选择模块，可选择 pixelshuffle, pixelshuffledirect, nearest+conv 或 None.
resi_connection: 残差连接之前的卷积块， 可选择 1conv 或 3conv.
"""
def __init__(self, img_size=64, patch_size=1, in_chans=3,
embed_dim=96, depths=[6, 6, 6, 6], num_heads=[6, 6, 6, 6],
window_size=7, mlp_ratio=4., qkv_bias=True, qk_scale=None,
drop_rate=0., attn_drop_rate=0., drop_path_rate=0.1,
norm_layer=nn.LayerNorm, ape=False, patch_norm=True,
use_checkpoint=False, upscale=2, img_range=1., upsampler='', resi_connection='1conv',
**kwargs):
super(SwinIR, self).__init__()
num_in_ch = in_chans
num_out_ch = in_chans
num_feat = 64
self.img_range = img_range
if in_chans == 3:
rgb_mean = (0.4488, 0.4371, 0.4040)
self.mean = torch.Tensor(rgb_mean).view(1, 3, 1, 1)
else:
self.mean = torch.zeros(1, 1, 1, 1)
self.upscale = upscale
self.upsampler = upsampler
self.window_size = window_size
self.conv_first = nn.Conv2d(num_in_ch, embed_dim, 3, 1, 1)
self.num_layers = len(depths)
self.embed_dim = embed_dim
self.ape = ape
self.patch_norm = patch_norm
self.num_features = embed_dim
self.mlp_ratio = mlp_ratio
self.patch_embed = PatchEmbed(
img_size=img_size, patch_size=patch_size, in_chans=embed_dim, embed_dim=embed_dim,
norm_layer=norm_layer if self.patch_norm else None)
num_patches = self.patch_embed.num_patches
patches_resolution = self.patch_embed.patches_resolution
self.patches_resolution = patches_resolution
self.patch_unembed = PatchUnEmbed(
img_size=img_size, patch_size=patch_size, in_chans=embed_dim, embed_dim=embed_dim,
norm_layer=norm_layer if self.patch_norm else None)
if self.ape:
self.absolute_pos_embed = nn.Parameter(torch.zeros(1, num_patches, embed_dim))
trunc_normal_(self.absolute_pos_embed, std=.02)
self.pos_drop = nn.Dropout(p=drop_rate)
dpr = [x.item() for x in torch.linspace(0, drop_path_rate, sum(depths))]
self.layers = nn.ModuleList()
for i_layer in range(self.num_layers):
layer = RSTB(dim=embed_dim,
input_resolution=(patches_resolution[0],
patches_resolution[1]),
depth=depths[i_layer],
num_heads=num_heads[i_layer],
window_size=window_size,
mlp_ratio=self.mlp_ratio,
qkv_bias=qkv_bias, qk_scale=qk_scale,
drop=drop_rate, attn_drop=attn_drop_rate,
drop_path=dpr[sum(depths[:i_layer]):sum(depths[:i_layer + 1])],
norm_layer=norm_layer,
downsample=None,
use_checkpoint=use_checkpoint,
img_size=img_size,
patch_size=patch_size,
resi_connection=resi_connection
)
self.layers.append(layer)
self.norm = norm_layer(self.num_features)
if resi_connection == '1conv':
self.conv_after_body = nn.Conv2d(embed_dim, embed_dim, 3, 1, 1)
elif resi_connection == '3conv':
self.conv_after_body = nn.Sequential(nn.Conv2d(embed_dim, embed_dim // 4, 3, 1, 1),
nn.LeakyReLU(negative_slope=0.2, inplace=True),
nn.Conv2d(embed_dim // 4, embed_dim // 4, 1, 1, 0),
nn.LeakyReLU(negative_slope=0.2, inplace=True),
nn.Conv2d(embed_dim // 4, embed_dim, 3, 1, 1))
if self.upsampler == 'pixelshuffle':
self.conv_before_upsample = nn.Sequential(nn.Conv2d(embed_dim, num_feat, 3, 1, 1),
nn.LeakyReLU(inplace=True))
self.upsample = Upsample(upscale, num_feat)
self.conv_last = nn.Conv2d(num_feat, num_out_ch, 3, 1, 1)
elif self.upsampler == 'pixelshuffledirect':
self.upsample = UpsampleOneStep(upscale, embed_dim, num_out_ch,
(patches_resolution[0], patches_resolution[1]))
elif self.upsampler == 'nearest+conv':
assert self.upscale == 4, 'only support x4 now.'
self.conv_before_upsample = nn.Sequential(nn.Conv2d(embed_dim, num_feat, 3, 1, 1),
nn.LeakyReLU(inplace=True))
self.conv_up1 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
self.conv_up2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
self.conv_hr = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
self.lrelu = nn.LeakyReLU(negative_slope=0.2, inplace=True)
self.conv_last = nn.Conv2d(num_feat, num_out_ch, 3, 1, 1)
else:
self.conv_last = nn.Conv2d(embed_dim, num_out_ch, 3, 1, 1)
self.apply(self._init_weights)
def _init_weights(self, m):
if isinstance(m, nn.Linear):
trunc_normal_(m.weight, std=.02)
if m.bias is not None:
nn.init.constant_(m.bias, 0)
elif isinstance(m, nn.LayerNorm):
nn.init.constant_(m.bias, 0)
nn.init.constant_(m.weight, 1.0)
def check_image_size(self, x):
_, _, h, w = x.size()
mod_pad_h = (self.window_size - h % self.window_size) % self.window_size
mod_pad_w = (self.window_size - w % self.window_size) % self.window_size
x = F.pad(x, (0, mod_pad_w, 0, mod_pad_h), 'reflect')
return x
def forward_features(self, x):
x_size = (x.shape[2], x.shape[3])
x = self.patch_embed(x)
if self.ape:
x = x + self.absolute_pos_embed
x = self.pos_drop(x)
for layer in self.layers:
x = layer(x, x_size)
x = self.norm(x)
x = self.patch_unembed(x, x_size)
return x
def forward(self, x):
H, W = x.shape[2:]
x = self.check_image_size(x)
self.mean = self.mean.type_as(x)
x = (x - self.mean) * self.img_range
if self.upsampler == 'pixelshuffle':
x = self.conv_first(x)
x = self.conv_after_body(self.forward_features(x)) + x
x = self.conv_before_upsample(x)
x = self.conv_last(self.upsample(x))
elif self.upsampler == 'pixelshuffledirect':
x = self.conv_first(x)
x = self.conv_after_body(self.forward_features(x)) + x
x = self.upsample(x)
elif self.upsampler == 'nearest+conv':
x = self.conv_first(x)
x = self.conv_after_body(self.forward_features(x)) + x
x = self.conv_before_upsample(x)
x = self.lrelu(self.conv_up1(torch.nn.functional.interpolate(x, scale_factor=2, mode='nearest')))
x = self.lrelu(self.conv_up2(torch.nn.functional.interpolate(x, scale_factor=2, mode='nearest')))
x = self.conv_last(self.lrelu(self.conv_hr(x)))
else:
x_first = self.conv_first(x)
res = self.conv_after_body(self.forward_features(x_first)) + x_first
x = x + self.conv_last(res)
x = x / self.img_range + self.mean
return x[:, :, :H*self.upscale, :W*self.upscale]
3.2 MLP
多层感知机 MLP 是 transformer 比较基础的部分，具体原理也很简单。
class Mlp(nn.Module):
def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.):
super().__init__()
out_features = out_features or in_features
hidden_features = hidden_features or in_features
self.fc1 = nn.Linear(in_features, hidden_features)
self.act = act_layer()
self.fc2 = nn.Linear(hidden_features, out_features)
self.drop = nn.Dropout(drop)
def forward(self, x):
x = self.fc1(x)
x = self.act(x)
x = self.drop(x)
x = self.fc2(x)
x = self.drop(x)
return x
3.3 Patch Embedding
主要的操作就是将原始 2 维图像 (特征图的一个 plane 或者说一个 channel) 转变为 1 维的 patch embeddings，通过 Swin Transformer 学习处理之后再重新组合成与原来特征图结构一致的新特征图。
(1) 将 2 维图像转变成 1 维 patch embeddings。
class PatchEmbed(nn.Module):
r""" Image to Patch Embedding
输入:
img_size (int): 图像的大小，默认为 224*224.
patch_size (int): Patch token 的大小，默认为 4*4.
in_chans (int): 输入图像的通道数，默认为 3.
embed_dim (int): 线性 projection 输出的通道数，默认为 96.
norm_layer (nn.Module, optional): 归一化层， 默认为N None.
"""
def __init__(self, img_size=224, patch_size=4, in_chans=3, embed_dim=96, norm_layer=None):
super().__init__()
img_size = to_2tuple(img_size)
patch_size = to_2tuple(patch_size)
patches_resolution = [img_size[0] // patch_size[0], img_size[1] // patch_size[1]]
self.img_size = img_size
self.patch_size = patch_size
self.patches_resolution = patches_resolution
self.num_patches = patches_resolution[0] * patches_resolution[1]
self.in_chans = in_chans
self.embed_dim = embed_dim
if norm_layer is not None:
self.norm = norm_layer(embed_dim)
else:
self.norm = None
def forward(self, x):
x = x.flatten(2).transpose(1, 2)
if self.norm is not None:
x = self.norm(x)
return x
(2) 将 1 维 patch embeddings 转变为 2 维图像。
class PatchUnEmbed(nn.Module):
r""" Image to Patch Unembedding
输入:
img_size (int): 图像的大小，默认为 224*224.
patch_size (int): Patch token 的大小，默认为 4*4.
in_chans (int): 输入图像的通道数，默认为 3.
embed_dim (int): 线性 projection 输出的通道数，默认为 96.
norm_layer (nn.Module, optional): 归一化层， 默认为N None.
"""
def __init__(self, img_size=224, patch_size=4, in_chans=3, embed_dim=96, norm_layer=None):
super().__init__()
img_size = to_2tuple(img_size)
patch_size = to_2tuple(patch_size)
patches_resolution = [img_size[0] // patch_size[0], img_size[1] // patch_size[1]]
self.img_size = img_size
self.patch_size = patch_size
self.patches_resolution = patches_resolution
self.num_patches = patches_resolution[0] * patches_resolution[1]
self.in_chans = in_chans
self.embed_dim = embed_dim
def forward(self, x, x_size):
B, HW, C = x.shape
x = x.transpose(1, 2).view(B, self.embed_dim, x_size[0], x_size[1])
return x
3.4 Window Attention
采用窗口注意力来减轻传统 Transformer 的全局注意力带来的计算负担，将注意力的计算限制在每一个窗口里，在每个窗口里其实还是原始的多头自注意力。（ 注：这个窗口注意力我之前也有过类似想法，主要是考虑到由于网络测试阶段的输入图像大小是不定的，如果在其中加入注意力机制得到的注意力图也是不定的，这一定程度上限制网络的泛化性能，将窗口注意力的思想迁移到 CNN 相信也能有不错的表现）
(1) 窗口分割
def window_partition(x, window_size):
"""
输入:
x: (B, H, W, C)
window_size (int): window size # 窗口的大小
返回:
windows: (num_windows*B, window_size, window_size, C) # 每一个 batch 有单独的 windows
"""
B, H, W, C = x.shape
x = x.view(B, H // window_size, window_size, W // window_size, window_size, C)
windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size, window_size, C)
return windows
(2) 窗口注意力
这里的相对位置索引比较有意思，有不明白的可以参考：图解Swin Transformer。
class WindowAttention(nn.Module):
r""" 基于有相对位置偏差的多头自注意力窗口，支持移位的(shifted)或者不移位的(non-shifted)窗口.
输入:
dim (int): 输入特征的维度.
window_size (tuple[int]): 窗口的大小.
num_heads (int): 注意力头的个数.
qkv_bias (bool, optional): 给 query, key, value 添加可学习的偏置，默认为 True.
qk_scale (float | None, optional): 重写默认的缩放因子 scale.
attn_drop (float, optional): 注意力权重的丢弃率，默认为 0.0.
proj_drop (float, optional): 输出的丢弃率，默认为 0.0.
"""
def __init__(self, dim, window_size, num_heads, qkv_bias=True, qk_scale=None, attn_drop=0., proj_drop=0.):
super().__init__()
self.dim = dim
self.window_size = window_size
self.num_heads = num_heads
head_dim = dim // num_heads
self.scale = qk_scale or head_dim ** -0.5
self.relative_position_bias_table = nn.Parameter(
torch.zeros((2 * window_size[0] - 1) * (2 * window_size[1] - 1), num_heads))
coords_h = torch.arange(self.window_size[0])
coords_w = torch.arange(self.window_size[1])
coords = torch.stack(torch.meshgrid([coords_h, coords_w]))
coords_flatten = torch.flatten(coords, 1)
relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :]
relative_coords = relative_coords.permute(1, 2, 0).contiguous()
relative_coords[:, :, 0] += self.window_size[0] - 1
relative_coords[:, :, 1] += self.window_size[1] - 1
relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
relative_position_index = relative_coords.sum(-1)
self.register_buffer("relative_position_index", relative_position_index)
self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
self.attn_drop = nn.Dropout(attn_drop)
self.proj = nn.Linear(dim, dim)
self.proj_drop = nn.Dropout(proj_drop)
trunc_normal_(self.relative_position_bias_table, std=.02)
self.softmax = nn.Softmax(dim=-1)
def forward(self, x, mask=None):
"""
输入:
x: 输入特征图，结构为 [num_windows*B, N, C]
mask: (0/-inf) mask, 结构为 [num_windows, Wh*Ww, Wh*Ww] 或者没有 mask
"""
B_, N, C = x.shape
qkv = self.qkv(x).reshape(B_, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
q, k, v = qkv[0], qkv[1], qkv[2]
q = q * self.scale
attn = (q @ k.transpose(-2, -1))
relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1)
relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous()
attn = attn + relative_position_bias.unsqueeze(0)
if mask is not None:
nW = mask.shape[0]
attn = attn.view(B_ // nW, nW, self.num_heads, N, N) + mask.unsqueeze(1).unsqueeze(0)
attn = attn.view(-1, self.num_heads, N, N)
attn = self.softmax(attn)
else:
attn = self.softmax(attn)
attn = self.attn_drop(attn)
x = (attn @ v).transpose(1, 2).reshape(B_, N, C)
x = self.proj(x)
x = self.proj_drop(x)
return x
(3) 窗口合并
def window_reverse(windows, window_size, H, W):
"""
输入:
windows: (num_windows*B, window_size, window_size, C) # 分割得到的窗口(已处理)
window_size (int): Window size # 窗口大小
H (int): Height of image # 原分割窗口前特征图的高
W (int): Width of image # 原分割窗口前特征图的宽
返回:
x: (B, H, W, C) # 返回与分割前特征图结构一样的结果
"""
B = int(windows.shape[0] / (H * W / window_size / window_size))
x = windows.view(B, H // window_size, W // window_size, window_size, window_size, -1)
x = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(B, H, W, -1)
return x
3.5 残差 Swin Transformer 块 (RSTB)
SwinIR 主要是使用 Swin Transformer 的思想来实现，残差 Swin Transformer 块 (RSTB) 可以理解为：
(1) Swin Transformer 块是 RSTB 的基础组件；
(2) 多个 Swin Transformer 块构成基础网络；
(3) 基础网络结尾处加上卷积操作后再引入残差构成 RSTB。
(1) Swin Transformer 块
class SwinTransformerBlock(nn.Module):
"""
输入:
dim (int): 输入特征的维度.
input_resolution (tuple[int]): 输入特征图的分辨率.
num_heads (int): 注意力头的个数.
window_size (int): 窗口的大小.
shift_size (int): SW-MSA 的移位值.
mlp_ratio (float): 多层感知机隐藏层的维度和嵌入层的比.
qkv_bias (bool, optional): 给 query, key, value 添加一个可学习偏置，默认为 True.
qk_scale (float | None, optional): 重写默认的缩放因子 scale.
drop (float, optional): 随机神经元丢弃率，默认为 0.0.
attn_drop (float, optional): 注意力图随机丢弃率，默认为 0.0.
drop_path (float, optional): 深度随机丢弃率，默认为 0.0.
act_layer (nn.Module, optional): 激活函数，默认为 nn.GELU.
norm_layer (nn.Module, optional): 归一化操作，默认为 nn.LayerNorm.
"""
def __init__(self, dim, input_resolution, num_heads, window_size=7, shift_size=0,
mlp_ratio=4., qkv_bias=True, qk_scale=None, drop=0., attn_drop=0., drop_path=0.,
act_layer=nn.GELU, norm_layer=nn.LayerNorm):
super().__init__()
self.dim = dim
self.input_resolution = input_resolution
self.num_heads = num_heads
self.window_size = window_size
self.shift_size = shift_size
self.mlp_ratio = mlp_ratio
if min(self.input_resolution) self.window_size:
self.shift_size = 0
self.window_size = min(self.input_resolution)
assert 0 self.shift_size < self.window_size, "shift_size must in 0-window_size"
self.norm1 = norm_layer(dim)
self.attn = WindowAttention(
dim, window_size=to_2tuple(self.window_size), num_heads=num_heads,
qkv_bias=qkv_bias, qk_scale=qk_scale, attn_drop=attn_drop, proj_drop=drop)
self.drop_path = DropPath(drop_path) if drop_path > 0. else nn.Identity()
self.norm2 = norm_layer(dim)
mlp_hidden_dim = int(dim * mlp_ratio)
self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)
if self.shift_size > 0:
attn_mask = self.calculate_mask(self.input_resolution)
else:
attn_mask = None
self.register_buffer("attn_mask", attn_mask)
def calculate_mask(self, x_size):
H, W = x_size
img_mask = torch.zeros((1, H, W, 1))
h_slices = (slice(0, -self.window_size),
slice(-self.window_size, -self.shift_size),
slice(-self.shift_size, None))
w_slices = (slice(0, -self.window_size),
slice(-self.window_size, -self.shift_size),
slice(-self.shift_size, None))
cnt = 0
for h in h_slices:
for w in w_slices:
img_mask[:, h, w, :] = cnt
cnt += 1
mask_windows = window_partition(img_mask, self.window_size)
mask_windows = mask_windows.view(-1, self.window_size * self.window_size)
attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)
attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
return attn_mask
def forward(self, x, x_size):
H, W = x_size
B, L, C = x.shape
shortcut = x
x = self.norm1(x)
x = x.view(B, H, W, C)
if self.shift_size > 0:
shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(1, 2))
else:
shifted_x = x
x_windows = window_partition(shifted_x, self.window_size)
x_windows = x_windows.view(-1, self.window_size * self.window_size, C)
if self.input_resolution == x_size:
attn_windows = self.attn(x_windows, mask=self.attn_mask)
else:
attn_windows = self.attn(x_windows, mask=self.calculate_mask(x_size).to(x.device))
attn_windows = attn_windows.view(-1, self.window_size, self.window_size, C)
shifted_x = window_reverse(attn_windows, self.window_size, H, W)
if self.shift_size > 0:
x = torch.roll(shifted_x, shifts=(self.shift_size, self.shift_size), dims=(1, 2))
else:
x = shifted_x
x = x.view(B, H * W, C)
x = shortcut + self.drop_path(x)
x = x + self.drop_path(self.mlp(self.norm2(x)))
return x
(2) 基础网络
class BasicLayer(nn.Module):
"""
输入:
dim (int): 输入特征的维度.
input_resolution (tuple[int]): 输入分辨率.
depth (int): SWin Transformer 块的个数.
num_heads (int): 注意力头的个数.
window_size (int): 本地(当前块中)窗口的大小.
mlp_ratio (float): MLP隐藏层特征维度与嵌入层特征维度的比.
qkv_bias (bool, optional): 给 query, key, value 添加一个可学习偏置，默认为 True.
qk_scale (float | None, optional): 重写默认的缩放因子 scale.
drop (float, optional): 随机丢弃神经元，丢弃率默认为 0.0.
attn_drop (float, optional): 注意力图随机丢弃率，默认为 0.0.
drop_path (float | tuple[float], optional): 深度随机丢弃率，默认为 0.0.
norm_layer (nn.Module, optional): 归一化操作，默认为 nn.LayerNorm.
downsample (nn.Module | None, optional): 结尾处的下采样层，默认没有.
use_checkpoint (bool): 是否使用 checkpointing 来节省显存，默认为 False.
"""
def __init__(self, dim, input_resolution, depth, num_heads, window_size,
mlp_ratio=4., qkv_bias=True, qk_scale=None, drop=0., attn_drop=0.,
drop_path=0., norm_layer=nn.LayerNorm, downsample=None, use_checkpoint=False):
super().__init__()
self.dim = dim
self.input_resolution = input_resolution
self.depth = depth
self.use_checkpoint = use_checkpoint
self.blocks = nn.ModuleList([
SwinTransformerBlock(dim=dim, input_resolution=input_resolution,
num_heads=num_heads, window_size=window_size,
shift_size=0 if (i % 2 == 0) else window_size // 2,
mlp_ratio=mlp_ratio,
qkv_bias=qkv_bias, qk_scale=qk_scale,
drop=drop, attn_drop=attn_drop,
drop_path=drop_path[i] if isinstance(drop_path, list) else drop_path,
norm_layer=norm_layer)
for i in range(depth)])
if downsample is not None:
self.downsample = downsample(input_resolution, dim=dim, norm_layer=norm_layer)
else:
self.downsample = None
def forward(self, x, x_size):
for blk in self.blocks:
if self.use_checkpoint:
x = checkpoint.checkpoint(blk, x, x_size)
else:
x = blk(x, x_size)
if self.downsample is not None:
x = self.downsample(x)
return x
(3) 残差 Swin Transformer 块 (RSTB)
class RSTB(nn.Module):
"""
输入:
dim (int): 输入特征的维度.
input_resolution (tuple[int]): 输入分辨率.
depth (int): SWin Transformer 块的个数.
num_heads (int): 注意力头的个数.
window_size (int): 本地(当前块中)窗口的大小.
mlp_ratio (float): MLP隐藏层特征维度与嵌入层特征维度的比.
qkv_bias (bool, optional): 给 query, key, value 添加一个可学习偏置，默认为 True.
qk_scale (float | None, optional): 重写默认的缩放因子 scale.
drop (float, optional): D 随机丢弃神经元，丢弃率默认为 0.0.
attn_drop (float, optional): 注意力图随机丢弃率，默认为 0.0.
drop_path (float | tuple[float], optional): 深度随机丢弃率，默认为 0.0.
norm_layer (nn.Module, optional): 归一化操作，默认为 nn.LayerNorm.
downsample (nn.Module | None, optional): 结尾处的下采样层，默认没有.
use_checkpoint (bool): 是否使用 checkpointing 来节省显存，默认为 False.
img_size: 输入图片的大小.
patch_size: Patch 的大小.
resi_connection: 残差连接之前的卷积块.
"""
def __init__(self, dim, input_resolution, depth, num_heads, window_size,
mlp_ratio=4., qkv_bias=True, qk_scale=None, drop=0., attn_drop=0.,
drop_path=0., norm_layer=nn.LayerNorm, downsample=None, use_checkpoint=False,
img_size=224, patch_size=4, resi_connection='1conv'):
super(RSTB, self).__init__()
self.dim = dim
self.input_resolution = input_resolution
self.residual_group = BasicLayer(dim=dim,
input_resolution=input_resolution,
depth=depth,
num_heads=num_heads,
window_size=window_size,
mlp_ratio=mlp_ratio,
qkv_bias=qkv_bias, qk_scale=qk_scale,
drop=drop, attn_drop=attn_drop,
drop_path=drop_path,
norm_layer=norm_layer,
downsample=downsample,
use_checkpoint=use_checkpoint)
if resi_connection == '1conv':
self.conv = nn.Conv2d(dim, dim, 3, 1, 1)
elif resi_connection == '3conv':
self.conv = nn.Sequential(nn.Conv2d(dim, dim // 4, 3, 1, 1), nn.LeakyReLU(negative_slope=0.2, inplace=True),
nn.Conv2d(dim // 4, dim // 4, 1, 1, 0),
nn.LeakyReLU(negative_slope=0.2, inplace=True),
nn.Conv2d(dim // 4, dim, 3, 1, 1))
self.patch_embed = PatchEmbed(
img_size=img_size, patch_size=patch_size, in_chans=0, embed_dim=dim,
norm_layer=None)
self.patch_unembed = PatchUnEmbed(
img_size=img_size, patch_size=patch_size, in_chans=0, embed_dim=dim,
norm_layer=None)
def forward(self, x, x_size):
return self.patch_embed(self.conv(self.patch_unembed(self.residual_group(x, x_size), x_size))) + x
3.6 HQ Image Reconstruction
高质量图像重建模块其实就是卷积和上采样操作的组合，在这块论文提出 4 种结构。
(1) 经典超分 (卷积 + pixelshuffle 上采样 + 卷积)；
(2) 轻量超分 (卷积 + pixelshuffle 上采样)；
(3) 真实图像超分 (卷积 + 卷积插值上采样 + 卷积插值上采样 + 卷积)；
(4) 像去噪和 JPEG 压缩去伪影 （卷积 + 引入残差）。
在这里主要看一下两种上采样操作：
(1) 先卷积再使用 pixelshuffle 上采样，特征图维度不是 3
class Upsample(nn.Sequential):
"""
输入:
scale (int): 缩放因子，支持 2^n and 3.
num_feat (int): 中间特征的通道数.
"""
def __init__(self, scale, num_feat):
m = []
if (scale & (scale - 1)) == 0:
for _ in range(int(math.log(scale, 2))):
m.append(nn.Conv2d(num_feat, 4 * num_feat, 3, 1, 1))
m.append(nn.PixelShuffle(2))
elif scale == 3:
m.append(nn.Conv2d(num_feat, 9 * num_feat, 3, 1, 1))
m.append(nn.PixelShuffle(3))
else:
raise ValueError(f'scale {scale} is not supported. ' 'Supported scales: 2^n and 3.')
super(Upsample, self).__init__(*m)
(2) 一步既上采样也实现输出降维，特征图维度是 3，即最后的恢复图像
class UpsampleOneStep(nn.Sequential):
"""一步上采样与前边上采样模块不同之处在于该模块只有一个卷积层和一个 pixelshuffle 层
输入:
scale (int): 缩放因子，支持 2^n and 3.
num_feat (int): 中间特征的通道数.
"""
def __init__(self, scale, num_feat, num_out_ch, input_resolution=None):
self.num_feat = num_feat
self.input_resolution = input_resolution
m = []
m.append(nn.Conv2d(num_feat, (scale ** 2) * num_out_ch, 3, 1, 1))
m.append(nn.PixelShuffle(scale))
super(UpsampleOneStep, self).__init__(*m)
3.7 一个测试实例
虽然 SwinIR 的整体参数不大，但是计算负担比较大。
upscale = 4
window_size = 8
height = (1024 // upscale // window_size + 1) * window_size
width = (720 // upscale // window_size + 1) * window_size
model = SwinIR(upscale=2, img_size=(height, width),
window_size=window_size, img_range=1., depths=[6, 6, 6, 6],
embed_dim=60, num_heads=[6, 6, 6, 6], mlp_ratio=2, upsampler='pixelshuffledirect')
print(model)
print(height, width)
x = torch.randn((1, 3, height, width))
x = model(x)
print(x.shape)
4. 参考文献
[1] Liang J, Cao J, Sun G, et al. SwinIR: Image restoration using swin transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 1833-1844.
[2] Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[J]. arXiv preprint arXiv:2103.14030, 2021.
5. 结语
转载请注明出处。SwinIR 本身没有太多创新，主要还是 Swin Transformer 在图像恢复领域进行应用，但是整体网络对显卡的要求已经很接近纯 CNN 的网络。完整的 SwinIR 的注释代码可以移步链接：https://download.csdn.net/download/Wenyuanbo/40284085。原作者没有公开具体的训练测试代码，自定义的完整训练测试代码可以移步链接：https://download.csdn.net/download/Wenyuanbo/78124947既然现在 Swin Transfomer 已经公开，那相信不久就会如同当初的 ResNet 一样有许多改进方案不断出现，先到先得。如果我的这篇文章对你有所帮助，希望能不吝点赞关注一波。
Original: https://blog.csdn.net/Wenyuanbo/article/details/121264131
Author: 听 风、
Title: 图像恢复 SWinIR : 彻底理解论文和源代码 (注释详尽）
原创文章受到原创版权保护。转载请注明出处：https://www.johngo689.com/532330/
转载文章受原作者版权保护。转载请注明原作者出处！