ImageNet数据集较大，解压前有136G(train)+6.3G(val)；解压后有140G(train)+6.4G(val)，并且较常用于图像分割，所以要快速导入ImageNet并用与分类，则需要做些预处理：
这其中会涉及到”内存大小是否足够、文件夹的删除、重命名”等等操作。
改存储
修改 /dev/shm/为160G：
sudo mount -o size=163840M -o remount /dev/shm
如果下载的数据集是压缩的，可以在下载的文件夹下使用以下命令进行解压和数据移动：
mkdir -p /dev/shm/ILSVRC12_256/{train,val} && tar -xvf ILSVRC2012_img_train.tar -C /dev/shm/ILSVRC12_256/train && tar -xvf ILSVRC2012_img_val.tar -C /dev/shm/ILSVRC12_256/val
cd /dev/shm/ILSVRC12_256/train
find . -name "*.tar" | while read NAME ; do mkdir -p "${NAME%.tar}"; tar -xvf "${NAME}" -C "${NAME%.tar}"; rm -f "${NAME}"; done
cd /dev/shm/ILSVRC12_256/val
wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash
图片预处理
[可选择操作，使用后在代码中就不用 resize了] 对所有图片进行做resize的预处理，加快图片从磁盘加载到内存（处理后68G(train)+2.9G(val)；linux默认没有安装convert，使用 sudo apt-get install imagemagick进行安装）：
cd /dev/shm/ILSVRC12_256/val
find . -name "*.JPEG" | xargs -I {} convert {} -resize "256^>" {}
cd /dev/shm/ILSVRC12_256/train
find . -name "*.JPEG" | xargs -I {} convert {} -resize "256^>" {}
scp -r username@servername:/var/www/remote_dir/（远程目录） /var/www/local_dir（本地目录）
df -h
du -hs *
mv dir1 dir2
find . -mindepth 2 -type f -exec mv -i -- {} . \;
find . -depth -mindepth 1 -type d -empty -exec rmdir {} \;
cp -a A/. B
cp -a A B
ln -s ../A A
rm -rf ./A
文件传输出现 No space left on device 问题（如下）
(base) HLBayes@ubuntu:~/HLBayes/a_Data$ scp -r user@xxx.xxx.xxx.xxx:~/HLBayses/ILSVRC2012_img_train.tar.gz ./
ILSVRC2012_img_train.tar.gz 100% 136GB 5.1MB/s 7:34:31
.//ILSVRC2012_img_train.tar.gz: No space left on device
先通过 df -h看看是不是磁盘满了（结果显示并没有）
(base) HLBayes@ubuntu:~/HLBayes/a_Data$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 2.2M 6.3G 1% /run
/dev/sda5 469G 445G 16M 100% /
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/loop1 56M 56M 0 100% /snap/core18/2128
/dev/loop0 219M 219M 0 100% /snap/gnome-3-34-1804/72
/dev/loop4 51M 51M 0 100% /snap/snap-store/547
/dev/loop3 33M 33M 0 100% /snap/snapd/12704
/dev/loop2 66M 66M 0 100% /snap/gtk-common-themes/1515
/dev/sdb1 1.8T 507G 1.3T 30% /home/HLBayes/sdb1
/dev/sda1 512M 6.6M 506M 2% /boot/efi
/dev/sda4 1.9T 1.0T 800G 57% /home/HLBayes/sda4
/dev/sda2 1.2T 902G 250G 79% /home/HLBayes/sda2
tmpfs 6.3G 8.0K 6.3G 1% /run/user/1000
再用 lsof | grep deleted确认是不是某些大文件被删除，但并没有释放空间而只是被标记为 (deleted) （结果显示确实是，并可以看到它们的描述符 354916、 354967…，将它们杀掉就好了。注意：在 kill 354916之前先用 htop查看是否是自己正在跑的程序，以防把自己在跑的程序弄断了。）
(base) HLBayes@ubuntu:/~/HLBayes/a_Data$ lsof | grep deleted
pulseaudi 16981 ws 6u REG 0,1 67108864 6144 /memfd:pulseaudio (deleted)
pulseaudi 16981 17058 null-sink ws 6u REG 0,1 67108864 6144 /memfd:pulseaudio (deleted)
pulseaudi 16981 17084 snapd-gli ws 6u REG 0,1 67108864 6144 /memfd:pulseaudio (deleted)
sh 354906 ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 354917 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 354918 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 354919 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 354920 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 354921 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 354922 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 354943 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 354944 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 354945 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354916 354946 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 354968 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 354969 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 354970 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 354971 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 354972 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 354976 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 354983 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 354984 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 354985 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 354967 354986 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355005 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355006 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355007 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355008 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355009 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355010 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355014 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355015 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355016 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355017 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355004 355040 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355019 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355020 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355021 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355022 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355023 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355024 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355025 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355026 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355027 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355028 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355029 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355018 355030 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355066 ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355066 355067 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355066 355068 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355066 355069 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355066 355070 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355066 355071 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 355066 355072 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 381610 ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 381610 381611 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 381610 381612 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 381610 381613 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 381610 381614 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 381610 381615 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 381610 381616 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node 381610 381617 node ws 99w REG 8,5 0 2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
torch导入数据时出现 RuntimeError: Trying to resize storage that is not resizable
数据的dataloader构造时没有使用 resize或者 crop对样本进行transform。
Original: https://blog.csdn.net/HLBoy_happy/article/details/125060319
Author: HLBayes
Title: ImageNet 数据集准备（用于分类）
原创文章受到原创版权保护。转载请注明出处：https://www.johngo689.com/661664/
转载文章受原作者版权保护。转载请注明原作者出处！