Node: ‘conv2d_transpose_1/conv2d_transpose/Conv2DBackpropFilter‘No algorithm worked Error messages

情况说明

用tensorflow在GPU上跑一个fit,模型里用到了两个卷积函数layers.Conv2D和layers.Conv2DTranspose
epoch时出错,具体错误日志见下文,一开始Conv2D位置报错

解决过程

(1)搜索到一个类似的:No algorithm worked!,即代码开头加上以下设置

from tensorflow.config.experimental import list_physical_devices, set_memory_growth
physical_devices = list_physical_devices('GPU')
set_memory_growth(physical_devices[0], True)

(2)现在Conv2DTranspose位置报错,据错误日志推测大概是调用cudnn跑卷积转置失败,使用错误日志搜索无果后,使用关键词”cudnn运行反卷积gpu失败”,同样参考类似问题:Could not load library cudnn_cnn_infer64_8.dll,可能是cudnn的底层cnn计算库本身有问题,学博主的降一个cudnn版本

总结

  • 解决方案就是【加代码+cudnn降版本】
  • 当前版本环境:window10、tensorflow2.8.0、cuda11.2.2、cudnn8.1.0.77,下载地址如下:
  • tensorflow官方文档
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple tensorflow

这里暂时使用清华镜像站作为源地址,因为官方源地址下载速度很慢。

[En]

Tsinghua mirror station is temporarily used as the source address here, because the official source address download is very slow.


pip config list

python -m pip install --upgrade pip

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

  • cuda11.2.2 我本来装的当前最新版cuda11.6+cudnn8.2,考虑到tensorflow官网上跑GPU的软件要求,就开始换cuda版本,先在控制面板里卸载带有CUDA名字的,大概有6个,再用CCleaner排查注册表问题,再安装11.2
    Node: ‘conv2d_transpose_1/conv2d_transpose/Conv2DBackpropFilter‘No algorithm worked Error messages
  • cudnncudnn8.1.0.77
  • 其间接连从anaconda里卸载tensorflow,直接安装到终端之类的,然并卵
  • 其间还尝试只用CPU跑来排除到底是不是GPU环境配置的问题
import os
os.environ["CUDA_VISIBLE_DEVICES"]="-1"

错误日志

NotFoundError: Graph execution error: Detected at node ‘gradient_tape/denoise/sequential_1/conv2d_transpose_1/conv2d_transpose/Conv2DBackpropFilter’ defined at (most recent call last):

Node: ‘gradient_tape/denoise/sequential_1/conv2d_transpose_1/conv2d_transpose/Conv2DBackpropFilter’
No algorithm worked! Error messages:
Profiling failure on CUDNN engine 1#TC: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
Profiling failure on CUDNN engine 1: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
Profiling failure on CUDNN engine 0#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
Profiling failure on CUDNN engine 0: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
Profiling failure on CUDNN engine 3#TC: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
Profiling failure on CUDNN engine 3: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
[[{{node gradient_tape/denoise/sequential_1/conv2d_transpose_1/conv2d_transpose/Conv2DBackpropFilter}}]] [Op:__inference_train_function_973]

Original: https://blog.csdn.net/qq_39380838/article/details/123777080
Author: polar-bear-lily
Title: Node: ‘conv2d_transpose_1/conv2d_transpose/Conv2DBackpropFilter‘No algorithm worked Error messages

原创文章受到原创版权保护。转载请注明出处:https://www.johngo689.com/497047/

转载文章受原作者版权保护。转载请注明原作者出处!

(0)

大家都在看

亲爱的 Coder【最近整理,可免费获取】👉 最新必读书单  | 👏 面试题下载  | 🌎 免费的AI知识星球