对于tensorflow GPU训练下Function call stack: train_function 问题的解决（.fit_generator callback）

一、错误记录：

callbacks = callbacks_list#callbacks_list

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func

    return func(*args, **kwargs)

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1479, in fit_generator

    initial_epoch=initial_epoch)

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\keras\engine\training.py", line 66, in _method_wrapper

    return method(self, *args, **kwargs)

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\keras\engine\training.py", line 848, in fit

    tmp_logs = train_function(iterator)

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in __call__

    result = self._call(*args, **kwds)

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\eager\def_function.py", line 644, in _call

    return self._stateless_fn(*args, **kwds)

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\eager\function.py", line 2420, in __call__

    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\eager\function.py", line 1665, in _filtered_call

    self.captured_inputs)

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\eager\function.py", line 1746, in _call_flat

    ctx, args, cancellation_manager=cancellation_manager))

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\eager\function.py", line 598, in call

    ctx=ctx)

  File "C:\Users\123\anaconda3\envs\py37\lib\site-packages\tensorflow\python\eager\execute.py", line 60, in quick_execute

    inputs, attrs, num_outputs)

tensorflow.python.framework.errors_impl.UnimplementedError:  Cast string to float is not supported

         [[node model/Cast (defined at train.py:306) ]] [Op:__inference_train_function_30543]

 

Function call stack:

train_function

 

 

二、解决方案

参照：https://*.com/questions/61304463/issue-with-keras-fit-generator-epoch

 

在使用cpu训练时没有问题，使用gpu时出现Function call stack:

train_function 本来以为是传参错误，

按照回溯检查了

ModelCheckpoint, EarlyStopping, ReduceLROnPlateau

.fit_generator   （ callback） 没有找出问题

 

在*上大神们的启发下

发现是在tensorflow2.x下调用tf.combat1没有关闭eager_execution()

按照上面给出的第三点

加入

tf.compat.v1.disable_eager_execution()

 

运行成功！

 

 

附录：pip list

Package                  Version
------------------------ ------------
-ensorflow-gpu           2.2.0
absl-py                  0.9.0
astor                    0.8.1
astunparse               1.6.3
attr                     0.3.1
attrs                    19.3.0
bleach                   1.5.0
cachetools               4.1.1
certifi                  2020.6.20
chardet                  3.0.4
coremltools              4.0b1
cycler                   0.10.0
gast                     0.3.3
google-auth              1.18.0
google-auth-oauthlib     0.4.1
google-pasta             0.2.0
grpcio                   1.30.0
h5py                     2.10.0
html5lib                 0.9999999
idna                     2.10
importlib-metadata       1.7.0
joblib                   0.14.1
Keras                    2.4.3
Keras-Applications       1.0.8
Keras-Preprocessing      1.1.2
kiwisolver               1.1.0
Markdown                 3.2.2
matplotlib               3.0.3
mock                     4.0.2
mpmath                   1.1.0
numpy                    1.19.0
oauthlib                 3.1.0
opt-einsum               3.2.1
pandas                   1.0.5
Pillow                   7.2.0
pip                      20.1.1
protobuf                 3.12.2
pyasn1                   0.4.8
pyasn1-modules           0.2.8
pyparsing                2.4.7
python-dateutil          2.8.1
pytz                     2020.1
PyYAML                   5.3.1
requests                 2.24.0
requests-oauthlib        1.3.0
rsa                      4.6
scikit-learn             0.22.2.post1
scipy                    1.4.1
seaborn                  0.9.1
setuptools               49.1.2
six                      1.15.0
sklearn                  0.0
sympy                    1.6.1
tensorboard              2.2.2
tensorboard-plugin-wit   1.7.0
tensorflow-estimator     1.15.1
tensorflow-gpu           2.2.0
tensorflow-gpu-estimator 2.2.0
termcolor                1.1.0
Theano                   1.0.4
tqdm                     4.47.0
urllib3                  1.25.9
webencodings             0.5.1
Werkzeug                 1.0.1
wheel                    0.29.0
wincertstore             0.2
wrapt                    1.12.1
zipp                     1.2.0
 

 

本文地址：https://blog.csdn.net/Link_night/article/details/107336040