TPU MNIST demo hangs in last batch
I am afraid this is not working for me.
_Remark : There have been various posts about this or very similar issues, but as far as I can see they have all been closed.
Example: #1590
In fact I posted this exact comment in the following issue, when it was already closed.
#1403
I am therefore creating this issue, because I think the closed issue is probably not receiving any attention, which is understandable._
now:
I have tried all the versions, given in the notebook. (The links are given below in NB1)
Additionally, I have also tried it with the version 20200516. That version is given in the official colab TPU MNIST example notebook which does not use pytorch-lightening, ie 20200516. A reference is below in NB2.
The summary of the results are:
"1.5" : wont run at all
"20200325" hangs in the final epoch (with 10 epochs in the 10th, with 3 epochs in the 3rd)
"nightly" crashes with : Exception: process 0 terminated with signal SIGABRT
"20200516" hangs after one epoch
I have tried this several times over the last few days. With the exception of the nightly all these results have always been the same.
NB1:
Locally I am on a Mac, not sure whether this makes a difference.
My terminal gives this
uname -a
Darwin osx-lhind6957 18.7.0 Darwin Kernel Version 18.7.0: Mon Apr 27 20:09:39 PDT 2020; root:xnu-4903.278.35~1/RELEASE_X86_64 x86_64
NB2:
The links for that official colab TPU MNIST example notebook which does not use pytorch lightning are here:
https://cloud.google.com/tpu/docs/colabs?hl=de
https://colab.research.google.com/github/pytorch/xla/blob/master/contrib/colab/mnist-training.ipynb?authuser=1#scrollTo=sPJVqAKyml5W
(The official notebook which does not use pytorch lightning has no problem and runs through with 20200516)