IterableDataset Issue, OverflowError: cannot convert float infinity to integer
Hey all,
I am very new to ML and PyTorch and PyTorch Lightning, so if this is a simple problem sorry to bother.
However, I am struggling to switch from PyTorch to PyTorch Lightning. The PyTorch code runs with no error on Google Colab hence I think that the structure is fine.
Now I am trying to implement Lightning following: https://towardsdatascience.com/from-pytorch-to-pytorch-lightning-a-gentle-introduction-b371b7caaf09
These are the link to my codes:
https://github.com/ykukkim/MLevent/blob/master/final_model.py, PyTorch
https://github.com/ykukkim/MLevent/blob/master/lightningtest.py PyTorch Lightning
However, I get the following error and it seems that this is to do with IterableDataset. As my datasets are imbalance, meaning that I do not have a constant length of the dataset as well as there are more 0’s than 1’s, approximately 100:1, hence I need to penalise the 0’s by multiplying it with an arbitrary number.
I guess that this issue has been raised a few times, and I am not too sure whether there is a general fix or I have to play around with my dataset.
Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py", line 1434, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/YKK/Documents/GitHub/mlevent/lightningtest.py", line 201, in <module>
    main(hparams)
  File "/Users/YKK/Documents/GitHub/mlevent/lightningtest.py", line 182, in main
    trainer.fit(model)
  File "/Users/YKK/anaconda3/envs/LMBTrain/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 707, in fit
    self.run_pretrain_routine(model)
  File "/Users/YKK/anaconda3/envs/LMBTrain/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 771, in run_pretrain_routine
    self.get_dataloaders(ref_model)
  File "/Users/YKK/anaconda3/envs/LMBTrain/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py", line 200, in get_dataloaders
    self.init_train_dataloader(model)
  File "/Users/YKK/anaconda3/envs/LMBTrain/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py", line 79, in init_train_dataloader
    self.val_check_batch = int(self.num_training_batches * self.val_check_interval)
**OverflowError: cannot convert float infinity to integer**