"Advanced" profiler not working
🐛 Bug
Hello Guys,
I am having issues with the advanced profiling option in lightning. Here is the colab file documenting the issue on a simple model.
The issue happens, whenever I set stochastic_weight_avg=True.
Also, I have my complicated setup where I faced issue regarding Advanced profiler but it was a different one. When I tried to reproduce it, I faced the above mentioned issue. Thus, I am just mentioning my actual error over here.
Here is how I call my trainer -
trainer = pl.Trainer.from_argparse_args(args,
                                            callbacks=[LoggingCallback(), checkpoint_callback],
                                            profiler="advanced",
                                            )

And here is the error stack -
Traceback (most recent call last):
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/nzb0040/.pyenv/versions/intersection_train/lib/python3.8/site-packages/pytorch_lightning/profiler/profilers.py", line 71, in profile
    yield action_name
  File "/home/nzb0040/.pyenv/versions/intersection_train/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1102, in call_hook
    trainer_hook(*args, **kwargs)
  File "/home/nzb0040/.pyenv/versions/intersection_train/lib/python3.8/site-packages/pytorch_lightning/trainer/callback_hook.py", line 35, in on_before_accelerator_backend_setup
    callback.on_before_accelerator_backend_setup(self, model)
  File "/home/nzb0040/.pyenv/versions/intersection_train/lib/python3.8/site-packages/pytorch_lightning/callbacks/swa.py", line 142, in on_before_accelerator_backend_setup
    self._average_model = deepcopy(pl_module)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'Profile' object
Exception ignored in: <function AdvancedProfiler.__del__ at 0x7f045ad834c0>
Traceback (most recent call last):
  File "/home/nzb0040/.pyenv/versions/intersection_train/lib/python3.8/site-packages/pytorch_lightning/profiler/profilers.py", line 280, in __del__
    if self.output_file:
AttributeError: 'AdvancedProfiler' object has no attribute 'output_file'