Turn off torch profilers for faster training
🚀 Feature
Motivation
PyTorch by default does not disable the autograd and other profilers while training there are a lot of them listed in one of the talks here.
To enhance speed while training it is recommended to turn them off manually.
Pitch
Since this feature should be on by default as it gives a good stack trace for errors. Can we provide a utility function as we did for seeding everything?
from utils.profiler import profiler
profiler(state=off)   # to turn off
profiler(state=on)  # to turn on

Something like this will allow ease of toggling the profiler.
Alternatives
Already users can use a context manager using pytorch (but nobody does sadly)
with torch.autograd.profiler.profile(enabled=False) as prof:
    PyTorch code

Alternatively, it can be an argument in trainer itself. But I guess this is not good.
Trainer(profiler=off)

Additional context
Do verify if we are following other recommend guideliness by NVIDIA
szymon_migacz-pytorch-performance-tuning-guide.pdf