Memory leak using AMP and transformers
Context can be found here: huggingface/transformers#8403
10x memory consumption increase with native amp on PL
I'm running under debugger pt15+apex side by side with pt16+native amp, huggingface/transformers#8403
and found the first issue of objects not being freed
I ruled out GradScaler I think there is a huge memory leak somewhere.
Typically this happens due to circular references, but I don't know PL codebase... so asking for help.
Oh, and I ruled out transformers  - the HF trainer using the same model code doesn't have this issue.