lightning and apex amp performance not improved
❓ lightning and apex amp performance not improved
Before asking:

search the issues.
search the docs.


I'm trying to use lightning and Apex amp to speed ddp training. I tried amp_level O0, O1, O2, and O3, and they use almost the same time (all around 45 minutes).

train_loader = DataLoader(dataset=train_dataset, batch_size=2, shuffle=True, num_workers=4)
val_loader = DataLoader(dataset=val_dataset, batch_size=1, shuffle=False, num_workers=4)
trainer = pl.Trainer(gpus= 8, num_nodes = 1, distributed_backend='ddp', precision = 16, amp_level = 'O1')
trainer.fit(model, train_dataloader=train_loader, val_dataloaders=val_loader)
i didn't change batchsize to be a multiple of 8 because i saw this post and my cudnn version is 7.6.5.
Thanks!
   
What have you tried?
I also tried torch.backends.cudnn.benchmark = True but got no improvement.
What's your environment?

OS: [e.g. iOS, Linux, Win] linx
Packaging [e.g. pip, conda] pip
Version [e.g. 0.5.2.1]; latest