Batch being moved to gpu repeatedly with multiple optimizers and single gpu training
If you have multiple optimizers, then transfer_batch_to_gpu winds up getting called once per opt_idx, and the batch is copied each time via copy.copy(batch) in training_forward. Why copy the batch when there is only a single gpu? By removing the copy.copy() my GAN model moves from 8.53it/s to 9.25it/s. Pretty significant speedup.