Logging with gradient accumulation & DDP
I have got two questions about logging if using gradient accumulation and DDP:


Is there a way to average logged values across the accumulated batches? Logging in training step simply as follows:
def validation_step(self, batch, batch_idx):
    ...
    self.log('val_loss', outputs.loss, sync_dist=True)
    return loss
with accumulate_grad_batches=N does not work and logs N values each 50*N steps instead of a single average value each 50 steps.


It is recommended to use sync_dist=True while using DDP. However, when logging text, images, etc., I would like to use something like:
self.logger.experiment.log({'loss' : ..., 'text' :...}, step=self.global_step)
but log it only on one device. Is it possible?


Note that I am using the W&B logger.