What sync_dist_op can be set in self.log()?
Hi, the document shows sync_dist_op= "mean" by default.
But in my case, I do training in ddp with 2 GPUs and I want to record the number of true positive samples in validation dataset. So sync_dist_op should be set like "sum" which I didn't see in the source code and the document.
Another thing is that I want to record validation results on epoch, so I aggregated the results in val_epoch_end() and call self.log("step", self.current_epoch) to overwrite steps. Then if I set syn_dist=True in self.log(), the following error occurred.
    self.log('Val loss', loss, sync_dist=True)
  File "/shared_home/r08922129/anaconda3/envs/tabfact_pepper/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py", line 281, in log
    return sync_ddp_if_available(tensor, group, reduce_op)
  File "/shared_home/r08922129/anaconda3/envs/tabfact_pepper/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py", line 124, in sync_ddp_if_available
    return sync_ddp(result, group=group, reduce_op=reduce_op)
  File "/shared_home/r08922129/anaconda3/envs/tabfact_pepper/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py", line 156, in sync_ddp
    self._current_dataloader_idx,
  File "/shared_home/r08922129/anaconda3/envs/tabfact_pepper/lib/python3.7/site-packages/pytorch_lightning/core/step_result.py", line 142, in log
    torch.distributed.all_reduce(result, op=reduce_op, group=group, async_op=False)
  File "/shared_home/r08922129/anaconda3/envs/tabfact_pepper/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 954, in all_reduce
    value = sync_fn(value, group=sync_dist_group, reduce_op=sync_dist_op)
  File "/shared_home/r08922129/anaconda3/envs/tabfact_pepper/lib/python3.7/site-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 344, in sync_tensor
    work = _default_pg.allreduce([tensor], opts)
RuntimeError: Tensors must be CUDA and dense
    return sync_ddp_if_available(tensor, group, reduce_op)
  File "/shared_home/r08922129/anaconda3/envs/tabfact_pepper/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py", line 124, in sync_ddp_if_available
    return sync_ddp(result, group=group, reduce_op=reduce_op)
  File "/shared_home/r08922129/anaconda3/envs/tabfact_pepper/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py", line 156, in sync_ddp
    torch.distributed.all_reduce(result, op=reduce_op, group=group, async_op=False)
  File "/shared_home/r08922129/anaconda3/envs/tabfact_pepper/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 954, in all_reduce
    work = _default_pg.allreduce([tensor], opts)
RuntimeError: Tensors must be CUDA and dense


The loss in the first line self.log('Val loss', loss, sync_dist=True) is the result of BCEWithLogitsLoss.
What is weird is that I could set sync_dist=True without that error if I call self.log() in validation_step() rather than in validation_epoch_end().