Error in Logger on epoch end when using Multiple GPUs
🐛 Bug
When using multiple GPUs with 'dp', the error RuntimeError: All input tensors must be on the same device. Received cuda:1 and cuda:0 occurs. It means the collections on epoch end would be from different device.
Expected behavior
While they might need to be on the same device, or maybe the aggregating function should be able to handle items from different device.
Environment

PyTorch Version (e.g., 1.0): 1.7.0
OS (e.g., Linux): Linux
How you installed PyTorch (conda, pip, source): pip
Python version: 3.8.0
CUDA/cuDNN version: 10.2
Any other relevant information: pytorch-lightning==1.0.8

A quick but not safe solution

modify collate_tensors function in pytorch_lightning/core/step_result.py:

def collate_tensors(items: Union[List, Tuple]) -> Union[Tensor, List, Tuple]:

    if not items or not isinstance(items, (list, tuple)) or any(not isinstance(item, Tensor) for item in items):
        # items is not a sequence, empty, or contains non-tensors
        return items

    # add the following line of code
    items = [item.type_as(items[0]) for item in items]

    if all(item.ndim == 0 for item in items):
        # all tensors are scalars, we need to stack
        return torch.stack(items)

    if all(item.ndim >= 1 and item.shape[1:] == items[0].shape[1:] for item in items):
        # we can concatenate along the first dimension
        return torch.cat(items)

    return items