validation_epoch_end does not contain all `validation_step` outputs when using DDP
🐛 Bug
When using DDP, valdiation_epoch_end does not receive all of the outputs from validation_step. For instance, running the following script:
import pytorch_lightning as pl
import torch
from torch import nn


class Module(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(5, 1)

    def configure_optimizers(self):
        return torch.optim.Adam(self.linear.parameters())

    def training_step(self, batch, batch_idx):
        return self.linear(batch).sum()

    def validation_step(self, batch, batch_idx):
        return batch_idx

    def validation_epoch_end(self, outputs):
        print("VALIDATING", len(outputs))


if __name__ == "__main__":
    m = Module()

    datasets = [torch.rand([5]) for __ in range(100)]
    train_loader = torch.utils.data.DataLoader(datasets, batch_size=8)
    val_loader = torch.utils.data.DataLoader(datasets, batch_size=1)

    trainer = pl.Trainer(
        num_sanity_val_steps=0,
        max_epochs=1,
        accelerator="ddp_cpu",
        num_processes=4,
    )
    trainer.fit(m, train_loader, val_loader)
Will output:
VALIDATING 25                                                                                                                                                                                                                                                                                                                                                              
VALIDATING 25                                                                                                                                                                                  
VALIDATING 25                                                                                                                                                                                  
VALIDATING 25                                                                                                                                                                                  

Expected behavior
IMO, PyTorch Lightning would handle the reduction across GPUs for me  This way, any metrics computations I do in validation_epoch_end would be exactly the same, whether or not I use DDP or not.
I can do this manually by inserting:
if torch.distributed.is_initialized():
    torch.distributed.barrier()
    gather = [None] * torch.distributed.get_world_size()
    torch.distributed.all_gather_object(gather, outputs)
    outputs = [x for xs in gather for x in xs]
into the validation_epoch_end, but ideally Lightning would handle this for me.
We could also consider just running the validation_epoch_end on a single process (which would mean there aren't any more implicit reductions), but I'm less convinced that's necessary/useful.
    if torch.distributed.get_rank() == 0:
        return

Environment

PyTorch Version (e.g., 1.0): 1.7
OS (e.g., Linux): Linux
How you installed PyTorch (conda, pip, source): conda
PyTorch Lightning Version: 1.2.1