Pass output from all val dataloaders to `validation_end`
Is your feature request related to a problem? Please describe.
I'm working on a retrieval problem. To compute validation scores, I'd like to compute feature vectors for a query set and a catalog set, then measure information retrieval metrics (precision, recall, etc) of the query set against the catalog set.
To do this, I'd like to return the two datasets from val_dataloader (already possible), then use validation_step to accumulate all the feature vectors for each dataset (already possible). Once I have all the feature vectors, I need some way to bring the vectors from the two datasets together so that I can compute retrieval metrics. Right now, validation_end processes the outputs of multiple validation dataloaders separately.
Describe the solution you'd like
Change the behavior of validation_end in cases where there are multiple validation dataloaders. I propose that it only be called once, but that it be passed a list of outputs dicts, one from each dataloader. Using it would look something like this:
def validation_end(self, outputs):
    query_output, catalog_output = outputs
    query_vectors = torch.cat([o['vectors'] for o in query_output])
    query_labels = torch.cat([o['labels'] for o in query_output])
    catalog_vectors = torch.cat([o['vectors'] for o in catalog_output])
    catalog_labels = torch.cat([o['labels'] for o in catalog_output])
    return retrieval_metrics(
        query_vectors,
        query_labels,
        catalog_vectors,
        catalog_labels
    )
Describe alternatives you've considered

Breaking outside of lightning entirely to do validation
Concatenating query and catalog datasets, and somehow adding a flag to indicate which dataset a given example came from