Completely overwrite validation/test block (including the batch-level loop)
🚀 Feature

Motivation

In certain cases, for example, an image+text multimodal retrieval model, the training and validation/testing logic can be very different. Specifically:

In training, for each input query, we construct the corresponding batches by sampling randomly from the dataset;
In validation/testing, for each input query, we need to loop through all candidates within the dataset

I'll be happy to go into the details, but in short, for these use cases, we often need a COMPLETELY different pipeline for validation/testing, which doesn't necessarily fit into the 'loop' as we would use in typical training scenarios.
Pitch

I would propose to have a higher-level function that could:

Overwrite the entire validation/testing logic, including the batch-level loop - basically, redefines the validation/testing behaviors
Have full control over what elements go to what device (i.e., manually define .cuda() or .cpu() - this is because, for retrieval-type tasks, it is highly impractical to stuff everything in VRAM at the same time, chances are we need to take things in and out of GPUs.