MetricList for moving entire list of metrics onto GPU
🚀 MetricList
MetricList serves the same function for Metrics in PL as nn.ModuleList in PyTorch for Modules.
As such, they take care of flexibly moving all Metrics and their states onto GPU or CPU.
Motivation

As a dynamic inference researcher, I want to be able to test multiple setups for a single model (f.e. executing my model at different depths or widths).
For each of these setups, I want to be able to track mostly the same metrics (f.e. Classification Accuracy) and making separate PL Metrics for each one of these metrics is tedious.
Moreover, I might want to specify the amount of setups with a parameter which makes constructing them in a list format the most elegant solution.

Pitch

In regular PyTorch they introduced nn.ModuleList especially for this purpose (f.e. swapping out different FC layers for different setups)

Alternatives

Evaluating if any of the internal variables is a list that might contain metrics does not seem like good design to me, but has the advantages that the user will not need to know of the existence of MetricList.  Maybe we can add a warning or error to use a MetricList when any errors as per "Additional Context" happens?

Additional context

Using regular lists that hold metrics and updating them while training on GPU will now keep metric states on CPU leading to errors, see a BoringModel example here, that as expected returns:   Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
Happy to work on a PR for this feature, but wanted to know your thoughts.