Get correct metrics for an epoch with DDP
❓ Questions and Help
What is your question?
Before #2528 is fixed to allow calculating metrics for an entire epoch (for example, average precision) with DDP, I am curious if there is a workaround at the moment, given the current metrics is calculated on each GPU before syncing data from each GPU?
Take average precision for example with 4 GPUs. First on each GPU based on all batches of the epoch an average precision is calculated. Then 4 average precision from 4 GPUs are averaged again to get the final average precision. This could cause sometime quite big discrepancy compared to the true average precision of the epoch.
Is it possible to sync all data for the entire epoch before calculating the metric, given what we have at the moment, while waiting for fixes in the near future? Thanks.
BTW: really incredible work on this! I already became a fan and in the process of migrating code from Pytorch to Lightning.
Code
   
What have you tried?
I have google/read all official documents and most of GitHub issues
What's your environment?

OS: [e.g. iOS, Linux, Win]. Linux
Packaging [e.g. pip, conda]. Pip
Version [e.g. 0.5.2.1]. 0.8.5