Fix DDP logging
Add a global_zero_only=true flag, if false- create individual files, prefixed with machine nun
Write a logging callback that will do map reduce
Can we do this in the metrics?
Aggregate all tensors on global zero first
(might run into memory issues)
Gather each output individually in CPU memory
We want to preserve the fact that logging is at 0