Iterations completing out of order (possibly) in ddp with torchelastic?
This might be bug or might be expected.  I'm running a pytorchlightning with torchelastic and ddp.  I'm noticing the iterations are being dumped out of order (below iteration 632 preceeds iteration 574).  This could be due to delays in parallel writing... or perhaps just issues in logging.  Is this expected behavior?
Validating: 60it [00:21,  3.61it/s]�[A
Epoch 26: : 632it [08:13,  1.28it/s, loss=0.111, v_num=0]

Validating: 62it [00:22,  4.62it/s]�[A
Validating: 0it [00:00, ?it/s]�[A
Epoch 26: : 572it [07:51,  1.21it/s, loss=0.111, v_num=0]

Validating: 2it [00:00, 18.62it/s]�[A
Epoch 26: : 574it [07:52,  1.22it/s, loss=0.111, v_num=0]

Running with 6 gpus in ddp.