How to gather results during inference in ddp
❓ Questions and Help
Hi, I am using multiple gpus and ddp mode for model inference. I am wondering how to gather the results from all distributed processes and save them into one file in the test_epoch_end. My code looks like this:
Code
class PLModel(pl.LightningModule):
    def on_test_epoch_start(self):
        self.text_predictions = []
    
    def test_step(self, batch, batch_idx):
         pred_ids = model(batch)
         pred_texts = ids_to_texts(pred_ids) # some function that converts predicted ids to texts
         self.text_predictions.extend(pred_texts)

    def test_epoch_end(self, outputs):
        # here I want to know how to gather the text_predictions from all processes

        # after gathering
        df_text_predictions = pandas.DataFrame{'predictions': self.text_predictions}
        df_text_predictions.to_csv('predictions.csv', index=False)
What have you tried?
I am able to save the results into separate files in each distributed process. But I have to do post-processing to put all the files together. I want to know how to gather the results automatically so that I don't need to run another script for post-processing.