Issue with running multiple models in PyTorch Lightning
🐛 Bug
I am developing a system which needs to train dozens of individual models (>50) using Lightning, each with their own TensorBoard plots and logs. My current implementation has one Trainer object per model and it seems like I'm running into an error when I go over ~90 Trainer objects. Interestingly, the error only appears when I run the .test() method, not during .fit().
As I just started with Lightning, I am not sure if having one Trainer/model is the best approach. However, I require individual plots from each model, and it seems that if I use a single trainer for multiple models the results get overridden.
To Reproduce
Steps to reproduce the behaviour:
1.Define more than 90 Trainer objects, each with their own model.
2. Run training for each model.
3. Run testing for each model.
4. See error
Traceback (most recent call last):
  File "lightning/main_2.py", line 193, in <module>
    main()
  File "lightning/main_2.py", line 174, in main
    new_trainer.test(model=new_model, test_dataloaders=te_loader)
  File "\Anaconda3\envs\pysyft\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1279, in test
    results = self.__test_given_model(model, test_dataloaders)
  File "\Anaconda3\envs\pysyft\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1343, in __test_given_model
    self.set_random_port(force=True)
  File "\Anaconda3\envs\pysyft\lib\site-packages\pytorch_lightning\trainer\distrib_data_parallel.py", line 398, in set_random_port
    default_port = RANDOM_PORTS[-1]
IndexError: index -1 is out of bounds for axis 0 with size 0

Code sample
Defining the Trainer objects:
for i in range(args["num_users"]):
    trainer_list_0.append(Trainer(max_epochs=args["epochs"], gpus=1, default_root_dir=args["save_path"],
                                          fast_dev_run=args["fast_dev_run"], weights_summary=None))
    trainer_list_1.append(Trainer(max_epochs=args["epochs"], gpus=1, default_root_dir=args["save_path"],
                                            fast_dev_run=args["fast_dev_run"], weights_summary=None))
    trainer_list_2.append(Trainer(max_epochs=args["epochs"], gpus=1, default_root_dir=args["save_path"],
                                            fast_dev_run=args["fast_dev_run"], weights_summary=None))

Training:
for i in range(args["num_users"]):
    trainer_list_0[i].fit(model_list_0[i], train_dataloader=dataloader_list[i],
                                      val_dataloaders=val_loader)
    trainer_list_1[i].fit(model_list_1[i], train_dataloader=dataloader_list[i],
                                        val_dataloaders=val_loader)
    trainer_list_2[i].fit(model_list_2[i], train_dataloader=dataloader_list[i],
                                        val_dataloaders=val_loader)

Testing:
for i in range(args["num_users"]):
    trainer_list_0[i].test(test_dataloaders=te_loader)
    trainer_list_1[i].test(test_dataloaders=te_loader)
    trainer_list_2[i].test(test_dataloaders=te_loader)

Expected behaviour
I expected the code to work without crashing.
Environment

PyTorch Version (e.g., 1.0): 1.4
OS (e.g., Linux): Windows 10 Pro 2004
How you installed PyTorch (conda, pip, source): conda
Python version: 3.7.6
CUDA/cuDNN version: CUDA 10.1/cuDNN 7.0
GPU models and configuration: RTX 2060 Super