can i run multiple ddp jobs on single node
I am running on a 14 core, 7 gpu machine. Ubuntu 18.04.2LTS, python 3.6.8, lightning 0.5.3.2, no virtual environment, no SLURM.
I have moved a tried and true model to ddp. It works great in all scenarios, including ddp as a single invocation.
I cannot succesfully start a second one, unfortunately. I get the following failure:
  File "/home/seth/.local/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon)
RuntimeError: Address already in use

This second job is running on different gpus and has a different log path.
After a brief investigation, it seems to me that the second job is trying to use the same master address as the first. I did not see any way to alter this with pytorch-lightning, though it seems straightforward in pytorch.
My questions are:
Can I run multiple simultaneous ddp jobs on the same node with different GPUs?
If so, how?
Thanks