Unexpected global_steps/ accelerator behaviour
🐛 Bug
I observed some, imo,  unexpected behaviour when experimenting with different combinations of gpus/ num_processes and accelerator trainer args.
To Reproduce
I use the following bug_report_model.py. I adapted the one from pl to take accelerator, gpus, and num_processes args, only use fake train_data, and train for ten epochs.

When running it with the default args (accelerator=None, gpus=0, num_processes=1), trainer.global_step=640 as expected:

.venv ❯ python -W ignore bug_report_model.py
GPU available: True, used: False
TPU available: None, using: 0 TPU cores
Epoch 9: 100%|████████████████████████████████████████████| 64/64 [00:00<00:00, 969.92it/s, loss=718]
trainer.accelerator_backend=<pytorch_lightning.accelerators.cpu_accelerator.CPUAccelerator object at 0x7f8e39ea7dc0>
trainer.gpus=0
trainer.num_processes=1
trainer.global_step=640


When running it with  --num_processes 2 --accelerator ddp_cpu, trainer.global_step=0 which is not expected:

.venv ❯ python -W ignore bug_report_model.py --num_processes 2 --accelerator ddp_cpu
GPU available: True, used: False
TPU available: None, using: 0 TPU cores
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Epoch 9: 100%|███████████████████████████████████████████| 32/32 [00:00<00:00, 305.89it/s, loss=45.2]
trainer.accelerator_backend=<pytorch_lightning.accelerators.ddp_cpu_spawn_accelerator.DDPCPUSpawnAccelerator object at 0x7f325f6e6df0>
trainer.gpus=0
trainer.num_processes=2
trainer.global_step=0


When running it with --gpus 2, trainer.accelerator_backend is DDPSpawnAccelerator and trainer.global_step=0 which is not expected:

.venv ❯ python -W ignore bug_report_model.py --gpus 2
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
Epoch 9: 100%|██████████████████████████████████████████| 32/32 [00:00<00:00, 284.00it/s, loss=0.159]
trainer.accelerator_backend=<pytorch_lightning.accelerators.ddp_spawn_accelerator.DDPSpawnAccelerator object at 0x7fa00deafe50>
trainer.gpus=2
trainer.num_processes=2
trainer.global_step=0


When running it with  --gpus 2 --accelerator ddp, trainer.global_step=320 which makes sense due to ddp, but changes logging behaviour and max_steps calculations:

.venv ❯ python -W ignore bug_report_model.py --gpus 2 --accelerator ddp
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Epoch 9: 100%|███████████████████████████████████████████| 32/32 [00:00<00:00, 233.42it/s, loss=0.19]
trainer.accelerator_backend=<pytorch_lightning.accelerators.ddp_accelerator.DDPAccelerator object at 0x7f1b88743370>
trainer.gpus='0,1'
trainer.num_processes=2
trainer.global_step=320
trainer.accelerator_backend=<pytorch_lightning.accelerators.ddp_accelerator.DDPAccelerator object at 0x7fb45ae2ee80>
trainer.gpus=2
trainer.num_processes=2
trainer.global_step=320

Expected behavior
Regarding 2.:  trainer.global_step should be 320 or 640?
Regarding 3.:  trainer.accelerator_backend should be DDPAccelerator   as this is the recommended one for multi GPU training?
Regarding 4.: Would it be possible and/ or make sense to have total_steps that multplies global_steps and gpus that is used for max_steps and logging?
Environment
* CUDA:
        - GPU:
                - GeForce RTX 2080 Ti
                - GeForce RTX 2080 Ti
                - GeForce RTX 2080 Ti
                - GeForce RTX 2080 Ti
        - available:         True
        - version:           10.2
* Packages:
        - numpy:             1.19.4
        - pyTorch_debug:     False
        - pyTorch_version:   1.7.1
        - pytorch-lightning: 1.1.2
        - tqdm:              4.55.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                - ELF
        - processor:         x86_64
        - python:            3.8.6
        - version:           #51~18.04.1-Ubuntu SMP Sat Sep 5 14:35:50 UTC 2020