Behaviour of accumulate_gradients and multi-gpu
Training setup:
2 GPUs on a single machine running in DDP mode. If I use a batch size of 16 and accumulate gradients=2, how does lightning handle this?
Possibility 1:
GPU1 processes one batch of size 16.
GPU2 processes one batch of size 16.

average gradients from GPU1 and GPU2 and apply weight update.

or
Possibility 2
GPU1 processes one batch of size 16.
GPU1 processes another batch of size 16
GPU1 averages the gradients of the two batches.

GPU2 processes one batch of size 16.
GPU2 processes another batch of size 16
GPU2 averages the gradients of the two batches.

average the averaged gradients from GPU1 and GPU2 and apply weight update

Which of the two ways does lightning handle this under DDP? I am asking because in the first scenario the effective batch size is 32 and in the second scenario the effective batch size is 64.