Allow not to freeze other optimizers' parameters
🚀 Feature

When using multiple optimizers, provide a flag which can control whether we should freeze other parameters or not (which is a 'FIX' in here)
Motivation

When implementing BicycleGAN with lightning, I'm having great trouble because it requires something like this:
opt_e.zero_grad()
opt_g.zero_grad()
loss_eg.backward(retain_graph=True)
opt_e.step()
opt_g.step()

opt_g.zero_grad()
loss_g.backward()
opt_g.step()
which means optimize two optimizers in one optimizer_step in lightning. However currently it is not possible because only one optimizer's parameters are not frozen
Pitch

Let's say we have a flag called freeze_other_parameters, then:
for opt_idx, optimizer in enumerate(self.optimizers):
    if freeze_other_parameters:
        # make sure only the gradients of the current optimizer's paramaters are calculated 
        # in the training step to prevent dangling gradients in multiple-optimizer setup.
        for param in self.get_model().parameters():
            param.requires_grad = False
        for group in optimizer.param_groups:
            for param in group['params']:
                param.requires_grad = True
Alternatives

A more elegant way might be supporting multiple optimizers optimize at one same optimizer_step, but that would break backward compatibility
Additional context

This is the original repo where I port from.