Change the way the configure_optimizers() returns
🚀 Feature
Force the method LightningModule.configure_optimizers() to return two lists.
Motivation
Right now you offer flexibility in return one of the following:
- Single optimizer
- List or Tuple - List of optimizers
- Two lists - The first list has multiple optimizers, the second a list of LR schedulers
But, that can be simplified to just returning two lists, one for optimizer and one for lr_schedulers since it does not make any sense to have that flexibility when you actually just convert it to two lists at the end of the day
Pitch
Forcing to return two lists, adds minimum overhead to researches and coders while being compliant to your codebase already.
The change aims at reducing variability while maintaining the same degree of flexibility. This will make memorizing the behavior of your framework easier and more straight forward.
It can be done as follows:
- Single optimizer - [optimizer], []
- List or Tuple - List of optimizers, [optimizer_1, optimizer_2], []
- Two lists - Multiple optimizers, single lr_scheduler - [optimizer_1, optimizer_2], [lr_scheduler]
- Two lists - Multiple optimizers, single lr_scheduler - [optimizer_1, optimizer_2], [lr_scheduler_1, lr_scheduler_2]
Alternatives
N/A
Additional context
Past Slack discussion: https://pytorch-lightning.slack.com/archives/CRBLFHY79/p1584064294116400