Add stochastic weight averaging
Looks like we need to keep two copies of the model. Let $m_r$ define the root model and $m_c$ the current model. Then at the end of each epoch $n$, we update the weights of $m_r$ using a weighted average:

Anyone interested in implementing?
maybe enable as a callback? not sure this needs a flag?
@PyTorchLightning/core-contributors