On the relationship between Result and Callback monitor
💬 Discussion
Since I started using Lightning, I have noticed many users, as well as myself, having trouble with the way metrics are saved for callbacks by the Result class.
The current design of having only {early_stop_on,checkpoint_on} forces different callbacks to use the same monitor. I believe that wanting different monitors for EarlyStopping, ModelCheckpoint, and LearningRateLogger is a very common use case. This is something we will also need to address if we ever want to support multiple ModelCheckpoint callbacks.
This is also unintuitive for new users since most callbacks include a monitor parameter which becomes completely useless when the Result object is used. On the other hand, using a Result object is the new recommended way to structure {training,validation,test}_step().
Additionally, the approach is not future-proof since we would need to update lightning to include, say, batchnorm_on or whatever new technique that becomes widespread.
I believe this is an important issue that needs a solution.
Requirements:
We could use this issue to have a productive discussion about how the final design should look and what are the necessary requirements to fit most use cases:
To start, I would say:

Have it be easy to understand: avoid the conflict between monitor in Callbacks and  {early_stop_on,checkpoint_on} in Result
Allow any number of different monitors to be used simultaneously
Allow a callback monitor to be changed on-the-run

Own proposal

Remove {early_stop_on,checkpoint_on}
Add callback to result.log() function (or add result.callback() -  both would have the same functionality) for users to save metrics on-the-run

train_result.log(
    "train_loss", train_loss,
    on_step=False, on_epoch=True,
    prog_bar=True, callback=True
)
...
...
# current behaviour would override `train_result`s checkpoint_on
# if it is set also in `eval_result`. This causes some limitations.
# `callback=True` here should not override `train_loss`
# in `train_result` but keep both.
eval_result.log(
    "valid_loss", valid_loss,
    on_step=False, on_epoch=True,
    prog_bar=True, callback=True
)
# note: prog_bar should also be removed from `.log()`
# and the ProgressBar class should instead have a `--monitor` parameter.

Have each callback set its monitor via --monitor
Set --monitor type to be Optional[Union[str, Callable[..., str]] so the following is possible:

# basic behaviour
pl.callbacks.ModelCheckpoint(
    monitor="valid_accuracy",
    mode="max"
)

# Can change monitor on-the-run
# Note: maybe trainer should be injected into the callback monitor function
pl.callbacks.EarlyStopping(
    monitor=lambda: "train_loss" if trainer.current_epoch <= 20 else "val_loss",
    mode="min"
)

# Could even have
complex_monitor = VeryComplexClassWhichDoesSomethingVeryFancyWithState()
assert callable(complex_monitor)
pl.callbacks.LearningRateLogger(monitor=complex_monitor)
# this would cover a big chunk of usecases
I am positive you guys have other great ideas.

cc: @williamFalcon @rohitgr7
#2976 (previous discussion)
#3243 (duplicate)
#2908 (related, would require these improvements)
#3254 (related, could use any callback_metric)
#3291
https://forums.pytorchlightning.ai/t/does-evalresults-also-work-with-early-stopping
Probably missing other related issues. Feel free to tag