strange behavior with tests in PL: tests influence each other
🐛 Bug
I observed that some tests under tests/trainer have very strange behavior. The order in which the tests are executed seems to matter and they are influencing each other!
To Reproduce

Checkout release/1.2-dev branch
Remove the predict() method in one of the accelerators (GPUAccelerator is enough)
Run py.test -v tests/trainer

You will see that a ton of unrelated tests fail, but none of them report "Accelerator has not attribute predict". Which should be the only case why a test would fail. But be prepared, it gets funnier.

Run only a single test out of all these that failed, for example:
py.test -v tests\trainer\optimization\test_optimizers.py::test_lr_scheduler_strict

And it passes! What?
Expected behavior
Tests should not have an influence on each other.
Environment

CUDA:
- GPU:
- GeForce GTX 1070
- available:         True
- version:           10.2
Packages:
- numpy:             1.18.1
- pyTorch_debug:     True
- pyTorch_version:   1.7.0
- pytorch-lightning: 1.2.0dev
- tqdm:              4.55.1
System:
- OS:                Windows
- architecture:
- 64bit
- WindowsPE
- processor:         AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
- python:            3.7.7
- version:           10.0.19041

Also on my linux node, same behavior
Additional context