Horovod with native 16 precision not working
🐛 Bug

To Reproduce
Steps to reproduce the behavior:

using precision=16 with distributed_backend=horovod

Traceback (most recent call last):
  File "/workspace/main_lightning.py", line 500, in <module>
    main(hyperparams)
  File "/workspace/main_lightning.py", line 492, in main
    trainer.fit(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 1068, in fit
    results = self.horovod_train(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/distrib_parts.py", line 213, in horovod_train
    model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/core/lightning.py", line 954, in configure_apex
    model, optimizers = amp.initialize(model, optimizers, opt_level=amp_level)

Code sample
trainer = Trainer(
        precision=16,
        gpus=1,
        distributed_backend="horovod")

Environment

PyTorch Version: 1.6.0+cu101
How you installed PyTorch: pip