trainer.test after fp16 training with apex
Summary
In the current huggingface examples/seq2seq/finetune.py,
trainer.test fails in fp16 mode with torch 1.5.1.

Nowhere in the huggingface code is model.half called.
Models maybe saved to disk in either fp16 or fp32 format, but since we are resuming from a pl checkpoint, I think pl is controlling the saving and loading here.
Bonus: this test will hang if you pass gpus=2 by switching this line.

To reproduce
You must be on a machine with cuda installed
git clone git@github.com:huggingface/transformers.git
cd transformers
pip install -e .
pip install -e .[examples]  # installs pytorch-lightning==0.8.5
git checkout broken-fp16-test
RUN_SLOW=1 USE_CUDA=1 pytest examples/seq2seq/test_bash_script.py
Small Traceback
 RuntimeError: Found param model.model.shared.weight with type torch.cuda.HalfTensor, expected torch.cuda.FloatTensor.
When using amp.initialize, you do not need to call .half() on your model
before passing it, no matter what optimization level you choose.
Full Traceback
(failure from this line)
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /home/shleifer/transformers_fork, inifile: pytest.ini
plugins: forked-1.1.3, xdist-1.31.0, requests-mock-1.8.0
collected 1 item

examples/seq2seq/test_bash_script.py Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.

Defaults for this optimization level are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic

Validation sanity check: 100%|██████████| 2/2 [00:00<00:00,  2.76it/s]
                                                                      

Training: 0it [00:00, ?it/s]
Training:   0%|          | 0/125 [00:00<?, ?it/s]
Epoch 1:   0%|          | 0/125 [00:00<?, ?it/s] Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0

Epoch 1:   1%|          | 1/125 [00:00<00:10, 11.29it/s, loss=12.438, v_num=219]
...                                                           �[A
Epoch 2:  74%|███████▍  | 93/125 [00:14<00:04,  6.55it/s, loss=12.427, v_num=219]
# DONE TRAINING
Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.

Defaults for this optimization level are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
F

=================================== FAILURES ===================================
______________________ test_train_mbart_cc25_enro_script _______________________

    @slow
    @pytest.mark.skipif(not CUDA_AVAILABLE, reason="too slow to run on CPU")
    def test_train_mbart_cc25_enro_script():
        data_dir = "examples/seq2seq/test_data/wmt_en_ro"
        env_vars_to_replace = {
            "$MAX_LEN": 200,
            "$BS": 4,
            "$GAS": 1,
            "$ENRO_DIR": data_dir,
            "facebook/mbart-large-cc25": MBART_TINY,
        }
    
        # Clean up bash script
        bash_script = Path("examples/seq2seq/train_mbart_cc25_enro.sh").open().read().split("finetune.py")[1].strip()
        bash_script = bash_script.replace("\\\n", "").strip().replace("$@", "")
        for k, v in env_vars_to_replace.items():
            bash_script = bash_script.replace(k, str(v))
        output_dir = tempfile.mkdtemp(prefix="output")
    
        if CUDA_AVAILABLE:
            gpus = 1  # torch.cuda.device_count()
        else:
            bash_script = bash_script.replace("--fp16", "")
            gpus = 0
    
        testargs = (
            ["finetune.py"]
            + bash_script.split()
            + [f"--output_dir={output_dir}", f"--gpus={gpus}", "--learning_rate=3e-1"]
        )
        with patch.object(sys, "argv", testargs):
            parser = argparse.ArgumentParser()
            parser = pl.Trainer.add_argparse_args(parser)
            parser = SummarizationModule.add_model_specific_args(parser, os.getcwd())
            args = parser.parse_args()
            # assert args.gpus == gpus THIS BREAKS
            # args.gpus = gpus
>           model = main(args)

examples/seq2seq/test_bash_script.py:71: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
examples/seq2seq/finetune.py:392: in main
    trainer.test()
../miniconda3/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py:1281: in test
    results = self.__test_using_best_weights(ckpt_path, test_dataloaders)
../miniconda3/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py:1321: in __test_using_best_weights
    results = self.fit(model)
../miniconda3/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py:1003: in fit
    results = self.single_gpu_train(model)
../miniconda3/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_parts.py:182: in single_gpu_train
    model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
../miniconda3/envs/nb/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py:1006: in configure_apex
    model, optimizers = amp.initialize(model, optimizers, opt_level=amp_level)
../miniconda3/envs/nb/lib/python3.7/site-packages/apex/amp/frontend.py:358: in initialize
    return _initialize(models, optimizers, _amp_state.opt_properties, num_losses, cast_model_outputs)
../miniconda3/envs/nb/lib/python3.7/site-packages/apex/amp/_initialize.py:171: in _initialize
    check_params_fp32(models)
../miniconda3/envs/nb/lib/python3.7/site-packages/apex/amp/_initialize.py:87: in check_params_fp32
    name, param.type()))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

msg = 'Found param model.model.shared.weight with type torch.cuda.HalfTensor, expected torch.cuda.FloatTensor.\nWhen using a...alize, you do not need to call .half() on your model\nbefore passing it, no matter what optimization level you choose.'

    def warn_or_err(msg):
        if _amp_state.hard_override:
            print("Warning:  " + msg)
        else:
>           raise RuntimeError(msg)
E           RuntimeError: Found param model.model.shared.weight with type torch.cuda.HalfTensor, expected torch.cuda.FloatTensor.
E           When using amp.initialize, you do not need to call .half() on your model
E           before passing it, no matter what optimization level you choose.

../miniconda3/envs/nb/lib/python3.7/site-packages/apex/amp/_amp_state.py:32: RuntimeError
=========================== short test summary info ============================
FAILED examples/seq2seq/test_bash_script.py::test_train_mbart_cc25_enro_script
============================== 1 failed in 38.75s ==============================