Multi-GPU (dp) on AWS p2.8xlarge instance
I don't think the AWS instance is the problem, since the model dies on the first forward pass. Here is the error:

16:17:51
Traceback (most recent call last):

16:17:51
File "/Siamese_BERT_blogpost/train.py", line 107, in <module>

16:17:51
trainer.fit(model)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 348, in fit

16:17:51
self.dp_train(model)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/dp_mixin.py", line 104, in dp_train

16:17:51
self.run_pretrain_routine(model)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 455, in run_pretrain_routine

16:17:51
self.evaluate(model, self.get_val_dataloaders(), self.nb_sanity_val_steps, self.testing)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop_mixin.py", line 50, in evaluate

16:17:51
test)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop_mixin.py", line 174, in evaluation_forward

16:17:51
output = model(*args)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__

16:17:51
result = self.forward(*i

16:17:51
wandb: Waiting for W&B process to finish, PID 162

16:17:51
nput, **kwargs)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/pt_overrides/override_data_parallel.py", line 65, in forward

16:17:51
outputs = self.parallel_apply(replicas, inputs, kwargs)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/pt_overrides/override_data_parallel.py", line 69, in parallel_apply

16:17:51
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/pt_overrides/override_data_parallel.py", line 199, in parallel_apply

16:17:51
raise output

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/pt_overrides/override_data_parallel.py", line 165, in _worker

16:17:51
output = module.validation_step(*input, **kwargs)

16:17:51
File "/Siamese_BERT_blogpost/wrapper.py", line 42, in validation_step

16:17:51
out = self.forward(batch)

16:17:51
File "/Siamese_BERT_blogpost/wrapper.py", line 35, in forward

16:17:51
return self.siamese(batch)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py",wandb: Program failed with code 1. Press ctrl-c to abort syncing.

16:17:51
line 541, in __call__

16:17:51
result = self.forward(*input, **kwargs)

16:17:51
File "/Siamese_BERT_blogpost/models.py", line 46, in forward

16:17:51
premise = self.language_model(premise)[0]

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__

16:17:51
result = self.forward(*input, **kwargs)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 735, in forward

16:17:51
embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__

16:17:51
result = self.forward(*input, **kwargs)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 186, in forward

16:17:51
inputs_embeds = self.word_embeddings(input_ids)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__

16:17:51
result = self.forward(*input, **kwargs)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 114, in forward

16:17:51
self.norm_type, self.scale_grad_by_freq, self.sparse)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1484, in embedding

16:17:51
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

16:17:51
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:400

16:17:55
wandb: Syncing 6 W&B file(s) and 0 media file(s)

16:17:56
wandb: - 0.01MB of 0.01MB uploaded wandb: \ 0.01MB of 0.01MB uploaded wandb: | 0.01MB of 0.01MB uploaded wandb: / 0.01MB of 0.01MB uploaded wandb: - 0.01MB of 0.01MB uploaded wandb: \ 0.01MB of 0.01MB uploaded wandb: | 0.01MB of 0.01MB uploaded wandb:

16:17:56
wandb: Synced vague-bush-37: https://app.wandb.ai/laksh/Siamese_SNLI/runs/7uabsn91