Speed Drastically Decrease Under Horovod
🐛 Bug

If the backend is set to horovod, training speed will drop drastically.
To Reproduce
Steps to reproduce the behavior:
I run the training job in docker, and the Dockerfile is shown below:
FROM nvidia/cuda:10.1-devel-ubuntu18.04

# TensorFlow version is tightly coupled to CUDA and cuDNN so it should be selected carefully
ENV TENSORFLOW_VERSION=2.1.0
ENV PYTORCH_VERSION=1.5.1
ENV TORCHVISION_VERSION=0.5.0
ENV CUDNN_VERSION=7.6.5.32-1+cuda10.1
ENV NCCL_VERSION=2.4.8-1+cuda10.1
ENV MXNET_VERSION=1.6.0

# Python 3.6 is supported by Ubuntu Bionic out of the box
ARG python=3.6
ENV PYTHON_VERSION=${python}

# Set default shell to /bin/bash
SHELL ["/bin/bash", "-cu"]

RUN apt-get update && apt-get install -y --allow-downgrades --allow-change-held-packages --no-install-recommends \
        build-essential \
        cmake \
        g++-4.8 \
        git \
        curl \
        vim \
        wget \
        ca-certificates \
        libcudnn7=${CUDNN_VERSION} \
        libnccl2=${NCCL_VERSION} \
        libnccl-dev=${NCCL_VERSION} \
        libjpeg-dev \
        libpng-dev \
        python${PYTHON_VERSION} \
        python${PYTHON_VERSION}-dev \
        python${PYTHON_VERSION}-distutils \
        librdmacm1 \
        libibverbs1 \
        ibverbs-providers

RUN ln -s /usr/bin/python${PYTHON_VERSION} /usr/bin/python

RUN curl -O https://bootstrap.pypa.io/get-pip.py && \
    python get-pip.py && \
    rm get-pip.py

# Install TensorFlow, Keras, PyTorch and MXNet
RUN pip install future typing
RUN pip install numpy \
        tensorflow-gpu==${TENSORFLOW_VERSION} \
        h5py
        # keras \

# RUN pip install https://download.pytorch.org/whl/cu101/torch-${PYTORCH_VERSION}-$(python -c "import wheel.pep425tags as w; print('-'.join(w.get_supported(None)[0][:-1]))")-linux_x86_64.whl \
# https://download.pytorch.org/whl/cu101/torchvision-${TORCHVISION_VERSION}-$(python -c "import wheel.pep425tags as w; print('-'.join(w.get_supported(None)[0][:-1]))")-linux_x86_64.whl
RUN pip install mxnet-cu101==${MXNET_VERSION}
COPY backbones backbones
RUN cd backbones && pip install -r requirements.txt


# Install Open MPI
RUN mkdir /tmp/openmpi && \
    cd /tmp/openmpi && \
    wget https://www.open-mpi.org/software/ompi/v4.0/downloads/openmpi-4.0.0.tar.gz && \
    tar zxf openmpi-4.0.0.tar.gz && \
    cd openmpi-4.0.0 && \
    ./configure --enable-orterun-prefix-by-default && \
    make -j $(nproc) all && \
    make install && \
    ldconfig && \
    rm -rf /tmp/openmpi

# Install Horovod, temporarily using CUDA stubs
RUN ldconfig /usr/local/cuda/targets/x86_64-linux/lib/stubs && \
    HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITH_PYTORCH=1 HOROVOD_WITH_MXNET=1 \
         pip install --no-cache-dir horovod && \
    ldconfig

# Install OpenSSH for MPI to communicate between containers
RUN apt-get install -y --no-install-recommends openssh-client openssh-server && \
    mkdir -p /var/run/sshd

# Allow OpenSSH to talk to containers without asking for confirmation
RUN cat /etc/ssh/ssh_config | grep -v StrictHostKeyChecking > /etc/ssh/ssh_config.new && \
    echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config.new && \
    mv /etc/ssh/ssh_config.new /etc/ssh/ssh_config

# Download examples
RUN apt-get install -y --no-install-recommends subversion && \
    svn checkout https://github.com/horovod/horovod/trunk/examples && \
    rm -rf /examples/.svn

WORKDIR "/examples"
My GPUs include 4 NVIDIA P100.
My training script is in /backbones, and if I run the script below, it trains smoothly:
export CUDA_VISIBLE_DEVICES=0,1,2,3
python backbones/tasks/mask_lm/trainer.py \
--data_dirs $DATA_DIR \
--bert_config_file $CONFIG_FILE \
--vocab_file $VOCAB_FILE \
--max_epochs=30 \
--gpus=1 \
--progress_bar_refresh_rate 1 \
--checkpoint_callback=True \
--val_check_interval 1.0 \
--default_root_dir=$OUTPUT_DIR \
--gradient_clip_val=5.0 \
--batch_size=32 \
--distributed_backend=ddp \
--accumulate_grad_batches 16 \
--lr 1e-4 \
--weight_decay 0.01 \
--workers 1 \
--max_length 128
If I run the script below which is using horovod, the speed will be about 1/3 of the one using ddp:
export CUDA_VISIBLE_DEVICES=0,1,2,3
horovodrun -np 4 python backbones/tasks/mask_lm/trainer.py \
--data_dirs $DATA_DIR \
--bert_config_file $CONFIG_FILE \
--vocab_file $VOCAB_FILE \
--max_epochs=30 \
--progress_bar_refresh_rate 1 \
--checkpoint_callback=True \
--val_check_interval 1.0 \
--default_root_dir=$OUTPUT_DIR \
--gradient_clip_val=5.0 \
--batch_size=32 \
--accumulate_grad_batches 16 \
--lr 1e-4 \
--weight_decay 0.01 \
--workers 1 \
--max_length 128 \
--distributed_backend horovod

Code sample

Expected behavior

The training speed under horovod will at least be around the one under ddp, not 1/3 or 1/4.