training_epoch_end() only gives outputs from one optimizer when multiple optimizers are being used
I'm making an abstract GAN class, where I have the following code:
    def configure_optimizers(self):
        return self.g_optimizer, self.d_optimizer

    def training_step(self, batch: Tuple[Tensor, Tensor], batch_idx, optimizer_idx) -> Dict:
        X, _ = batch

        batch_size = X.shape[0]

        z = self.Z.sample((batch_size,))
        if self.on_gpu:
            z = z.cuda(X.device)

        g_X = self.G(z)

        if optimizer_idx == 0:
            # Train generator
            d_z = self.D(g_X)

            g_loss = self.generator_loss(d_z)
            step_dict = {'loss': g_loss, 'g_loss': g_loss}
            return step_dict
        if optimizer_idx == 1:
            # Train discriminator
            d_x = self.D(X)
            d_z = self.D(g_X)

            d_loss = self.discriminator_loss(d_x, d_z)
            step_dict = {'loss': d_loss, 'd_loss': d_loss}
            return step_dict
In training_epoch_end(), outputs only contains the discriminator losses. Is this a bug, or am I doing something wrong?