Mismatching stats logged while training and while evaluating with the saved model
❓ Questions and Help
Before asking:

Try to find answers to your questions in the Lightning Forum!
Search for similar issues.
Search the docs.


What is your question?
I manually print the training accuracy after every epoch.
But get different accuracy when evaluate the training set using the saved model at that epoch.
Code
`
def training_step(self, batch, batch_idx):
    logits, softmax_logits = self.forward(**batch)
    loss, prediction_label_count = self.loss_function(logits, batch["labels"])
    accuracy = self.compute_accuracy(logits, batch["labels"])
    return {"loss": loss,
            "prediction_label_count": prediction_label_count,
            "accuracy": accuracy,
            "log": {"train_loss": loss, "training_accuracy": accuracy}}

`
`
def training_epoch_end(self, outputs_of_training_steps):
    avg_loss = torch.cat([x['loss'].view(-1) for x in outputs_of_training_steps]).mean()
    training_accuracy = torch.cat([x['accuracy'].view(-1) for x in outputs_of_training_steps]).mean()

    print("\nTraining epoch stats below:")

    for label in range(parameters["num_labels"]):
        pred_freq = torch.cat(
            [x['prediction_label_count'][label].view(-1) for x in outputs_of_training_steps]).sum()
        print("Frequency of prediction of label {label} is : {freq}".format(label=label, freq=pred_freq))


    if (parameters["use_weighted_CE"] == True):
        print("weights in this epoch were: ", self.weights)

    print("training loss: ", avg_loss)
    print("training accuracy: ", training_accuracy)

    print("***Training epoch stats finish***\n")

    log = {"train_accuracy": training_accuracy}
    return {"train_accuracy": training_accuracy, 'log': log}

`

What have you tried?
3 class classification problem, I'm using a "batch_size" of 32, 8 gpus, and training and validation sets are of size 5k each.
This is what I get
Epoch 0: 100%|██████████| 314/314 [01:50<00:00,  2.85it/s, loss=0.751, v_num=7143817]
Validation epoch stats below:157/157 [00:33<00:00,  5.06it/s]
Frequency of prediction of label 0 is : 2208.0
Frequency of prediction of label 1 is : 1457.0
Frequency of prediction of label 2 is : 1335.0
val accuracy:  tensor(0.6483, device='cuda:0')
val loss:  tensor(0.8154, device='cuda:0')
Validation epoch stats finish.
Epoch 0: 100%|██████████| 314/314 [01:53<00:00,  2.76it/s, loss=0.751, v_num=7143817]
Training epoch stats below:
Frequency of prediction of label 0 is : 346.25
Frequency of prediction of label 1 is : 437.0
Frequency of prediction of label 2 is : 466.75
training loss:  tensor(0.9375, device='cuda:0')
training accuracy:  tensor(0.5388, device='cuda:0')
Training epoch stats finish
But when I evaluate the training set using the saved model, I get
Frequency of prediction of label 0 is : 2148.0
Frequency of prediction of label 1 is : 1537.0
Frequency of prediction of label 2 is : 1315.0
accuracy:  tensor(0.7282, device='cuda:0')
Note: Accuracy on Val set comes the same as in the logs i.e 0.6483.
Why there is a mismatch in the accuracy on the training set
What's your environment?

OS: [e.g. iOS, Linux, Win]
Packaging [e.g. pip, conda]
Version [e.g. 0.5.2.1]