Why do some metrics require `num_classes=1` for binary classification?
❓ Why do some metrics require num_classes=1 for binary classification?
What is your question?
Why do some metrics require the argument num_classes=1 for binary classification (and some don't) to give the correct results?
I find it rather unintuitively to calculate Recall/Precision/F1 with the argument num_classes=1 for a binary classification, whereas e.g. ConfusionMatrix requires num_classes=2 in the same situation.
Furthermore, using Recall/Precision/F1 with num_classes=2 for a binary classification gives wrong results - so this also might be considered a bug-report.
It took me quite some time to figure out, why calculated metrics are different from what I calculated by hand from the confusion matrix.
Code
import torch
from pytorch_lightning import metrics

# example data
preds = [0] * 200 + [1] * 30 + [0] * 10 + [1] * 20
targets = [0] * 200 + [1] * 30 + [1] * 10 + [0] * 20

preds = torch.tensor(preds)
targets = torch.tensor(targets)

# define method for printing metrics


def _print_some_metrics(preds, targets, num_classes):
    precision = metrics.classification.Precision(num_classes=num_classes)
    recall = metrics.classification.Recall(num_classes=num_classes)
    f1 = metrics.classification.F1(num_classes=num_classes)

    accuracy = metrics.classification.Accuracy()
    avg_precision = metrics.classification.AveragePrecision(
        num_classes=1)
    confusion_matrix = metrics.ConfusionMatrix(num_classes=2)

    # print results
    print("Precision:\n{}\n".format(precision(preds, targets)))
    print("Recall:\n{}\n".format(recall(preds, targets)))
    print("F1:\n{}\n".format(f1(preds, targets)))

    print("AVG Precision:\n{}\n".format(avg_precision(preds, targets)))
    print("Accuracy:\n{}\n".format(accuracy(preds, targets)))
    print("ConfMat:\n{}\n".format(confusion_matrix(preds, targets)))


_print_some_metrics(preds, targets, num_classes=1)
_print_some_metrics(preds, targets, num_classes=2)
Results in

$ _print_some_metrics(preds, targets, num_classes=1)
Precision:
0.6000000238418579
Recall:
0.75
F1:
0.6666666865348816
AVG Precision:
0.48846155405044556
Accuracy:
0.8846153616905212
ConfMat:
tensor([[200.,  20.],
[ 10.,  30.]])
$ _print_some_metrics(preds, targets, num_classes=2)
Precision:
0.8846153616905212
Recall:
0.8846153616905212
F1:
0.8846153616905212
AVG Precision:
0.48846155405044556
Accuracy:
0.8846153616905212
ConfMat:
tensor([[200.,  20.],
[ 10.,  30.]])

As one can see, Precision/Recall/F1 give different (wrong) results when setting num_classes=2 in a binary classification.
AveragePrecision doesn't even work with the binary usecase when setting num_classes=2 whereas ConfusionMatrix doesn't work when setting num_classes=1.
I wonder if there is a specific reason why one would set num_classes=1 in a binary classification (where actually 2 classes exist).
Wouldn't it be more straightforward to set num_classes=2 for binary classification for all metrics?