How to log performance metrics in the validation step/loop?
❓ How to log performance metrics in the validation step/loop?
Before asking:

search the issues.   Done
search the docs.    Done

What is your question?
I would like to log the performance metrics of each validation batch using TestTubeLogger (I am open to using alternative loggers) as well as the loss function of each training batch. I am struggling to understand the difference between adding 'log' to the output of my validation step, and including a call to the logger itself through self.logger.experiment.log({k, v}). I have seen both methods implemented in the tutorials and docs, but I do not understand the best practices nor which method to use to achieve my end goal.

Is the call to self.logger.experiment.log({k, v}) mutually exclusive with adding 'log' to the output of a training/validation step? If not, should they be used together and in what circumstance?
Reference What have I tried (3) below; why does the validation_step method not register the entries in the 'log' dictionary outputs? Are the only validation outputs that are logged those from the validation_epoch_end method?
With all of this in mind, how can I log performance metrics such as accuracy, recall, precision, from the validation loop? What are the best practices around this?

What have I tried?


I trained the network without any calls to self.logger.experiment.log({k, v}) and without adding a 'log' key to the outputs of any training/validation step or epoch end methods. This resulted in the creation of a metrics.csv file that contained only a column for the created time and epoch number.


I added in only calls to the logger experiment such that my training and validation code looked like such:



    def training_step(self, batch, batch_nb):
        x, y = batch
        x, y = x.float(), y.float().unsqueeze(1)
        y_hat = self.forward(x)
        loss = self.loss_function(y_hat, y)
        self.logger.experiment.log({'loss': loss})
        return {
            'loss': loss
        }

    def training_epoch_end(self, outputs):
        avg_val_loss = torch.stack([x['loss'] for x in outputs]).mean()
        return {
            'avg_train_loss': avg_val_loss
        }     

    def validation_step(self, batch, batch_nb):
        x, y = batch
        x, y = x.float(), y.float().unsqueeze(1)
        y_hat = self.forward(x)
        loss = self.loss_function(y_hat, y)
        metrics = self.performance_metrics(F.softmax(y_hat), y)
        self.logger.experiment.log({'metrics': metrics['accuracy']})
        return {
            'val_loss': loss,
        }

    def validation_epoch_end(self, outputs):
        avg_val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {
            'avg_val_loss': avg_val_loss,
        }

When running trainer.fit(model) with this arrangement the first few lines of the metrics.csv look like:


I removed all calls to the logger experiment and included only 'log' dictionary keys in the outputs such that the code looked like such:


    def training_step(self, batch, batch_nb):
        x, y = batch
        x, y = x.float(), y.float().unsqueeze(1)
        y_hat = self.forward(x)
        loss = self.loss_function(y_hat, y)
        return {
            'loss': loss,
            'log': {'train_loss': loss, 'batch_nb': batch_nb}
        }

    def training_epoch_end(self, outputs):
        avg_val_loss = torch.stack([x['loss'] for x in outputs]).mean()
        return {
            'avg_train_loss': avg_val_loss,
            'log': {'avg_train_loss': avg_val_loss}
        } 

    def validation_step(self, batch, batch_nb):
        x, y = batch
        x, y = x.float(), y.float().unsqueeze(1)
        y_hat = self.forward(x)
        loss = self.loss_function(y_hat, y)
        metrics = self.performance_metrics(F.softmax(y_hat), y)
        return {
            'val_loss': loss,
            'log': {'val_loss': loss, 'metrics': metrics}
        }

    def validation_epoch_end(self, outputs):
        avg_val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {
            'avg_val_loss': avg_val_loss,
            'log': {'avg_val_loss': avg_val_loss}
        }

Upon running trainer.fit(model) the first few lines of my metrics.csv file look like:

What's your environment?

OS: Ubuntu 18.04
Packaging: Conda
Python: 3.8
Pytorch Lightning: 0.7.5

Additional content
This is how I configure the logger and trainer:

logger = pl.loggers.TestTubeLogger(
    save_dir=config['logging_params']['save_dir'],
    name=config['logging_params']['name'],
    debug=False,
    create_git_tag=False,
)

trainer = pl.Trainer(default_save_path=f'{logger.save_dir}',
	         	min_nb_epochs=1,
                	logger=logger,
                	log_save_interval=5,
	                num_sanity_val_steps=5,
	                **config['trainer_params'])

I am using a binary classifier and compute my performance metrics with the follorwing method:

    def performance_metrics(self, y_hat, y):
        '''0 is typical, 1 is novel, also works with boolean arrays'''
        counts = {'TP': 0, 'FP': 0, 'FN': 0, 'TN': 0}
        for quadrant in zip(y_hat, y):
            
            if quadrant == (1, 1): # TP
                counts['TP'] += 1
            elif quadrant == (0, 1): # FN
                counts['FN'] += 1
            elif quadrant == (1, 0): # FP
                counts['FP'] += 1
            elif quadrant == (0, 0): # TN
                counts['TN'] += 1
                
        precision = counts['TP'] / (counts['TP'] + counts['FP'])
        recall    = counts['TP'] / (counts['TP'] + counts['FN'])
        accuracy  = (counts['TP'] + counts['TN']) / len(y)
        f1score   = (2 * precision * recall) / (precision + recall)
        
        return {
            'counts': counts,
            'precision': precision,
            'recall': recall,
            'accuracy': accuracy,
            'f1score': f1score
        }