Make all grad_norm logs in a single section
🚀 Feature
Make all grad_norm logs in a single section
Motivation
current tensorboard logs of gradient norm looks like:

e.g. for each parameter, there is a separate section that contains a single plot. It takes too much space especially in case of really deep networks or when you're experimenting with different architectures. In both cases, it may take several "screens" and scrolling it becomes annoying. Even if you logged gradnorm in a single experiment and turn it off, this empty plots still exist.

Pitch

instead of logging metrics with tag grad_norm_parameter_name make single section grad_norm by changing tag on grad_norm/parameter_name
or, change 
  
    
      pytorch-lightning/pytorch_lightning/trainer/logging.py
    
    
         Line 60
      in
      ca18e11
    
  
  
    

        
          
           metrics.update(grad_norm_dic) 
        
    
  


to
metrics.update({f'grad_norm/{k}': v for k, v in grad_norm_dic.items()})