Feature/fidelity metrics from emnlp2020 evaluating and characterizing human rationales
Hi ! This looks interesting, thanks for adding it :) 

For metrics there should only be two features fields: references and predictions.
Both of them can be defined as you want using nested structures if you need to.
Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for > Hi ! This looks interesting, thanks for adding it :)
> 
> For metrics there should only be two features fields: references and predictions.
> Both of them can be defined as you want using nested structures if you need to.
> Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
> All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for

The `predictions` are the predicted labels by a model for a particular input. Do you mean making `prob_y_hat` - the probability of the prediction being the predicted label, `prob_y_hat_alpha` - the probability of the prediction being the predicted label when the input is reduced subject to alpha and the `null_difference` is the difference between the probability of the prediction being the predicted label in full information minus the probability in zero information a part of references? Also, I have added the description for other parameters in kwargs_description. I can expand it if that makes sense? I think every value that is generated by the model (so label, prob_y_hat, prob_y_hat_alpha etc.) should be in `predictions`.
Feel free to add more details in the kwargs_description, this is very useful for the end user. Hi @lhoestq , I have updated the code according to your feedback. Please, let me know if it looks good and can be merged now. Metrics are deprecated in `datasets` and `evaluate` should be used instead: https://github.com/huggingface/evaluate
This metric computes fidelity (Yu et al. 2019, DeYoung et al. 2019) and normalized fidelity (Carton et al. 2020).