Posted as part of the AI Alignment Forum sequence on Value Learning. Rohin's note: So far, we’ve seen that ambitious value learning needs to understand human biases, and that we can't simply learn the biases in tandem with the reward. Perhaps we could hardcode a specific model of human biases? Such a model is likely to be incomplete and inaccurate, but it will perform better than assuming an optimal human, and as we notice failure modes we can improve the model. In the language of this post by Jacob Steinhardt (original  here), we are using a mis-specified human model. The post talks about why model mis-specification is worse than it may seem at first glance. This post is fairly technical and may not be accessible if you don’t have a background in machine learning. If so, you can skip this post and still understand the rest of the posts in the sequence. However, if you want to do ML-related safety research, I strongly recommend putting in the effort to understand the problems that can arise with mis-specification. -------------------------------------------------------------------------------- Machine learning is very good at optimizing predictions to match an observed signal — for instance, given a dataset of input images and labels of the images (e.g. dog, cat, etc.), machine learning is very good at correctly predicting the label of a new image. However, performance can quickly break down as soon as we care about criteria other than predicting observables. There are several cases where we might care about such criteria:  * In scientific investigations, we often care less about predicting a specific    observable phenomenon, and more about what that phenomenon implies about an    underlying scientific theory.  * In economic analysis, we are most interested in what policies will lead to    desirable outcomes. This requires predicting what would counterfactually    happen if we were to enact the policy, which we (usually) don’t have any data    about.  * In ma