Explainable AI methods aim to explain black-box machine learning models and thereby give users insight into the model's decision-making process. But what makes a good explanation and how do we know whether the explanation is correct and complete? And wouldn’t it be better to build interpretability directly into the predictive model? In this talk, Meike Nauta will present 12 desired properties for a good explanation and provides an extensive collection of quantitative evaluation methods for explainable AI. Secondly, she will present ProtoTree: an interpretable image classifier where prototypical parts learned by a neural network are combined in an interpretable decision tree.