The subjectivity associated with evaluating conversations is a key element underlying the challenge of building non-goal oriented dialogue systems. This paper proposes a comprehensive evaluation strategy with multiple metrics designed to reduce subjectivity by selecting metrics which correlate well with human judgement. To our knowledge, to date it is the largest setting for evaluating agents […]