Using a concrete example, this position paper makes a case for evaluating the correctness of software recommendation systems in an automated way, prior to conducting user studies, in order to assess the validity of the results and ideal configuration of the system to be evaluated.
Lozano Rodriguez, A., Kellens, A., & Mens, K. (2012). Automated assessment of correctness of recommendation systems. Proceedings of the 3rd International Workshop on Recommendation Systems for Software Engineering, 52-53. https://doi.org/10.1109/RSSE.2012.6233410