�1����
Scope������������
�2����
References���
�3����
Definitions���
�4����
Abbreviations and acronyms��
�5����
Conventions
�6����
Subjective test and objective algorithms���������������
������� 6.1����
Aspects related to subjective testing������������
������� 6.2����
Aspects related to objective algorithms�������
�7����
Evaluation framework��������������
������� 7.1����
Data preparation�������������
������� 7.2����
Analysis types��
������� 7.3����
Prediction on a numerical quality scale�������
������� 7.4����
Uncertainty of subjective results��
������� 7.5����
Statistical evaluation metrics��������
������� 7.6����
Statistical significance evaluation
������� 7.7����
Statistical evaluation in the context of subjective uncertainty: epsilon
insensitive rmse and its statistical significance����
������� 7.8����
Statistical evaluation of the overall performance�����
�8����
Guidance on algorithm selection�������������
������� 8.1����
Per experiment performance���������
������� 8.2����
Overall figure of merit�����
������� 8.3����
Worst performance cases���������������
������� 8.4����
�Averaging statistical metrics across experiments������
�9����
Special cases���������������
������� 9.1����
Evaluation of algorithms with more than one output��������������
������� 9.2����
Evaluation of algorithms against pre-defined minimum performance
requirements��������
10����
Demonstration cases
Appendix I � Algorithm mapping to the subjective scale����
Appendix II � The impact of the third order versus first order
mapping����
������ II.1����
Application of third order and first order mappings��
������ II.2����
Gain of third order mapping����������
Appendix III � Confidence intervals calculation����
����� III.1����
The standard deviation for file-based analysis���������
����� III.2����
The standard deviation for condition-based analysis��������������
����� III.3����
Exceptional cases�����������
Appendix IV � Normality test����
Appendix V � Statistical significance of the rmse_tot* across all
experiments����
Bibliography����