---

• Handbook – Practical procedures for subjective testing 2011
  • Foreword
  • Table of Contents
  • Section 1 – General
    • 1.0 General
    • 1.1 Comments on subjective methods used in the laboratory
    • 1.2 Utilization of the outcome of experiments
    • 1.3 ITU-T Recommendations
    • 1.4 The continuing need for subjective tests
    • 1.5 Laboratory tests versus field tests
    • 1.6 Recommended methods
      • 1.6.1 Conversational opinion tests
      • 1.6.2 Listening opinion tests
      • 1.6.3 Interview and survey tests
      • 1.6.4 Other subjective test methods
  • Section 2 – Test purpose
    • 2.0 Introduction
    • 2.1 Qualification
    • 2.2 Selection
    • 2.3 Characterization
    • 2.4 Verification
    • 2.5 Research
    • 2.6 Others
  • Section 3 – Experimental designs
    • 3.0 Introduction
    • 3.1 Steps to designing a subjective test
    • 3.2 Design elements and design principles
      • 3.2.1 Sources of variability (languages, talkers, speech material, subjects, ...)
      • 3.2.2 Factors - random or fixed effects
    • 3.3 Types of test designs
      • 3.3.1 Factorial design
      • 3.3.2 Balanced blocks experimental design
      • 3.3.3 Latin Square and related designs
      • 3.3.4 Graeco-Latin Square design
      • 3.3.5 Nested design
      • 3.3.6 Other designs
    • 3.4 Scales
      • 3.4.1 Nominal scale
      • 3.4.2 Ordinal scale
      • 3.4.3 Interval scale
      • 3.4.4 Ratio scale
  • Section 4 – Conversational opinion tests
    • 4.0 Introduction
    • 4.1 Test facilities
    • 4.2 Test design
    • 4.3 Conversation task
    • 4.4 Test procedure
      • 4.4.1 Selection of test subjects
      • 4.4.2 Subjective evaluation criteria and opinion scales
      • 4.4.3 Example instructions to subjects of the conversational test
      • 4.4.4 Data collection
      • 4.4.5 Data analysis and report of results
  • Section 5 – Listening opinion tests
    • 5.0 Introduction
    • 5.1 Database material
      • 5.1.1 Speech
      • 5.1.2 Noise database
      • 5.1.3 Processing for addition of noise and speech
    • 5.2 Reference systems
      • 5.2.1 Intermediate reference system
      • 5.2.2 Modified Intermediate Reference System
      • 5.2.3 Flat filtering systems
      • 5.2.4 Other filtering systems
    • 5.3 Collection of source material
      • 5.3.1 Recording environment
      • 5.3.2 Recording system
      • 5.3.3 Control of the speech/music data base
      • 5.3.4 Processing of the speech files
      • 5.3.5 Source/Processed speech file naming convention
    • 5.4 Listening test procedure
      • 5.4.1 Listening environment
      • 5.4.2 Listening system
      • 5.4.3 Listening level
      • 5.4.4 Eligibility of subjects
      • 5.4.5 Subjective evaluation criteria and Opinion scales
      • 5.4.6 Equivalence of opinion scales between languages
      • 5.4.7 Category judgement
      • 5.4.8 Instructions to subjects
  • Section 6 – Statistical analysis and reporting of results
    • 6.0 Introduction
    • 6.1 Basic concepts of statistics
      • 6.1.1 General principles of statistics
      • 6.1.2 Definitions
      • 6.1.3 Application in subjective tests
    • 6.2 Student t-tests
      • 6.2.1 Advantages and disadvantages
      • 6.2.2 Definitions
    • 6.3 Analysis of Variance
      • 6.3.1 Purpose and underlying assumptions
      • 6.3.2 Analysis
      • 6.3.3 Post hoc comparison of means
  • Section 7 – Experimental design for speech coder evaluations (controlling sources of error)
    • 7.0 Introduction
    • 7.1 Source material factors (recording procedures, talker factors)
    • 7.2 Laboratory factors
    • 7.3 Language factors
    • 7.4 Listener factors (selection and sampling, instructions and training)
    • 7.5 Time/order factors, balance factors (generation of randomization), test context
    • 7.6 Sequences/sessions duration
    • 7.7 Presentation of results
      • 7.7.1 Analysis tools
      • 7.7.2 Data collection
      • 7.7.3 GAL
  • Section 8 – Examples of Test plans/Results on coders
  • Annex A – Processing principles (Software Tool Library)
    • A.1 Scope
    • A.2 History
    • A.3 Motivation
    • A.4 Software tools to emulate transmission chains
    • A.5 Software tools to process references for subjective tests
    • A.6 Software tool for ITU-T speech and audio codecs specification and objective performance assessment
    • A.7 Using the STL for implementing processing associated with a test design
  • Annex B – Relationship between subjective and objective test methods
    • B.1 Recommendation ITU-T P.862
    • B.2 Perceptually Weighted Misclassification Counting
  • Appendix 1 – Misclassified active frame replacement procedure
  • Bibliography