|
Work item:
|
P.SDS-MV (ex. P.ASR)
|
|
Subject/title:
|
Objective evaluation methods for speech dialogue systems in motor vehicles
|
|
Status:
|
Under study
|
|
Approval process:
|
AAP
|
|
Type of work item:
|
Recommendation
|
|
Version:
|
New
|
|
Equivalent number:
|
-
|
|
Timing:
|
2026 (Medium priority)
|
|
Liaison:
|
ETSI STQ, FG VM, ITU-T SG21
|
|
Supporting members:
|
HEAD acoustics GmbH, Qualcomm Inc., VWIF, Cerence
|
|
Summary:
|
This Recommendation provides specification of test environments and test setups for performing automatic speech recognition (ASR) and speech dialogue system (SDS) testing in a laboratory type environment, while simulating/controlling relevant driving and operating conditions in vehicles. Quality evaluation of the system's response (e.g., synthesized speech) is considered as well.
Even though all test methods and requirements target in the present document are intended to ensure a certain minimum quality of experience and lower driver distraction, any type of live testing, like e.g., drive testing or subjective testing that include human factors are out of scope.
Test methods, performance metrics and requirements for the performance evaluation of SDS and ASR functionality of devices focus on distant-talking scenarios. The methods include definition of input speech signal generation, positional relations of talkers and devices; acoustic environment characterization and reproduction, including background noise and reverberation. A collection of performance characteristics and statistical analyses is provided.
The following levels of speech recognition and dialogue functionalities are considered, which include technical/diagnostic testing (access to system internals may be required) as well as user-centric testing (black-box testing may apply/is possible):
Analysis of signals that are used as input to the SDS
Word recognition output / input to the semantic analysis of the SDS
Semantic output of the SDS
Interactivity
Analysis of the synthesized speech output/response of the SDS
|
|
Comment:
|
-
|
|
Reference(s):
|
|
|
Historic references:
|
|
Contact(s):
|
|
| ITU-T A.5 justification(s): |
|
|
|
|
First registration in the WP:
2022-06-26 13:07:32
|
|
Last update:
2025-01-23 15:19:17
|
