Rec. ITU-T F.748.11 (08/2020) Metrics and evaluation methods for a deep neural network processor benchmark
Summary
History
FOREWORD
Table of Contents
1 Scope
2 References
3 Definitions
     3.1 Terms defined elsewhere
     3.2 Terms defined in this Recommendation
4 Abbreviations and acronyms
5 Conventions
6 Overview of a deep neural network processor benchmark
     6.1 Evaluation object
     6.2 Evaluation principle
          6.2.1 Practicality
          6.2.2 Fairness
          6.2.3 Reproducibility
          6.2.4 Evaluation mechanism
          6.2.5 Test environment
7 Architecture framework of the deep neural network processor benchmark
     7.1 Workload
     7.2 AI framework
     7.3 Hardware acceleration SDK
     7.4 Processor hardware system
8 Benchmark metrics and evaluation methods for AI chip under training tasks
     8.1 Metrics
          8.1.1 Training time
          8.1.2 Accuracy
     8.2 Benchmark specification
          8.2.1 Test data set
          8.2.2 Model
          8.2.3 Reference implementation
9 Benchmark metrics and evaluation methods for AI chip under inference tasks
     9.1 Metrics
          9.1.1 Inference latency
          9.1.2 Throughput
          9.1.3 Power consumption
          9.1.4 Energy efficiency
          9.1.5 Accelerator utilization
          9.1.6 Accuracy
     9.2 Benchmark specification
          9.2.1 Test system environment
          9.2.2 Acceleration SDK
          9.2.3 Model information
          9.2.4 Reference implementation
10 Benchmark application scenarios
Appendix I  Reference implementations of benchmark application scenarios