YY/T 1833.3-2022 Artificial intelligence medical device - Quality requirements and evaluation - Part 3: General requirement for data annotation
1 Scope
This document specifies the general requirements and evaluation methods for data annotation of artificial intelligence medical device.
It is applicable to data annotation activities of artificial intelligence medical device.
2 Normative references
The following documents contain provisions which, through reference in this text, constitute provisions of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.
YY/T 1833.1 Artificial intelligence medical device - Quality requirements and evaluation - Part 1: Terminology
YY/T 1833.2 Artificial intelligence medical device - Quality requirements and evaluation - Part 2: General requirements for datasets
3 Terms and definitions
For the purposes of this document, the terms and definitions given in YY/T 1833.1 and YY/T 1833.2 as well as the following apply.
3.1
annotation task
activity for purposefully analyzing a set of data to add external knowledge
3.2
annotation object
specific information such as data type, features and attributes analyzed in an annotation task
3.3
structured annotation
annotation task that records results by fixed formats and rules
3.4
non-structured annotation
annotation task that records results by unfixed formats and rules
3.5
semi-structured annotation
annotation task that records results by fixed formats and unfixed rules
3.6
manual annotation
annotation task performed entirely in manual way
3.7
automatic annotation
annotation task performed entirely by machines, and reviewed manually after the annotation is completed
3.8
semi-automatic annotation
annotation task performed jointly by human and machine
3.9
semantic annotation
annotation task that takes the meaning and relationship represented by data as the annotation object
3.10
annotator
personnel who have the ability to complete specific annotation task and meet quality requirements, perform annotation task, and directly contribute to the annotation task
Note: Including initial annotator, annotation reviewer, arbitrator, etc.
3.11
initial annotator
person who performs the annotation task and gives the preliminary annotation results
3.12
annotation reviewer
person responsible for the review and quality control of the preliminary annotation results
3.13
arbitrator
person responsible for giving the final result when many annotators have inconsistent annotation results for the same data
Note: Under normal circumstances, the qualification requirements of arbitrators > annotation reviewers ≥ initial annotators.
3.14
annotator performance
representational ability of the annotator to perform the annotation task
3.15
annotation responsible organization
entity that organizes and carries out the annotation task and is directly responsible for the annotation quality
4 Documents describing the annotation task
4.1 Annotation task classification
Before the annotation task starts, the annotation responsible organization shall clearly define the classification of the annotation task, including data mode, execution subject, annotation result format, annotation result property, annotation result form and other dimensions.
The data modes of the annotation task may be divided into image, signal, video, text, etc. According to the execution subject, annotation tasks may be divided into manual annotation, automatic annotation, semi-automatic annotation, etc. According to the annotation result format, annotation tasks may be divided into structured annotation, non-structured annotation, semi-structured annotation, etc. The annotation result property may be divided into GT value, reference standard, gold standard and other types. The annotation result form may be divided into detection, classification, segmentation, semantics and other types.
Note: Semantic annotation is often used to describe relationships or connections between objects, such as the relative positions of muscle and fat on ultrasound images.
4.2 Annotation task description
4.2.1 Annotation rules
Foreword i
Introduction ii
1 Scope
2 Normative references
3 Terms and definitions
4 Documents describing the annotation task
5 Data annotation quality features
6 Annotation and quality control process
7 Annotation tools
8 Evaluation method
Annex A (Informative) Examples of annotation task description
Annex B (Informative) Examples of business architecture (pulmonary nodules in thoracic CT )
Annex C (Informative) Evaluation of AI-assisted annotation performance
Bibliography
