Hand Symbol Classification for Human-Computer Interaction Using the Fifth Version of YOLO Object Detection

Human-Computer Interaction (HCI) nowadays mostly uses physical contact, such as people using the mouse to choose something in an application. However, there are certain problems that people face in using conventional HCI. The research tries to overcome some problems when people use conventional HCI using the computer vision method. The research focuses on creating and evaluating the object detection model for classifying hand symbols. The research applies the fifth version of YOLO with the architecture of YOLOv5m to classify hand symbols in real time. The methods are divided into three steps. Those steps are dataset creation consisting of 100 images in each class, training phase, and performance evaluation of the model. The hand gesture classes made in the research are ‘ok’, ‘cancel’, ‘previous’, ‘next’, and ‘confirm’, the dataset is made by the researchers custom. After the training phase, the validation results show 93% for accuracy, 99% for precision, 100% for recall, and 99% for F1 score. Meanwhile, in real-time detection, the performance of the model for classifying hand symbols is 80% for accuracy, 95% for precision, 84% for recall, and 89% for F1 score. Although there are differences, it
still acceptable for the research and can be improved in future research.