Convolutional Neural Networks for Measuring Service Satisfaction of Hajj Pilgrims through Facial Expression Analysis

Abstrak

Facial expressions serve as important non-verbal indicators of human emotions and can be leveraged to assess satisfaction levels in service environments. In the context of Hajj and Umrah, where verbal feedback may be limited due to language barriers or cultural factors, facial expression recognition offers a non-intrusive method to evaluate service quality. This study proposes a Convolutional Neural Network (CNN)-based model to detect emotional states such as happiness and dissatisfaction through facial expressions of pilgrims. A quantitative approach was adopted, employing preprocessing techniques including normalization, augmentation, and image resizing. The CNN architecture comprised multiple convolutional, pooling, and fully connected layers. The model was evaluated using accuracy, precision, recall, and F1-score metrics. Experiments with varying batch sizes (32, 64, 128, 256) across 50 epochs revealed that the optimal performance was achieved with a batch size of 64, resulting in an accuracy of 63%, precision of 66%, recall of 60%, and F1-score of 62%. During deployment, the model correctly classified 12 out of 16 real-world images, achieving a real-time accuracy of 78%. Therefore, the deployment results should be considered preliminary. Future studies will involve larger deployment samples, n-fold stratified cross-validation to obtain statistically reliable model performance, and subgroup analyses (e.g., lighting, facial pose, age, and gender) to better understand model behavior under diverse real-world conditions. All deployment images were collected with participant consent and processed without storing biometric data. These findings suggest that CNN-based emotion recognition can support real-time service evaluation and enhance the quality of pilgrim services during the Hajj and Umrah.