Comparing Sentiment Labeling with RoBERTa and IndoBERTweet on Public Opinion of Program Makan Bergizi Gratis

Abstrak

The Program Makan Bergizi Gratis (MBG) is a flagship program of the Prabowo Subianto administration launched in 2024, triggering diverse public responses on social media. Sentiment analysis using deep learning models offers an effective approach to understanding public opinion at scale. However, selecting the appropriate model for Indonesian social media text remains challenging. This study aims to compare the performance of two pretrained transformer models, RoBERTa Base and IndoBERTweet Base, in conducting automatic sentiment labeling on Indonesian tweets related to the MBG program using a zero-shot labeling approach without human-annotated ground truth. A total of 1,831 tweets were collected from platform X and preprocessed using case folding, normalization, and stopword removal. Both models were applied in parallel to label each tweet with sentiment categories (positive, neutral, negative) along with confidence scores. The comparison was evaluated using agreement rate, Cohen's Kappa, and confidence score analysis. RoBERTa Base exhibits a conservative tendency with 75.20% neutral labels, while IndoBERTweet Base produces a more balanced distribution (68.16% neutral). The comparison shows 77.28% agreement with Cohen's Kappa of 0.490 (Moderate Agreement). RoBERTa Base achieves higher confidence (mean: 0.9559, 83.01% above 0.95) compared to IndoBERTweet Base (mean: 0.9236, 68.65% above 0.95). IndoBERTweet Base is more effective in detecting negative sentiment, identifying nearly twice as many negative tweets (13.54% vs. 7.48%). This study recommends IndoBERTweet Base for exploratory research requiring sensitive sentiment detection and RoBERTa Base for precision-critical applications. An ensemble approach combining both models is recommended for production-critical applications