Design and Evaluation of a CMS-Integrated Academic Chatbot Using Gemini AI

Abstrak

Efficient academic services are a crucial component, and their implementation is the responsibility of higher education institutions. Telkom University Purwokerto (TUP) faces challenges in providing responsive academic services, especially in conventional online services. This research proposes the development and integration of an artificial intelligence-based academic chatbot, ‘Akif’, utilizing the Gemini 1.5 Flash model, which is linked to the institution’s Content Management System (CMS). This integration enables the retrieval of real-time information and automatic updates to the model. The tuning and evaluation process was conducted using the BLEU metric, with a value of 0.88 being reached, indicating a fairly good level of agreement between the generated answers and the reference. Although the results are promising, the system still faces limitations, particularly the risk of hallucination, which is a common challenge with generative models. Additionally, the use of BLEU as an initial evaluation metric overlooks aspects of semantic depth and user satisfaction. This research contributes a modular integration framework between generative AI and institutional systems, and highlights its potential and limitations in academic service automation.