Privacy-Preserving Automated QA Dataset Generation for Fine-Tuning LLMs with Local Models and Information Retrieval

Abstrak

This paper introduces a novel framework for automated question-answering(QA) dataset construction, integrating information retrieval (IR) with a lightweight locallarge language model (LLM), SmolLM2- 360M-Instruct, to ensure privacy and scalabilityfor domain-specific applications. Addressing the limitations of manual dataset creationand cloud-based LLMs, our approach leverages PyPDF2 for robust PDF text extractionand a novel sentence segmentation algorithm to generate concise, contextually relevantQA pairs from domain-specific corpora. The framework employs IR techniques to alignquestions with precise answers, enhancing dataset quality while maintaining data privacythrough localized processing. Rigorous evaluation using automated metrics and manualexpert review confirms the high quality and semantic alignment of the generated QA pairs.This approach offers significant benefits for fine-tuning LLMs in niche domains, such aseducation and technical support, by providing scalable, privacy-preserving datasets thatimprove contextual understanding and adaptability. Our work contributes to efficient NLPdataset generation, offering a robust solution for advancing LLM performance in special-ized real-world applications.