Optimizing customer support using Text2SQL to query natural language databases

Abstract

PURPOSE: This paper explores the challenges and potential solutions associated with integrating Text2SQL technology into customer support operations. By leveraging large language models (LLMs) and tools like Vanna.AI, the study aims to enhance the efficiency and accuracy of handling customer queries without requiring specialized SQL knowledge.

DESIGN/METHODOLOGY/APPROACH: A comprehensive analysis was conducted comparing the effectiveness of three large language models—Llama3:70b-instruct, Gemma2:27b, and Codegemma—in generating correct SQL queries from natural language questions. The models were trained with identical datasets and evaluated using six benchmark questions over two iterations, with and without detailed database schema information. Performance metrics included correctness of the generated queries and response times.

FINDINGS: The results indicated that while Llama3 and Gemma2 initially demonstrated higher accuracy, the addition of detailed database schema information did not improve model performance. Instead, it led to decreased accuracy and increased response times, particularly for Llama3. Codegemma showed shorter response times but slightly lower accuracy. The study highlights that excessive contextual information can overwhelm LLMs, suggesting the need for optimized context provision.

PRACTICAL IMPLICATIONS: The findings suggest that simplifying database schema information and focusing on essential contextual data can enhance the performance of LLMs in generating SQL queries. Implementing tools like Vanna.AI, which utilize Retrieval Augmented Generation (RAG), can improve customer support processes by enabling quick and accurate data access without specialized SQL expertise.

ORIGINALITY/VALUE: This paper provides valuable insights into the practical challenges of implementing Text2SQL technology in customer support. It offers recommendations for balancing context provision and model capabilities, contributing to the optimization of LLM performance in real-world applications.