Benchmarking CSV Data-Based Question Answering

The LangChain team focused on CSV data-based question answering (QA) tasks, which lacked clear recipes unlike text data. Previously, evaluating LLM applications was difficult because it was hard to predict what questions users would ask and there was a lack of appropriate metrics for evaluating natural language outputs.

To address this, they built a Streamlit app to collect real user question distributions and used LangSmith to debug issues with the initial solution. Instead of BLEU or ROUGE, they applied an LLM-based evaluation method as the evaluation metric.

The final improved solution is a custom agent that uses OpenAI functions. This agent handles question answering on CSV data by utilizing two tools: a Python REPL and a retriever. The app, dataset, and evaluation scripts were all released as open source.