Context
These sources present a framework for Conformal Language Modeling, a method designed to provide statistical guarantees for the accuracy of generative artificial intelligence. By adapting conformal prediction to the complex output space of large language models, the researchers introduce a system that generates a set of candidate responses rather than a single answer. This process utilizes a calibrated stopping rule to determine when enough samples have been drawn to likely include a correct response, alongside a rejection rule to filter out low-quality or redundant entries. Beyond full responses, the methodology also identifies specific sub-components, such as individual sentences, that are independently verified as reliable. Experimental results across question answering, text summarization, and radiology report generation demonstrate that this approach effectively manages the risk of “hallucinations.” Ultimately, the research offers a rigorous mathematical pathway to make unpredictable language models more trustworthy and precise for real-world applications.
Chapters
0:00— Introduction au problème0:34— Solution : troisième option1:06— Implémentation et logique1:40— Workflow intelligent
Sources
- Calibrating LLMs for Selective Prediction: Balancing Coverage and Risk - OpenReview
- Conformal Language Modeling - Google Research
- Conformal Language Modeling - arXiv
- Conformal Regression under Distribution Shift: A Reinforcement Learning Method for Adaptive Uncertainty Quantification | OpenReview
- Online Selective Conformal Prediction: Errors and Solutions - arXiv
- Robust Conformal Prediction under Joint Distribution Shift - OpenReview
- Selective Conformal Risk Control - arXiv
- Selective Generation for Controllable Language Models - NIPS papers
- UNCERTAINTY QUANTIFICATION VIA REASON- ING–EXPLANATION SYMMETRY IN LLMS - OpenReview
- [2405.01563] Mitigating LLM Hallucinations via Conformal Abstention - arXiv