{"repo":"google-deepmind/long-form-factuality","free":true,"listed":false,"github":"https://github.com/google-deepmind/long-form-factuality","clone":"git clone https://github.com/google-deepmind/long-form-factuality.git","description":"Benchmarking long-form factuality in large language models. Original code for our paper \"Long-form factuality in large language models\".","language":"Python","stars":692,"topics":["benchmark","dataset","evaluation","factuality","language","language-modeling","large-language-models","metrics"],"license":null,"category":"analytics","readme_excerpt":"Long-Form Factuality in Large Language Models This is the official code release accompanying our paper \"Long-form factuality in large language models\". This repository contains: 1. LongFact : A prompt set of 2,280 fact-seeking prompts requiring long-form responses. 2. Search-Augmented Factuality Evaluator (SAFE) : Automatic evaluation of model responses in long-form factuality settings. 3. F1@K : Extending F1 score to long-form settings using recall from human-preferred length. 4. Experimentation pipeline for benchmarking OpenAI and Anthropic models using LongFact + SAFE. Installation First, clone our GitHub repository. Then navigate to the newly-created folder. Next, create a new Python 3.10+ environment using conda . Activate the newly-created environment. All external package requirements are listed in requirements.txt . To install all packages, and run the following command. Usage LongFact The full prompt set for LongFact is available in the longfact/ folder. See the README in longfact/ for more details about the dataset. To run the data-generation pipeline that we used to generate LongFact, use the following command. Refer to the README in data creation/ for additional details about the data-generation pipeline. SAFE Our full implementation of SAFE is located in eval/safe/ . See the README in eval/safe/ for more information about how SAFE works. To run the pipeline for evaluating SAFE against FActScore human annotations, use the following command. Refer to the README in ","default_branch":null,"files":null,"tree":[],"storefront":"/r/google-deepmind","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/google-deepmind/long-form-factuality/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}