{"repo":"aimagelab/ReflectiVA","free":true,"listed":false,"github":"https://github.com/aimagelab/ReflectiVA","clone":"git clone https://github.com/aimagelab/ReflectiVA.git","description":"[CVPR 2025] Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering","language":"Python","stars":57,"topics":["knowledge-base","mllm","multimodal","vqa","vlm"],"license":"Apache-2.0","category":"productivity","readme_excerpt":"Reflective LLaVA (ReflectiVA) Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025) This repository contains the reference code for the paper Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering. 🎯 Project web page Paper 🤗 HuggingFace Model 🤗 HuggingFace Dataset Table of Contents 1. Citation 2. Overview 3. Installation 4. Model 5. Dataset 6. Training 7. Knowledge Based 8. Inference 9. Acknowledgements Citation Please cite this work with the following BibTeX: Overview Multimodal LLMs (MLLMs) are the natural extension of large language models to handle multimodal inputs, combining text and image data. They have recently garnered attention due to their capability to address complex tasks involving both modalities. However, their effectiveness is limited to the knowledge acquired during training, which restricts their practical utility. In this work, we introduce a novel method to enhance the adaptability of MLLMs by integrating external knowledge sources. Our proposed model, Reflective LLaVA ( ), utilizes reflective tokens to dynamically determine the need for external knowledge and predict the relevance of information retrieved from an external database, ultimately enables the MLLM to manage external knowledge while preserving fluency and performance on tasks where external knowledge is not needed. Installation To create the conda environment named reflectiva use the fo","default_branch":null,"files":null,"tree":[],"storefront":"/r/aimagelab","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/aimagelab/ReflectiVA/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}