LLaVA-Plus - Large Language and Vision Assistants that Plug and Learn to Use Skills

Abstract

LLaVA-Plus is a general-purpose multimodal assistant that expands the capabilities of large multimodal models. It maintains a skill repository of pre-trained vision and vision-language models and can activate relevant tools based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on multimodal instruction-following data to acquire the ability to use tools, covering visual understanding, generation, external knowledge retrieval, and compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in existing capabilities and exhibits new ones. It is distinct in that the image query is directly grounded and actively engaged throughout the entire human-AI interaction sessions, significantly improving tool use performance and enabling new scenarios.

Paper: https://arxiv.org/abs/2311.05437

Code: https://github.com/LLaVA-VL/LLaVA-Plus-Codebase

Demo: https://llavaplus.ngrok.io/

Dataset: https://huggingface.co/datasets/LLaVA-VL/llava-plus-data

Model: https://llava-vl.github.io/llava-plus/