{"repo":"hkchengrex/MMAudio","free":true,"listed":false,"github":"https://github.com/hkchengrex/MMAudio","clone":"git clone https://github.com/hkchengrex/MMAudio.git","description":"[CVPR 2025] MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis","language":"Python","stars":2259,"topics":["audio","audio-synthesis","computer-vision","deep-learning","video-to-audio","text-to-audio"],"license":"MIT","category":"machine-learning","readme_excerpt":"MMAudio Paper Webpage Models Huggingface Demo Colab Demo Replicate Demo Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji University of Illinois Urbana-Champaign, Sony AI, and Sony Group Corporation CVPR 2025 Highlight MMAudio generates synchronized audio given video and/or text inputs. Our key innovation is multimodal joint training which allows training on a wide range of audio-visual and audio-text datasets. Moreover, a synchronization module aligns the generated audio with the video frames. Check out this fun video: [[Does Your Voice Match Your Face? https://youtu.be/SLz3NWLyHxg]](https://youtu.be/SLz3NWLyHxg) Results (All audio from our algorithm MMAudio) Videos from Sora: https://github.com/user-attachments/assets/82afd192-0cee-48a1-86ca-bd39b8c8f330 Videos from Veo 2: https://github.com/user-attachments/assets/8a11419e-fee2-46e0-9e67-dfb03c48d00e Videos from MovieGen/Hunyuan Video/VGGSound: https://github.com/user-attachments/assets/29230d4e-21c1-4cf8-a221-c28f2af6d0ca For more results, visit https://hkchengrex.com/MMAudio/video main.html. Installation We have only tested this on Ubuntu. Prerequisites We recommend using a miniforge environment. - Python 3.9+ - PyTorch 2.5.1+ and corresponding torchvision/torchaudio (pick your CUDA version https://pytorch.org/, pip install recommended) 1. Install prerequisite if not yet met: (Or any other CUDA versions ","default_branch":null,"files":null,"tree":[],"storefront":"/r/hkchengrex","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/hkchengrex/MMAudio/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}