{"repo":"kiranbaby14/TalkMateAI","free":true,"listed":false,"github":"https://github.com/kiranbaby14/TalkMateAI","clone":"git clone https://github.com/kiranbaby14/TalkMateAI.git","description":"🎭 Real-time voice-controlled 3D avatar with multimodal AI - speak naturally and watch your AI companion respond with perfect lip-sync","language":"TypeScript","stars":105,"topics":["fastapi","multimodal-ai","nextjs","vlm","websocket","flash-attention-2","huggingface","kokoro-tts","smolvlm","whisper-ai"],"license":"MIT","category":"networking-infra","readme_excerpt":"🎭 TalkMateAI Real-time Voice-Controlled 3D Avatar with Multimodal AI Your 3D AI companion that never stops listening, never stops caring. Transform conversations into immersive experiences with AI-powered 3D avatars that see, hear, and respond naturally. πŸŽ₯ Demo Video ✨ Features 🎯 Core Capabilities - 🎀 Real-time Voice Activity Detection - Advanced VAD with configurable sensitivity - πŸ—£οΈ Speech-to-Text - Powered by OpenAI Whisper (tiny model) for instant transcription - πŸ‘οΈ Vision Understanding - SmolVLM2-256M-Video-Instruct for multimodal comprehension - πŸ”Š Natural Text-to-Speech - Kokoro TTS with native word-level timing - 🎭 3D Avatar Animation - Lip-sync and emotion-driven animations using TalkingHead πŸš€ Advanced Features - πŸ“Ή Camera Integration - Real-time image capture with voice commands - ⚑ Streaming Responses - Chunked audio generation for minimal latency - 🎬 Native Timing Sync - Perfect lip-sync using Kokoro's native timing data - 🎨 Draggable Camera View - Floating, resizable camera interface - πŸ“Š Real-time Analytics - Voice energy visualization and transmission tracking - πŸ”„ WebSocket Communication - Low-latency bidirectional data flow πŸ—οΈ Architecture πŸ› οΈ Technology Stack Backend (Python) - 🧠 AI Models from HuggingFaceπŸ€—: - openai/whisper-tiny - Speech recognition - HuggingFaceTB/SmolVLM2-256M-Video-Instruct - Vision-language understanding - Kokoro TTS - High-quality voice synthesis - ⚑ Framework: FastAPI with WebSocket support - πŸ”§ Processing: PyTorch, Trans","default_branch":null,"files":null,"tree":[],"storefront":"/r/kiranbaby14","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/kiranbaby14/TalkMateAI/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}