{"repo":"19PINE-AI/aoi","free":true,"listed":false,"github":"https://github.com/19PINE-AI/aoi","clone":"git clone https://github.com/19PINE-AI/aoi.git","description":"AOI: a model-agnostic perception layer giving computer-use agents adaptive video + audio perception with zero retraining, plus DynaCU-Bench (100 dynamic browser tasks). Paper + benchmark + code.","language":"HTML","stars":10,"topics":["ai-agents","benchmark","computer-use","computer-use-agents","llm-agents","multimodal","perception","playwright","vision-language-model","whisper"],"license":null,"category":"ai-agents","readme_excerpt":"Agent-Computer Observation Interface (AOI) & DynaCU-Bench A model-agnostic perception layer that gives existing computer-use (CU) agents adaptive visual and audio perception of dynamic screen content — video, animations, transient UI events, meetings, notifications, and spoken instructions — with zero retraining , plus DynaCU-Bench , a benchmark of dynamic browser tasks that screenshot-only agents cannot solve. Across nine CU models (closed and open-source, 7B to frontier scale) the AOI yields +17 to +48 pp on DynaCU-Bench with no retraining — Gemini 3 Flash the lone exception, where keyframe-token dilution means components must be selected per model. 📄 Paper: arXiv:2606.29472 · 🌐 Website: https://01.me/research/aoi · 📊 Results map: results/README.md Authors: Bojie Li (Pine AI) and Noah Shi (University of Washington). --- Contents - What problem does this solve? - Repository layout - DynaCU-Bench - Installation - Quick start - Reproducing the paper - Tests - License & citation What problem does this solve? Every current CU agent ties observation to action: one screenshot per action step, 3–5 s apart, and no audio. Between screenshots the agent is blind and deaf. The AOI decouples observation (continuous, adaptive) from action (discrete) : it sits between the environment and any image-based CU model, watches the full interval between agent steps, and converts the screen and audio streams into the sparse images and text the model already accepts. It has three gated component","default_branch":null,"files":null,"tree":[],"storefront":"/r/19PINE-AI","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/19PINE-AI/aoi/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}