{"repo":"OpenAdaptAI/OmniMCP","free":true,"listed":false,"github":"https://github.com/OpenAdaptAI/OmniMCP","clone":"git clone https://github.com/OpenAdaptAI/OmniMCP.git","description":"OmniMCP uses Microsoft OmniParser and Model Context Protocol (MCP) to provide AI models with rich UI context and powerful interaction capabilities.","language":"Python","stars":72,"topics":["anthropic","aws","computeruse","gemini","generative-ai","model-context-protocol","omniparser","openai"],"license":null,"category":"ai-agents","readme_excerpt":"OmniMCP OmniMCP provides rich UI context and interaction capabilities to AI models through Model Context Protocol (MCP) and microsoft/OmniParser. It focuses on enabling deep understanding of user interfaces through visual analysis, structured planning, and precise interaction execution. Core Features - Visual Perception: Understands UI elements using OmniParser. - LLM Planning: Plans next actions based on goal, history, and visual state. - Agent Executor: Orchestrates the perceive-plan-act loop ( omnimcp/agent executor.py ). - Action Execution: Controls mouse/keyboard via pynput ( omnimcp/input.py ). - CLI Interface: Simple entry point ( cli.py ) for running tasks. - Auto-Deployment: Optional OmniParser server deployment to AWS EC2 with auto-shutdown. - Debugging: Generates timestamped visual logs per step. Overview cli.py uses AgentExecutor to run a perceive-plan-act loop. It captures the screen ( VisualState ), plans using an LLM ( core.plan action for ui ), and executes actions ( InputController ). Demos - Real Action (Calculator): python cli.py opens Calculator and computes 5 9. - Synthetic UI (Login): python demo synthetic.py uses generated images (no real I/O). (Note: Pending refactor to use AgentExecutor). Prerequisites - Python =3.10, / . Note on MCP Server: An experimental MCP server ( OmniMCP class in omnimcp/mcp server.py ) exists but is separate from the primary cli.py / AgentExecutor workflow. Architecture 1. CLI ( cli.py ) - Entry point, setup, starts Executor. ","default_branch":null,"files":null,"tree":[],"storefront":"/r/OpenAdaptAI","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/OpenAdaptAI/OmniMCP/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}