Screen MCP: give AI agents eyes and hands on your phone and browser

What is a screen MCP server?

The Model Context Protocol (MCP) is an open standard for connecting AI applications to outside tools and data. An MCP client, such as Claude Desktop or Claude Code, starts or connects to an MCP server and asks what it offers. The server answers with tools (functions the agent can call), prompts (ready-made instructions) and resources (data the agent can read).

It is the kind of MCP server whose tools are about a screen. It usually exposes three groups of tools:

• Perception: take a screenshot, read the UI hierarchy, list recent frames.
• Action: tap, swipe, type, press a key, open an app or a URL.
• Verification: wait until the screen changes, compare two frames, read the logs.

With those, an agent can run the loop a person runs without thinking about it: look, act, look again, decide. Without one, you are the agent's eyes. You take screenshots, paste them into a chat, describe what happened after a tap, and repeat. A server like this takes you out of that loop for the mechanical parts and keeps you in it for the parts that need judgement, such as approving a risky click.

Transport matters too. Most servers of this kind run locally and talk to the client over stdio: the client launches the server as a child process on your computer. Others are remote servers reached over HTTPS, and some combine a local MCP server with a relay so the device can sit on a different network.

Three kinds: phone, browser and desktop

Phone

A phone server reads and drives a mobile device. On Android the hard parts are permission and transport, because the operating system guards screen capture and input closely, so every approach makes a trade-off (covered in the next section). iOS is more locked down still; servers that support it lean on Xcode tooling, simulators or cloud-hosted devices.

Browser

A browser server works inside a real browser tab, usually through an extension or the browser's debugging protocol. It can read the page's DOM and accessibility tree, not just pixels, so clicks land on the right element and extracted text is exact. The better ones act in your own signed-in browser and stop for your approval before anything you cannot undo.

Desktop

A desktop server captures a Windows, macOS or Linux display and sends mouse and keyboard input through operating-system APIs. It reaches native desktop apps that have no DOM, at the cost of having less structure to read than a web page or a mobile UI tree.

ScreenSync MCP covers the first two in one hub, and adds three small desktop tools (os_type, os_mouse_click and os_hotkey) for the computer the hub runs on.

How Android screen capture and control work

Android offers three practical routes, and every Android screen MCP server uses one of them or a mix. None is best in every case; each trades setup effort against reach.

MediaProjection: capture

MediaProjection is Android's official screen-capture API, the one screen recorders use. The app asks for consent in a system dialog, Android shows that the screen is being captured, and the app receives frames. It needs no root and no ADB, and it captures what you actually see, including apps that expose no accessibility labels. It only watches; it cannot tap. Consent lasts for a session, so capture stops when you stop it.

AccessibilityService: read and act

An accessibility service is an app component meant for assistive tools. Once you enable it in Settings, it can read the on-screen UI tree, take screenshots on recent Android versions and perform gestures, with no computer or cable involved. That is why it suits relay-based remote control. The cost is trust: an accessibility service can see and act across every app, so Android and Google Play treat it as a sensitive permission, and screens that hide content from accessibility give it less to work with.

ADB and scrcpy: the developer bridge

ADB, the Android Debug Bridge, is the developer channel between a computer and a phone with USB debugging or wireless debugging turned on. From the computer it can dump the UI hierarchy, take screenshots, read logcat and send taps, swipes, text and key events, all without root. scrcpy builds on ADB to mirror the screen with low latency and forward input. The cost is setup: developer options, a cable or a wireless-debugging pairing, and ADB installed on the computer.

ScreenSync MCP combines the first and the third. The app captures with MediaProjection, and the hub sends taps, swipes and text through ADB. Capture works with no developer setup at all; control is there once you turn on USB or wireless debugging.

Live streaming vs screenshot polling

A server can hand the agent pixels in two ways. With polling, the agent calls a screenshot tool and the server grabs one frame. With streaming, frames flow continuously to a local hub, and the agent's tools read the newest one, wait for a change, or compare two moments.

Polling is simple and costs nothing between calls, but the agent only sees the moments it asks about. A toast that lasts two seconds, a spinner, a crash dialog that flashes and closes, or the one frame where an animation glitches can all fall between two calls. Each poll also pays the full capture cost while the agent is waiting.

Streaming keeps a fresh frame ready, so “what is on screen now?” returns at once. Tools such as wait_for_frame and compare_frames can block until the screen really changes instead of sleeping for a guessed number of seconds, which makes agent loops faster and less flaky. ScreenSync MCP delivers frames in about 94 ms median on our LAN tests.

Streaming does not mean the model watches video. The agent still pulls the frames it needs, so model costs follow the screenshots it actually reads.

The privacy model: where your screen goes

A screen is the most personal data on a device: messages, banking apps, one-time codes. Before you install any server of this kind, ask three questions.

1. Where do frames travel? Local servers keep them on your computer or network. Relay designs pass commands, and often screenshots, through a server in between, run by the vendor or by you. Cloud-phone services run the device itself in a data centre.
2. Who can call the tools? Look for authentication between the device and the server, such as a bearer token, so another device on the same network cannot drive your phone.
3. What does the model provider see? Every screenshot an agent reads is sent to that agent's model provider as part of the conversation. No local design changes that, so decide what you show.

ScreenSync MCP's answers: frames go from phone to hub over your own network with a bearer token, and nothing goes to EpsolDev servers. The optional Google Drive fallback uses your Drive account. Screenshots that your AI agent requests are sent to that agent's model provider, so close sensitive apps before a session. The browser extension adds a human approval gate for risky actions. The data policy lists where every byte is stored.

How ScreenSync MCP does it

ScreenSync MCP is an Android screen MCP server built from three parts that you run yourself:

• The Android app (Android 7.0 or later) captures the screen with MediaProjection after your consent each session. It has a floating capture bubble, shake-to-capture, a live mirror stream to the hub, pairing by QR code or mDNS, a local capture history and an optional Google Drive fallback.
• The desktop hub is a Node.js 18+ program for Windows, macOS or Linux. It serves HTTP on port 3000, speaks MCP to your AI client over stdio, checks a bearer token, and drives the phone through ADB when you ask for control.
• The browser extension for Chrome and Edge (Manifest V3) lets the agent see, read, click, type and navigate in your real tab, and asks you before risky actions. See the ScreenSync browser extension page.

Together they expose 226 MCP tools: 29 for your phone and desktop, 197 for your browser. The phone and desktop side has 12 control tools (among them control_tap, control_swipe, control_type, control_tap_text, control_launch_app, control_open_url and control_screenshot), the 3 desktop os_* tools, and 14 capture and hub tools such as get_latest_screenshot, get_ui_hierarchy, get_logcat, record_screen, wait_for_frame, compare_frames and get_device_status. The other 197 are web_* tools for the browser. The server also ships 17 MCP prompts and 3 resources.

It is free while in active development, with no per-command quota. Current versions: Android app 2.5.7 (build 35), hub and extension 1.14.4.

Screen MCP options compared

ScreenSync MCP is our product, so weigh this short table with that in mind. Facts about other projects come from their own sites as of 1 October 2026.

The full table covers eight servers across ten criteria, with a source for every fact: compare Android MCP servers.

Set up an Android screen MCP server in four steps

1. Install ScreenSync MCP from Google Play on a phone running Android 7.0 or later.
2. Download the hub and run start-hub.bat on Windows or sh start-hub.sh on macOS and Linux. It needs Node.js 18 or later.
3. Scan the hub's pairing QR code in the app, or let the app find the hub on your network over mDNS.
4. Add the hub to your MCP client: an mcpServers entry with the command node and the hub script as its argument.

For the tap, swipe and type tools, turn on USB debugging or wireless debugging so the hub can reach the phone over ADB. Copy-ready configs for Claude Code, Claude Desktop, Cursor and VS Code, plus environment variables and troubleshooting, are in the ScreenSync MCP setup guide.

What people use it for

App debugging

The agent reads the screen, the UI hierarchy and logcat together, so “the button does nothing” turns into a stack trace and a likely cause instead of a guess.

UI testing

Describe a flow in plain words. The agent taps through it, waits for each screen, and reports where the app differs from what you expected. ScreenSync MCP ships an autonomous_ui_test prompt for this.

Bug reproduction

Hand the agent a bug report. It follows the steps on a real device, records the screen and collects logs at the moment things go wrong, using the reproduce_bug prompt.

Accessibility audits

The live UI tree shows missing labels, small touch targets and unlabelled images, and the accessibility_audit prompt walks through them on the running app.

Browser tasks

With the extension, the agent fills forms, compares pages, extracts tables and checks a web app in your own signed-in tab, and stops to ask before anything risky.