Control Android with AI Without Root: How It Works

You can control Android with AI without root by splitting the job in two. Android's official MediaProjection API lets an app capture the screen after you consent, and the Android Debug Bridge (ADB) lets a computer send taps, swipes and text over USB or wireless debugging. ScreenSync MCP puts both behind a Model Context Protocol (MCP) server, so Claude Code, Claude Desktop or any MCP client that runs local stdio servers can see your phone and act on it without root or a modified system.

This guide covers what each half does, which tools the agent gets, how to set it up, and where the limits are. If the idea of a screen-aware server is new to you, start with what a screen MCP server is.

Why an AI agent does not need root

Root gives software the same power as the operating system. An agent that uses apps the way you do needs far less: a picture of the screen, a structured view of what is on it, and a way to inject input. Android offers all three through supported channels.

• Seeing. MediaProjection is the capture API that screen recorders use. You approve it in a system dialog and the app receives frames.
• Reading and acting. ADB is the developer bridge between a computer and a phone with USB debugging or wireless debugging turned on. From the computer it can dump the UI hierarchy, read logcat, take screenshots and inject input events.
• The third route. An accessibility service can read the UI tree and perform gestures on the phone itself. Android treats it as a sensitive permission because it can see and act across every app.

ScreenSync MCP uses the first two. The Android app contains no accessibility service and never asks for root. The Google Play build declares permissions for the screen-capture foreground service, the floating capture bubble, notifications, network access, vibration, a battery-optimization exemption request and the camera for QR pairing.

Seeing the screen: MediaProjection, with your consent

When you tap Grant screen capture in the app, Android shows its own consent dialog. After you accept, a foreground service captures frames and sends them over your local network to the hub on your computer. Consent lasts for the session: stop capture, or let Android revoke it, and frames stop until you grant it again.

This half needs no developer options, no cable and no ADB. While capture is granted, the live mirror sends frames continuously, with a median delivery time of about 94 ms on a LAN in our tests. Because a fresh frame is already on the hub, these tools answer quickly:

• get_latest_screenshot returns the newest frame as an image the model can inspect.
• get_recent_screenshots returns up to six recent frames in one call.
• wait_for_frame waits for a frame newer than the current one (or a timeout, 30 seconds by default), so the agent does not judge a stale screen.
• get_device_status reports whether the phone is streaming, linked with no frames yet, or absent.

MediaProjection only watches; it cannot tap anything.

Acting on the screen: ADB input from the hub

Control runs on your computer, not on the phone. The hub calls ADB's input command for taps, swipes, text and key events, uiautomator for the UI tree, screencap for on-demand screenshots and logcat for logs. It behaves the same over a USB cable or over wireless debugging, which Android supports from Android 11.

Of the server's 226 tools, 29 cover the phone, the desktop and the hub, and 12 of those are control tools:

The inspection tools matter as much as the input ones. get_ui_hierarchy returns each element's text, content description, resource ID, class, clickable flag, pixel bounds and center, so the agent taps an exact point instead of estimating coordinates from a picture. compare_frames captures two ADB screenshots a set delay apart (1,200 ms by default) and returns both images with a coarse changedRatio, which shows whether the screen is still changing. get_logcat filters recent log lines by app package or by text.

What the see-act loop looks like

A well-behaved agent runs the loop a person runs without thinking: look, act, look again. Ask it to "open Settings and find the dark theme switch," and the calls look roughly like this:

1. control_status confirms that ADB can reach the phone.
2. control_launch_app opens com.android.settings.
3. control_swipe_until scrolls until an entry matching "Display" is visible.
4. control_tap_text taps that entry by its label.
5. get_ui_hierarchy, filtered to "dark", shows how this phone labels the setting, and control_tap_text taps that label.
6. control_screenshot grabs the result so the agent can confirm the switch changed.

[
  { "name": "control_status", "arguments": {} },
  { "name": "control_launch_app", "arguments": { "package": "com.android.settings" } },
  { "name": "control_swipe_until", "arguments": { "query": "Display", "direction": "down", "maxSwipes": 8 } },
  { "name": "control_tap_text", "arguments": { "query": "Display" } },
  { "name": "get_ui_hierarchy", "arguments": { "filter": "dark" } },
  { "name": "control_tap_text", "arguments": { "query": "Dark theme" } },
  { "name": "control_screenshot", "arguments": {} }
]

Settings labels differ between manufacturers, which is exactly why the agent reads the UI tree instead of replaying fixed coordinates. You never write these calls yourself: you describe the goal and the client picks the tools. For longer sessions the server ships prompts such as autonomous_ui_test, which guides the agent through navigating and inspecting each screen, and reproduce_bug, which has it capture before-and-after frames and logcat at each repro step.

Set it up

ScreenSync MCP is free while in active development. These steps match app 2.5.7 and hub 1.14.4.

1. Install ScreenSync MCP from Google Play on a phone running Android 7.0 or later.
2. Download the hub and start it with start-hub.bat on Windows or sh start-hub.sh on macOS and Linux. It needs Node.js 18 or later and listens on port 3000.
3. Pair the phone by scanning the QR code the hub prints (also served at /pair on the hub machine), by pasting the pairing link, or by letting the app find the hub over mDNS, with a LAN scan as the fallback.
4. Grant screen capture when the app asks.
5. Register the hub with your MCP client. In Claude Code, run claude mcp add screensync node "<path>/screensync-hub.js", or add this block to .mcp.json. Claude Desktop takes the same block in claude_desktop_config.json. If you change the token from the default, use the block so the client passes the same SCREEN_SYNC_TOKEN as the hub.

{
  "mcpServers": {
    "screensync": {
      "command": "node",
      "args": ["<HUB_DIR>/screensync-hub.js"],
      "env": { "SCREEN_SYNC_TOKEN": "<TOKEN>" }
    }
  }
}

6. For control, install Android platform-tools on the computer, turn on developer options, then USB debugging or wireless debugging. Accept the prompt on the phone and confirm it is listed:

adb devices

With more than one device attached, set SCREEN_SYNC_ADB_TARGET to the phone's serial. If adb is not on your PATH, point SCREEN_SYNC_ADB_BIN at it. Cursor and VS Code also run local stdio servers; check your client's MCP docs for where the config lives. To drop the cable entirely, follow setting up Android wireless debugging for an MCP server.

The honest limits of control without root

Plan around these limits:

• Control needs developer options. Capture works without them; taps, swipes and typing do not. On some MIUI-class phones, input injection also needs USB debugging (Security settings) turned on.
• Typing is filtered. control_type goes through adb shell input text, and the hub strips quotes, backticks, backslashes, $, ;, &, |, <>, parentheses and braces before sending. Text with those characters will not arrive intact, so it is a poor tool for complex passwords.
• Keys are allow-listed. control_key accepts navigation and hardware keys such as back, home, recents, enter and the volume keys, not arbitrary key codes.
• There is no general shell. The agent is limited to the listed tools, not raw ADB commands.
• The UI tree is not always complete. Custom-drawn content such as games can expose few nodes, and the agent then falls back to screenshot coordinates.
• Secure screens stay dark. Apps that mark their windows as secure, common in banking and password managers, typically appear black in captures.
• Android and local clients only. There is no iOS support, and ChatGPT's web app cannot use the hub because it connects only to remote HTTPS MCP servers.

Safety and privacy when an AI drives your phone

An agent that can tap can tap the wrong thing, and the phone control tools do not pause for approval the way the browser extension does before risky actions. Treat a control session like handing someone your unlocked phone:

• Use a test device, or close sensitive apps before the session starts.
• Run the hub only on a trusted network, and change SCREEN_SYNC_TOKEN from the default screensync-local-dev. The hub's capture and control routes require that bearer token, and the pairing page that hands it out answers only on the hub machine by default.
• Turn off wireless debugging when you finish.
• Know where frames go. They travel from phone to hub over your own network and never reach EpsolDev servers; the optional Google Drive fallback uses your own Drive. Any screenshot the agent reads is sent to that agent's model provider.

The ScreenSync MCP data policy lists where every byte is stored, and the MCP security checklist for letting AI control your phone goes deeper on permissions and tokens.

Next steps

• Follow the ScreenSync MCP setup guide to install the hub, pair the phone and connect your client.
• Start read-only: ask for a screenshot and the UI hierarchy before you let the agent tap anything.
• Weigh other approaches, including accessibility-service and scrcpy-based servers, in the comparison of Android MCP servers.