The Human Taps, the Agent Reads the Logs: AI Agents on Real Phones

Our unit tests were green. The build was green. The app crashed on launch.

That happened more than once on the production product we build with AI coding agents. Unit tests stub out images and native modules, so whole categories of bugs never show up in them: a launch crash on a brand-new OS version, a label clipped on one manufacturer's Android skin, a bottom sheet that silently swallows taps. You only see those on hardware.

So we stopped treating "please test it on your phone" as the end of the agent's job, and made the device part of the harness.

The split: who does what

The rule that made everything else work came from a skill we wrote for the agent. Its core line:

The agent reads a 5 MB logcat in milliseconds but cannot tap a screen. The human drives the device but should never have to grep.

In practice:

• The agent owns the diagnostic chain. It builds, installs, starts the bundler from the right checkout, attaches log capture, takes screenshots, reads the app's own API log and decides what the evidence means.
• The human owns what the agent physically can't do. Unlocking the phone, typing a one-time password, signing in, and tapping through a flow that automation can't reach.
• Every hand-off is one exact step list. Not "test the checkout", but "open the app, tap History, tap the second row, tell me when the sheet opens."

Over about four weeks on one of the mobile apps, the developer replied "done" to those step lists around thirty times. Each "done" was followed by the agent reading logs and screenshots, not by the human describing what they saw.

Prove the chain before you ask a human

The most expensive mistake in device work is asking a person to test a build that isn't the build you think it is. So before any "please test", the skill runs a preflight:

• exactly one device is connected, and it's the one we expect
• a debuggable build is installed
• the JavaScript bundler is serving from *this* worktree, not the main checkout (we prove it with a log line injected at build time)
• log capture is running, detached, into a known file
• the agent has taken a screenshot and actually looked at it

"Prove the chain, don't assert it." A surprising number of "the fix doesn't work" reports were really "the phone was running yesterday's bundle."

A driver script the agent can use on its own

Halfway through the project we turned the device workflow into a committed project skill, backed by a shell script of about 200 lines. The script starts the bundler on a free port from the current checkout, relaunches the app, takes screenshots the agent can read, taps at screenshot coordinates and prints the app's API log.

Over the following weeks the agent called it about 1,500 times: roughly 560 screenshots and 450 taps, on top of about 800 direct adb calls. For most screens, "please test" became the agent's own job. The human came in only for logins and one-time passwords.

The script collected its own gotchas too. Two of our favourites:

• back on a tab root closes the app. One run opened the phone's dialler instead.
• Never uninstall the production app on a shared phone. It's someone's real login.

Simulators were a mirage for us

Simulators barely featured: on the consumer app, single-digit simulator commands against hundreds of real-device ones. Several native SDKs in that app ship device-only binaries, so a simulator build couldn't run those flows at all. Rather than fight that, we put the effort into the physical-device driver.

When we did use a simulator, it was to reproduce a launch crash on a brand-new OS version for the business app, and there the agent had no way to tap. It called the app's JavaScript login function directly over the bundler's debugger protocol to trigger the system sign-in prompt. Not elegant, but it verified the fix.

Reproduce with the real thing, not a shell shortcut

A few bugs taught us that a shortcut reproduction can hide the behaviour real users hit:

• Links opened from a mail app. Opening a link with a plain shell am start adds launch flags that a real mail app doesn't. A link tapped in a mail app created a second copy of the app's screen stack; our shell reproduction never did. We only trusted the fix after a 300-tap stress run through the real apps: 100 out of 100 clean, against 80 out of 100 before.
• Coming back from a browser. The browser handed control back to the app in a way that destroyed the screen the user had come from, leaving them stuck mid-flow. Only the real browser hand-back reproduced it.
• Older OS versions. A hardening change passed on the newest iOS and on macOS, then failed on an older iPhone because that OS version was missing a newer root certificate. The fix was on the server's certificate chain. We checked it against each OS version's root store, then on the actual old phone. No app release was needed.

The lesson: test on the oldest OS you support, not just the newest, and reproduce with the app your user will actually use.

When the app was right

Device loops are also how you find out the code is fine. In one session the developer reported "unable to tap" on a button the agent had just built. The logs showed the feature was switched off for that test account. Second attempt, "still unable to tap": the device clock showed the allowed time window had passed. The third attempt surfaced a real bug: a success response came back without an ID the next step needed, and the app had been treating any success status as success. That became a fix with a test. A separate failure further down the chain went to the client's backend team.

Without the logs, the first two reports would have been "fixed" by changing code that wasn't broken.

Sharing one phone between several agents

Later we ran several agent sessions in parallel, and they shared the same physical phones. That created a new class of failure: one session restarting the bundler cut off another session's log watchers. We fixed it with a plain protocol:

• each session gets its own bundler port and pins the device by serial number
• a session says so before it takes the device, and the developer can answer "Android is busy, check iOS first"
• a session stops only the processes it started

What to copy

• Write down the human/agent split, and put it in a skill the agent loads rather than in your head.
• Make the agent prove which build is on the device before anyone taps anything.
• Commit a device driver script so the agent can screenshot and tap on its own.
• Reproduce with real apps and real OS versions, including the oldest one you support.
• When the agent "couldn't reproduce it", go and find the real sequence. Don't send that reply to QA.