AI Agent Browser Automation — Live Chrome, Saved Sessions, No Headless Trap (2026)

Most AI agent browser automation tools use headless Playwright: a fresh, anonymous browser with no saved sessions, no cookies, and no auth. Every run hits a login wall or a CAPTCHA. amux’s live-Chrome backend uses your real, saved browser sessions — so agents can access any site you’re already logged into, with no re-authentication required. Updated July 2026.

The headless trap: why most AI agent browser automation is brittle

The default approach for AI agent browser automation is to launch a Playwright browser in headless mode, navigate to the target URL, and let a vision model or DOM-reader drive the interaction. On paper, this sounds straightforward. In practice, it fails on the first step of almost every real-world task: the login page.

Headless Playwright starts a fresh, anonymous browser process every time. This means:

• No saved cookies. Every run lands on a login page, not the dashboard you wanted.
• No session tokens. OAuth flows, SSO redirects, and SAML assertions must be replayed from scratch.
• CAPTCHA on every run. Sites fingerprint headless browsers (missing plugins, predictable viewport, automation-specific headers) and serve CAPTCHAs that block the agent before it reaches any content.
• MFA required every time. If the site requires a TOTP or a push notification to log in, the agent has to trigger and answer that flow on every single run — which is either impossible unattended or requires a fragile workaround.
• Session expiration accumulates. Even if you work around the first login, tokens expire. A task that ran fine Monday fails Tuesday because the session cookie went stale overnight.

The result: agent workflows that work in a demo (against a staging site with no auth) break immediately against your real Stripe dashboard, your internal Grafana instance, or any other tool that actually requires authentication. The automation becomes a treadmill of login debugging rather than useful work.

This is the headless trap: the technology works, but only on sites that don’t matter.

The live-Chrome approach: how amux solves this

amux takes a different starting point. Instead of launching a fresh browser process, the agent loads a saved auth profile — a snapshot of your real browser’s cookies, localStorage, and session tokens, taken at the moment you were logged in. The agent then runs Playwright against this profile, which means it inherits your authentication state exactly as if you had just opened the browser yourself.

The workflow is three steps:

1. You log in once, normally, through your real browser. No scripts, no tokens, no API keys. Just open the site and log in the way you always do.
2. You save the profile via the amux dashboard (Auth Profiles tab) or the CLI. amux snapshots the browser state to ~/.amux/playwright-auth/profile.
3. Every agent run uses that profile. The agent starts with your session already active — it opens to the dashboard, not the login page. As long as the session hasn’t expired, no re-authentication is needed.

The practical impact: tasks that would fail in 30 seconds with headless Playwright (login wall, CAPTCHA, MFA) run end-to-end unattended with the live-Chrome approach. The agent walks into a browser that already knows who it is.

For sessions that do eventually expire, you refresh the profile by logging in again and saving it — a one-time manual step, not a per-run re-authentication.

How it works: the automation execution ladder

Not every browser task needs a vision-driving AI agent. amux uses a tiered approach, matching the mechanism to the task:

The amux browser agent API sits at rung 2: the AI model receives screenshots of the current page state and emits Playwright actions (click, type, scroll, navigate) in a loop until the task is done or the observation budget is exhausted. Rung 1 (CDP / direct API) is still available and preferred when you know the site’s structure; rung 3 is available for truly native workflows but is documented separately.

Code examples: running AI browser agent tasks

1. Save an auth profile via the dashboard

Open the amux dashboard, log into your target site in a separate browser tab, then save the profile. The dashboard snapshots your current session state:

# After saving via the dashboard, verify the profile exists
ls ~/.amux/playwright-auth/

# You should see: profile/ (default) or profiles// for named profiles

2. Start the browser with your saved profile

# Start the amux browser with the default saved profile
curl -sk -X POST -H 'Content-Type: application/json' \
  -d '{"profile":"default","url":"https://stripe.com/dashboard"}' \
  $AMUX_URL/api/browser/start

# Response: {"status":"started","url":"https://stripe.com/dashboard"}

3. Run an AI agent task in plain English

# Submit a plain-English task to the browser agent
curl -sk -X POST -H 'Content-Type: application/json' \
  -d '{
    "task": "Go to stripe.com/dashboard and get yesterday'\''s payout total",
    "profile": "stripe"
  }' \
  $AMUX_URL/api/browser/agent

# Response: {"status":"running","agent_id":"brw-abc123"}

4. Poll for completion

# Check the agent status (poll until status == "done")
curl -sk $AMUX_URL/api/browser/agent/status

# Response when done:
# {
#   "status": "done",
#   "result": "Yesterday'\''s payout total was $4,821.33",
#   "steps": 7,
#   "frames_used": 12
# }

5. Take a screenshot of the current browser state

# Capture a screenshot (returns the file path)
result=$(curl -sk $AMUX_URL/api/browser/screenshot)
path=$(echo "$result" | python3 -c "import sys,json; print(json.load(sys.stdin)['path'])")
echo "Screenshot saved to: $path"

# The path is a valid JPEG at ~/.amux/browser-screenshots/latest.jpg

6. Step-by-step control (when you need precision)

# Click at specific coordinates
curl -sk -X POST -H 'Content-Type: application/json' \
  -d '{"action":"click","x":640,"y":400}' \
  $AMUX_URL/api/browser/action

# Type text
curl -sk -X POST -H 'Content-Type: application/json' \
  -d '{"action":"type","text":"monthly report"}' \
  $AMUX_URL/api/browser/action

# Press a key
curl -sk -X POST -H 'Content-Type: application/json' \
  -d '{"action":"key","key":"Enter"}' \
  $AMUX_URL/api/browser/action

# Scroll down
curl -sk -X POST -H 'Content-Type: application/json' \
  -d '{"action":"scroll","dy":500}' \
  $AMUX_URL/api/browser/action

# Evaluate JavaScript and return the result
curl -sk -X POST -H 'Content-Type: application/json' \
  -d '{"action":"eval","script":"document.querySelector(\".revenue-total\").textContent"}' \
  $AMUX_URL/api/browser/action

Observation budgets: cost control for browser agents

Each time the AI browser agent takes a screenshot and sends it to the vision model, it consumes tokens — typically 1,000–3,000 input tokens per frame depending on model and image size. A task that requires 20 page views and 3 screenshots per page burns 60,000+ tokens before it does anything useful. Without a cap, slow pages, redirects, and unexpected UI states can run up large bills before the agent gives up.

Observation budgets set an upper bound on how many frames (screenshots) the agent is allowed to take per task step. When the budget is exhausted, the agent must either commit to its current best answer or report failure — it cannot keep observing indefinitely.

amux enforces observation budgets at the API level. You can set them per-task:

curl -sk -X POST -H 'Content-Type: application/json' \
  -d '{
    "task": "Find the renewal date on the billing page",
    "profile": "saas-tool",
    "max_frames": 10
  }' \
  $AMUX_URL/api/browser/agent

The default budget is generous enough for most tasks but prevents runaway cost on stuck pages. The status response includes frames_used so you can tune the budget based on what real tasks actually consume.

A reasonable starting point: 15 frames for read tasks (pulling a number, finding a status), 30 frames for form-fill tasks with validation. Increase if the agent consistently hits the budget on legitimate multi-step flows; decrease if you see high token usage on simple reads.

Composing browser agents with coding agents

Browser agents and coding agents are complementary. Browser agents are good at reading things that only exist in a web UI: dashboards, reports, status pages, rendered outputs. Coding agents are good at acting on information: writing code, updating configs, making decisions, calling APIs. Combining them is where the real productivity gain comes from.

A typical composition pattern in amux:

# Step 1: Browser agent reads the dashboard
result=$(curl -sk -X POST -H 'Content-Type: application/json' \
  -d '{"task":"Check the error rate on the Grafana dashboard for the last hour","profile":"grafana"}' \
  $AMUX_URL/api/browser/agent | python3 -c "import sys,json; print(json.load(sys.stdin).get('agent_id',''))")

# Step 2: Wait for completion
status=""
while [ "$status" != "done" ]; do
  sleep 3
  status=$(curl -sk $AMUX_URL/api/browser/agent/status | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))")
done

# Step 3: Get the result and post to a board item
answer=$(curl -sk $AMUX_URL/api/browser/agent/status | python3 -c "import sys,json; print(json.load(sys.stdin).get('result',''))")

curl -sk -X POST -H 'Content-Type: application/json' \
  -d "{\"title\":\"Investigate high error rate\",\"desc\":\"$answer\",\"status\":\"todo\",\"session\":\"investigator\"}" \
  $AMUX_URL/api/board

# Step 4: Notify the coding session directly
curl -sk -X POST -H 'Content-Type: application/json' \
  -d "{\"text\":\"Browser agent found: $answer — please investigate and fix if needed\"}" \
  $AMUX_URL/api/sessions/investigator/send

The coding session receives the dashboard reading as a natural-language message and can act on it: look at logs, write a patch, open a PR. The browser agent does not need to know anything about the codebase; the coding agent does not need to know how to read a Grafana dashboard. Each does what it is good at.

What works well vs. what doesn’t

A realistic assessment of where AI agent browser automation is reliable in 2026:

Works well

• Pulling reports from dashboards. Reading a number, a status, a table of data from a web UI. The page is mostly static after load; the agent just needs to find the right element.
• Monitoring status pages. Checking if a service is up, reading incident banners, scraping uptime percentages. Low-stakes reads that don’t require login on most sites.
• Form filling with deterministic fields. Submitting the same form fields every time (scheduled report export, standing order, recurring config update). Works well when the form doesn’t change between runs.
• Reading emails and notifications via web UI. If a tool doesn’t have an API but has a web inbox, the browser agent can read it. Combine with the amux email API when the source is Gmail.
• Administrative actions in internal tools. Approving a request, updating a user record, toggling a feature flag in a web admin panel. Works well when the steps are few and the UI is stable.

Doesn’t work well

• Sites with aggressive bot detection beyond fingerprinting. Some sites (financial institutions, certain SaaS products) use behavioral analysis, mouse-movement heuristics, and JS integrity checks that Playwright cannot fake even with a real profile. The agent passes authentication but gets blocked mid-session.
• Heavy 2FA flows during the run. If a site re-challenges with MFA mid-session (not just on login), the agent cannot open your authenticator app or approve a push notification. You can handle the initial login challenge manually; mid-session re-challenges are difficult.
• Rate-limited APIs with browser-side CAPTCHAs. If the target site intercepts high-frequency requests with a browser CAPTCHA (not just a login wall), the agent hits it repeatedly. Observation budgets help by limiting request rate, but they are not a CAPTCHA bypass.
• Multi-step write workflows with ambiguous success. Filling a form and submitting is fine. Complex multi-page wizards where each step can fail in different ways are harder — the agent may not recognize it took the wrong branch until several steps later.
• Tasks requiring real-time interaction. Chat interfaces, live collaboration tools, or anything where the page changes while the agent is reading it are unpredictable. The screenshot-based loop assumes pages are relatively stable between frames.

Comparison: amux live-Chrome vs. headless Playwright vs. RPA tools

Frequently Asked Questions

# Homebrew (recommended)
brew install mixpeek/amux/amux

# PyPI
pipx install amux

# Zero-install (uvx)
uvx amux serve

amux serve  # → https://localhost:8822

Related guides

• AI agent automations overview
• Getting started with amux
• Claude Code resume dialog auto-answer
• AI agent quality gates
• Scheduling AI agents
• Multi-agent orchestrator architecture