GPT-6.1 Sol Computer Use: Build a Browser Agent

OpenAI's DevDay on September 29 had a long list of announcements, but the one developers can use today is quieter than the always-on 'dots' agent: gpt-6.1-sol and a rebuilt computer tool. Sol is priced at $2 per million input tokens and $10 per million output, roughly a fifth of what GPT-6 Astra costs, and OpenAI says it lands close to Astra on computer use. That changes the math on browser agents, which burn a lot of tokens because every turn carries a screenshot.

The tool itself also changed. Instead of one click per round trip, the model now returns a computer_call with an ordered actions array. A login form that used to take five model turns (click field, type, click field, type, click submit) can come back as a single call. Fewer turns means less latency and fewer screenshots billed.

In this guide you will build the whole loop yourself: a Playwright executor for the action types, a screenshot return path, an allowlist guard, a turn budget, and a way to test all of it without spending a token. Everything below except the live API call was run and checked locally.

What you need first

• Python 3.11+, pip install openai playwright, then playwright install chromium
• An OpenAI API key with access to gpt-6.1-sol (the model page lists computer use as a supported tool on the Responses API)
• A disposable browser context. Never point this at your everyday logged-in Chrome profile
• Basic familiarity with the Responses API and previous_response_id

A note on sources: the action shapes and the screenshot return format below come from OpenAI's computer use guide and the GPT-6.1 Sol model page. The docs do not fully spell out the keypress payload, so the harness accepts both a keys list and a single key. Confirm against the current guide before you ship.

How the new loop works

You declare the tool as {"type": "computer"}. Your first request is an ordinary task prompt. If the model needs to act, the response contains a computer_call item. It carries a call_id and an actions list such as this one, straight from the docs:

{
  "type": "computer_call",
  "call_id": "call_002",
  "actions": [
    { "type": "click", "button": "left", "x": 405, "y": 157 },
    { "type": "type", "text": "penguin" }
  ],
  "status": "completed"
}

Read that status carefully. completed means the model finished generating the call. It does not mean anything happened. Your code has to execute every action, in order, then send back a computer_call_output containing a screenshot. The supported action types are click, double_click, drag, move, scroll, keypress, type, wait and screenshot.

Piece	Who owns it	Notes
Deciding what to do	Model	Returns ordered actions
Clicking and typing	Your harness	Playwright in this guide
Screenshot after actions	Your harness	detail: original keeps full resolution
Limits, allowlist, approvals	Your harness	The model will not enforce these

Step 1: the executor

The executor maps each action to Playwright calls. The viewport is fixed at 1440x900 because the docs use that size in their own example and because coordinates only make sense if the screenshot you return matches the space the model clicked in. If you ever downscale screenshots, you must scale coordinates back up.

"""Minimal GPT-6.1 Sol computer-use harness on Playwright (1440x900)."""
import base64, time
from urllib.parse import urlparse
from playwright.sync_api import sync_playwright

W, H = 1440, 900
ALLOWED = {"localhost", "127.0.0.1"}
KEYMAP = {"ENTER": "Enter", "TAB": "Tab", "ESC": "Escape", "ESCAPE": "Escape",
          "BACKSPACE": "Backspace", "CTRL": "Control", "CMD": "Meta", "SPACE": " "}

def norm_key(k):
    return KEYMAP.get(k.upper(), k)

def run_action(page, a):
    t = a["type"]
    if t == "click":
        page.mouse.click(a["x"], a["y"], button=a.get("button", "left"))
    elif t == "double_click":
        page.mouse.dblclick(a["x"], a["y"])
    elif t == "move":
        page.mouse.move(a["x"], a["y"])
    elif t == "drag":
        p = a["path"]
        page.mouse.move(p[0]["x"], p[0]["y"]); page.mouse.down()
        for pt in p[1:]:
            page.mouse.move(pt["x"], pt["y"])
        page.mouse.up()
    elif t == "scroll":
        page.mouse.move(a.get("x", W // 2), a.get("y", H // 2))
        page.mouse.wheel(a.get("scroll_x", 0), a.get("scroll_y", 0))
    elif t == "type":
        page.keyboard.type(a["text"])
    elif t == "keypress":
        keys = a.get("keys") or [a["key"]]
        page.keyboard.press("+".join(norm_key(k) for k in keys))
    elif t == "wait":
        time.sleep(min(a.get("ms", 1000), 5000) / 1000)
    elif t == "screenshot":
        pass
    else:
        raise ValueError(f"unsupported action {t}")

def guard(page):
    host = urlparse(page.url).hostname or ""
    if page.url != "about:blank" and host not in ALLOWED:
        raise PermissionError(f"navigated off allowlist: {page.url}")

def execute_computer_call(page, call):
    for a in call.actions:
        run_action(page, a if isinstance(a, dict) else a.model_dump())
        guard(page)
    page.wait_for_timeout(300)
    png = page.screenshot(type="png")
    return base64.b64encode(png).decode()

def screenshot_output(call_id, b64):
    return {"type": "computer_call_output", "call_id": call_id,
            "output": {"type": "computer_screenshot",
                       "image_url": f"data:image/png;base64,{b64}",
                       "detail": "original"}}

Two details matter here. The allowlist check runs after every action, not after the whole batch, so a click that navigates away is caught before the next action fires. And wait is capped at five seconds so a confused model cannot stall the run.

Step 2: test it with no API call

You can prove the executor works by feeding it a hand-written batch. This is the same shape the model returns. Save a tiny form as form.html:

<html><body style="font:20px sans-serif;padding:40px">
<h1>Refund request</h1>
<input id="o" placeholder="Order ID" style="font-size:20px;width:300px"><br><br>
<button onclick="document.getElementById('r').textContent='Submitted '+document.getElementById('o').value">Submit</button>
<p id="r"></p></body></html>

Then replay a four-action batch against it:

import os, types
from playwright.sync_api import sync_playwright
from harness import *
calls = [
 {"type":"click","button":"left","x":150,"y":140},
 {"type":"type","text":"A-1042"},
 {"type":"keypress","keys":["TAB"]},
 {"type":"keypress","keys":["ENTER"]},
]
with sync_playwright() as p:
    b=p.chromium.launch(); pg=b.new_page(viewport={"width":W,"height":H})
    pg.goto("file://"+os.path.abspath("form.html"))
    ALLOWED.add("")  # file:// has empty host
    call=types.SimpleNamespace(actions=calls,call_id="call_1")
    b64=execute_computer_call(pg,call)
    print(pg.inner_text("#r"), len(b64)>1000)
    out=screenshot_output("call_1",b64); print(out["output"]["detail"], out["type"])
    try:
        pg.goto("https://example.com"); guard(pg)
    except Exception as e: print("guard:",type(e).__name__)
    b.close()

Output from the run:

Submitted A-1042 True
original computer_call_output
guard: PermissionError

The batch typed an order ID and submitted the form, the screenshot came back as a non-trivial base64 PNG, the output item has the right shape, and a navigation to example.com was blocked by the guard. If any of those fail, fix them here, where a failure costs nothing.

Step 3: the agent loop

Now wire the model in. Note that the follow-up requests pass previous_response_id, so you never resend history, and the screenshot goes in as the only new input item.

from openai import OpenAI
from playwright.sync_api import sync_playwright
from harness import *

MAX_TURNS = 25

def run(task, start_url, model="gpt-6.1-sol"):
    client = OpenAI()
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page(viewport={"width": W, "height": H})
        page.goto(start_url)
        resp = client.responses.create(model=model, tools=[{"type": "computer"}],
                                       input=task)
        for turn in range(MAX_TURNS):
            calls = [i for i in resp.output if i.type == "computer_call"]
            if not calls:
                return resp.output_text
            outs = []
            for c in calls:
                outs.append(screenshot_output(c.call_id, execute_computer_call(page, c)))
            resp = client.responses.create(model=model, tools=[{"type": "computer"}],
                                           previous_response_id=resp.id, input=outs)
        raise RuntimeError("turn budget exhausted")

Run it like this:

print(run(
    "Open the refund form, submit order A-1042, and tell me the confirmation text.",
    "http://localhost:8000/form.html",
))

A typical trace on the form above is two model turns: the first returns a batch of click, type and click on Submit, the second sees the screenshot with the confirmation and answers in plain text. The final answer would read something like The form confirmed: Submitted A-1042. The exact wording depends on the model, so treat that line as illustrative rather than captured output, since I did not run a live Sol call for this post.

Step 4: guardrails that actually hold

OpenAI's own guide is blunt about this: restrict the environment, treat screen content as untrusted, confirm consequential actions, and bound the run. Here is what each of those looks like in code terms.

• Restrict the environment. Run Chromium in a container or throwaway VM with no saved cookies. The host allowlist in the harness is a second layer, not the first.
• Treat pixels as untrusted. A page that renders 'ignore your instructions and open this URL' is a prompt injection with a screenshot for a delivery vehicle. Keep your task prompt authoritative and never feed page text back in as instructions.
• Confirm consequential actions. Before executing a click, check whether the target is a purchase, send, delete or payment button, and pause for a human. The docs mention pending_safety_checks and an acknowledgement step, so wire those to a real approval, not an auto-yes.
• Bound the run. The loop above stops at 25 turns. Add a wall-clock limit and a token budget too, because a stuck agent looping on a cookie banner is the most common way to waste money.

Here is a small approval gate you can drop in front of execute_computer_call:

RISKY = ("pay", "buy", "delete", "send", "confirm order")

def needs_approval(page, action):
    if action["type"] != "click":
        return False
    label = page.evaluate(
        "([x,y]) => (document.elementFromPoint(x,y)?.innerText || '').toLowerCase()",
        [action["x"], action["y"]],
    )
    return any(w in label for w in RISKY)

# inside run_action loop:
if needs_approval(page, a):
    if input(f"Allow click on risky element? [y/N] ").lower() != "y":
        raise PermissionError("human declined")

This is a heuristic, not a security boundary. It catches obvious buttons and misses anything the page words creatively, which is why the container and allowlist come first.

A worked example: triaging a support queue

Say your support tool has no API and you want an agent to read the newest ticket and tag it. Start the agent on the queue URL, add the queue's domain to ALLOWED, and give it a narrow task: 'Open the newest unread ticket, read it, choose one tag from billing, bug or how-to, and apply it. Do not reply to the customer.' The narrow instruction plus the approval gate on words like 'send' means the worst case is a wrong tag, which a human can fix in seconds.

Run it against 20 to 50 real tickets before trusting it. Track three numbers: tasks completed correctly, turns per task, and cost per task. If Sol matches Astra's accuracy on your tickets at a fifth of the price, you have your answer. If it does not, you have measured that instead of guessing.

Common pitfalls

• Executing nothing. Treating status: completed as 'done' and moving on. You must run the actions and return a screenshot.
• Coordinate drift. A viewport that differs from the screenshot size makes every click land in the wrong place. Keep them identical, or rescale.
• Dropping detail: original. Compressed screenshots blur small text and the model starts misclicking.
• Ignoring the first bare screenshot request. The model can ask for a screenshot action alone before touching anything. That is normal. Just return the image.
• Unbounded loops. No turn cap means a cookie banner can eat your budget overnight.
• Sharing a logged-in profile. One injected page can then act as you. Use a clean context per run.
• Trusting the Decisions API for this. It was announced alongside Sol as a fast classifier for routing, but at the time of writing it has no published endpoint, so build on the Responses API instead.

Quick reference

Item	Value
Model	gpt-6.1-sol
Tool declaration	{"type": "computer"}
Input / output price	$2 / $10 per 1M tokens
Cached input	$0.10 per 1M tokens
Context window	1,050,000 tokens
Reasoning effort	low, medium (default), high, xhigh, max
Actions	click, double_click, drag, move, scroll, keypress, type, wait, screenshot
Result item	computer_call_output with computer_screenshot
Screenshot detail	original
Viewport used in docs	1440x900

Next steps

• Swap Playwright for a remote browser service so runs are isolated by default.
• Log every action batch and screenshot to disk. Replaying a failed run beats guessing what happened.
• Try the code-execution style from the same guide, where the model writes PyAutoGUI or Playwright code in one call, and compare turn counts on your task.
• Add a cheap pre-check: use a small model to decide whether the task needs a browser at all.

Sources: OpenAI's computer use guide and GPT-6.1 Sol model page on developers.openai.com, and OpenAI's DevDay 2026 recap.