Browser
Let an agent use a real Chrome or Edge on the HQ computer, with saved profiles, a live view that you can control, and clear limits.
The browser tool lets an agent use websites as you do. It can open a page, read it, click, type, fill in forms, change tabs, and download files. It uses a real Google Chrome or Microsoft Edge on the computer that runs your HQ.
The browser is headless: it shows no window on the screen. You can watch it, or control it, in the live view.
Use the browser when web_fetch is not sufficient. Examples are pages that need JavaScript, sites that need a sign-in, forms with many steps, and tasks where the agent must click through pages.
The browser does not try to look like a person. Its user agent says Chrome, not HeadlessChrome. All other automation signals, for example navigator.webdriver, stay the same. The browser does not try to get past bot checks. If a site asks for a CAPTCHA or a 2FA code, a person must solve it in the live view. Refer to Hand off a CAPTCHA or 2FA prompt.
Get a browser
The HQ looks for a browser in this sequence. It uses the first browser that starts:
- The path in
BROWSER_EXECUTABLE_PATH, if you set it and the file is there. Refer to Configuration. - Google Chrome, then Microsoft Edge, in the usual install location:
- macOS:
/Applicationsor~/Applications. - Windows: the Chrome and Edge install folders under
%LOCALAPPDATA%,%PROGRAMFILES%and%PROGRAMFILES(X86)%. - Linux:
google-chrome,google-chrome-stable,chromium,chromium-browserormicrosoft-edgeon yourPATH.
- macOS:
- A managed browser that dashsquad downloaded for you.
If an installed browser does not start in approximately 15 seconds, the HQ tries the next browser.
If the HQ finds no browser, the agent gets this error:
No browser available. Open dashsquad Desktop → Settings → Browser to install one.The managed browser
If you do not have Chrome or Edge, Desktop can download a browser for your agents. The download is approximately 190 MB, and the browser uses approximately 370 MB on the disk. dashsquad downloads it only when you tell it to.
You can start the download from two locations:
- The install card. If an agent has the browser on and the HQ finds no browser, the agent page shows the card Your agent needs a browser. When you create an agent, Advanced shows the same card, because the form selects the Web group by default. Click Download browser (about 190 MB) to download it. Or click I'll install Chrome myself to open the Chrome download page. When the download is complete and the browser passes a test start, the card shows Browser ready.
- Settings → Browser. Click Download browser. The page shows a progress bar. Then it shows
Checking that it starts…while the HQ starts the new browser one time as a test.
The managed browser updates itself one time each week. An update does not stop a browser that an agent uses. It waits until no browser is open. You can also click Check for an update now on the Browser settings page.
If you install Chrome or Edge later, click Check again on Settings → Browser, or restart the HQ. Then your agents use the new browser. The managed browser then stops its updates. It stays on the disk, and the Browser settings page tells you this.
Settings → Browser
Open Settings → Browser in Desktop to see which browser your agents use: Google Chrome, Microsoft Edge or the managed browser. The page shows the full path of the browser and the number in Agent browsers open right now. This number counts only the browsers that your agents use. Your own Chrome windows are not in it. If the browser did not start in time, the page shows its name with (did not start). Click Check again after you install or remove a browser.
Turn it on for an agent
- In Desktop, open the agent page and click the Configuration tab.
- Open the Tools card.
- Select the Web group.
- Click Save.
The Web group turns on web_search, web_fetch and the browser together. It is one checkbox for the full group. You can also select the group when you create an agent in the deploy wizard.
When the browser is on, the agent page gets a Browser tab. Use it to manage the profiles of the agent.
The browser tools
When you turn on the browser, the agent gets the tools in this table. Each browser belongs to one conversation. Two conversations with the same agent get two different browsers.
| Tool | What it does | Parameters |
|---|---|---|
browser_open | Opens a browser for this conversation and gives a snapshot. With profile, it opens one of your persistent profiles, where cookies and sign-ins stay. Without profile, it opens a new temporary browser. | profile (optional): the profile name (a-z, 0-9 and "-"). The HQ creates the profile if it is new.url (optional): a URL to load after the browser opens. |
browser_navigate | Loads a URL, or goes back, goes forward or loads the page again. Gives the new snapshot. | url (optional): an http or https URL.action (optional): back, forward or reload. |
browser_snapshot | Reads the current page. After a new snapshot, the refs from earlier snapshots are not valid. | mode (optional): interactive (default) gives the accessibility tree with [ref=...] markers. text gives the page text without refs, and uses fewer tokens. |
browser_act | Does an action on an element (by its ref from the latest snapshot), pushes a key, or answers an open dialog. Gives the new snapshot. | action (required): click, type, select, hover, press, upload or dialog.ref (optional): an element ref from the latest snapshot, for example e12 or f1e12.value (optional): the text for type, the option label for select, the key name for press (Enter, Tab), a workspace file path for upload, or accept or dismiss for dialog. |
browser_screenshot | Captures the page or one element as an image. The agent uses it when the layout or the visual appearance is important. | ref (optional): capture only this element.fullPage (optional): true captures the full page, not only the visible part. The maximum height is 8,000 px. |
browser_tabs | Lists, opens, selects or closes tabs. | action (required): list, new, switch or close.index (optional): a tab number from the list. |
browser_profiles | Lists your persistent profiles, or deletes one profile by its name. | action (required): list or delete.name (optional): the profile to delete. |
browser_close | Closes the browser of this conversation. A named profile keeps its cookies. | None. |
Important points:
- Only image-capable models get
browser_screenshot. If the model of the agent cannot read images, the agent does not get this tool. It uses snapshots. - Sub-agents get only temporary browsers. A sub-agent with the browser gets its own browser, which is not the browser of its parent. It cannot open a named profile: its
browser_openhas only theurlparameter. It does not getbrowser_profiles. - Web pages are not trusted. The HQ marks all browser results to the agent as web page data, not as instructions.
- Downloads go to a
downloadsfolder in the workspace of the agent. Uploads (browser_actwithupload) accept only files in the workspace of the agent. - The browser opens only
httpandhttpsaddresses. - The browser refuses all site permissions. For example, it refuses requests for your location, camera or notifications.
Profiles
When the agent opens a browser, it selects one of two types of session:
- A temporary session: a new browser with no cookies and no history. The HQ deletes it when the browser closes. The agent gets this type if it does not give a profile name.
- A named profile: a persistent browser profile with a name that the agent selects, for example
shoppingorwork-email. Cookies, sign-ins and site data stay between conversations, so you sign in only one time. If the profile does not exist, the HQ creates it the first time the agent opens it.
The agent selects the type each time it opens the browser. If you want a specific profile, tell the agent to use it. Profile names use lowercase letters, digits and -, with a maximum of 32 characters. Each agent has its own profiles. An agent cannot open the profiles of a different agent.
One conversation at a time. Only one conversation can use a profile at a time. If a different conversation has the profile open, the agent gets this error:
profile "<name>" is in use by conversation "<conversation>"<conversation> is the title of that conversation, or its ID if it has no title. To free the profile, tell the agent in the other conversation to close its browser. Or wait: the browser closes automatically after 15 minutes with no activity.
The HQ keeps profiles on its computer under ~/.dash/gateway/browser-profiles/, with one folder for each agent. For an HQ that Desktop starts, ~/.dash/gateway on this page means ~/.dashsquad/<hq id>. See Data folders.
Manage profiles in Desktop
Open the agent page in Desktop and click the Browser tab. The tab shows only when the agent has the browser turned on. For each saved profile, the tab shows its name, the time of last use and its size on the disk. An In use badge shows if a conversation has the profile open.
To delete a profile, click Delete and confirm. When you delete a profile, the agent is signed out of all the sites in it. Delete is not available while a conversation uses the profile. Then the tab shows Close the agent's browser first.
If an agent has no saved profiles, the tab shows: This agent has no saved browser profiles. It creates one when it opens the browser with a profile name.
An agent can also list and delete its own profiles with browser_profiles.
If you try to open or delete a profile before deletion completes, the HQ returns profile "<name>" is being deleted. Wait for deletion to finish, then retry. If you use that name again after deletion, the HQ creates a new, empty profile. Other profiles remain available.
Live view
The live view shows you the browser of the agent while it works. Between turns, you can use the browser yourself. The live view is available in Desktop, the web app, the iOS app (iPhone and iPad) and the Android app.
Open it
In a chat with an agent that has a browser open, open View browser:
| App | Where to find View browser | How it opens |
|---|---|---|
| Desktop | A globe icon in the status bar of the chat | A panel next to the chat |
| Web app | A button in the top bar | A column next to the chat. If the window is 768 px wide or less, a full-screen overlay. |
| iOS app | The ⋯ menu of the conversation | Full screen |
| Android app | A globe icon in the top bar | Full screen |
The view shows Connecting to the browser… for a short time. Then it shows the page title, the page address and a live picture of the page.
If the conversation has no open browser, the view shows No browser is open in this conversation.
Who has control
The top of the view shows who controls the browser:
Agent is driving: a turn is in progress. You can watch, but your input does not go to the page.- You have control: the agent does not work at this time. Use the picture of the page as you use a usual browser.
When you have control in Desktop or the web app:
- Keys go to the page, not to the app.
- Cmd+V (Ctrl+V on Windows and Linux) pastes your clipboard into the page. The HQ cuts pasted text to 4,096 characters.
- If the page opens a dialog, the view shows
The page is showing a dialog: "…"with the message of the dialog.
When you have control in the iOS app or the Android app:
- Tap to click. Drag to scroll.
- Type in the text field above the keyboard, then push Return to send the text to the page. If the field is empty, Return pushes the Enter key of the page.
- You cannot paste or hover.
- Dialogs show the same message as in Desktop.
If you only watch, the browser does not stay open. A browser that nobody uses for 15 minutes closes, also when the live view is on the screen. If you use the browser yourself, that counts as use.
One viewer at a time
Each browser can have only one open live view. If you open the live view in a different window or app, the first live view stops. When a live view stops, it tells you the cause:
| Message | Cause |
|---|---|
| Live view ended. (opened somewhere else) | A person opened the live view for this browser in a different window or app. |
| Live view ended. (the browser closed) | The agent closed the browser, or the browser closed after 15 minutes with no activity. |
| Live view ended. (connection lost) | The app lost its connection to the HQ. In the iOS app and the Android app, this also occurs when you move the app to the background. Open the view again when you come back. |
You cannot open a live view of the browser of a sub-agent.
Hand off a CAPTCHA or 2FA prompt
The browser does not solve CAPTCHAs or sign-in challenges. In the instructions of the agent, or in the chat, tell the agent to stop and ask you when it finds one. Then do these steps:
The agent stops and asks
The agent tells you that it cannot continue. For example, a sign-in page shows a CAPTCHA or asks for a code. Then it ends its turn.
Solve it in the live view
Open View browser. When the view shows You have control, solve the CAPTCHA or type the code into the page.
Reply to the agent
Tell the agent that you are done. Use the same conversation.
The agent continues
The agent continues from the page where you stopped. If it used a named profile, the profile keeps the sign-in for the next time.
Screenshots in chat
When an agent uses browser_screenshot, the tool row in the chat shows a thumbnail of the capture. All apps show it, also the iOS app and the Android app.
- In Desktop and the web app, click the thumbnail to make it larger. Click again to make it smaller.
- In the iOS app and the Android app, tap the thumbnail to open the screenshot full screen. Pinch to zoom, or double-tap to zoom in and out. A tall screenshot opens at the full width of the screen, at the top of the page. Close it from there.
The conversation keeps the thumbnail.
Limits
| Limit | Value |
|---|---|
| Tabs in each browser | 8. A request for a ninth tab fails with a session can hold at most 8 tabs. If a page opens a tab above this limit, the browser closes that tab immediately. You cannot close the last tab. |
| Browsers open at the same time, for each HQ | 3 by default, for all agents and conversations. One more fails with this HQ is already running <n> browser sessions, where <n> is the limit. Change the limit with BROWSER_MAX_SESSIONS. |
| Screenshot height | 8,000 px. If a full page or an element is taller, the screenshot shows only its top 8,000 px. The result tells the agent the full height. |
| Time with no activity | A browser that nobody uses for 15 minutes closes automatically. A named profile keeps its cookies. The HQ deletes a temporary session. |
Privacy and security
-
The browser cannot get access to the HQ. Pages cannot load the ports of the HQ on the HQ computer. This is true for
localhost,127.0.0.1, the network address of the computer and its hostname. A web page thus cannot send the browser of the agent into the management API or the chat API of your HQ. Other local addresses stay open, so an agent can test a web app that you run locally.If the agent tries to open a blocked port, it gets
port <port> on this machine is not allowed. If a page tries to load a blocked port, the browser stops the request. -
The logs do not keep what you see and type in the live view. The HQ does not write live-view pictures or your input to its logs. It records only the type of input (a click or a key), not the text that you typed. The agent does not see the live view or your input.
-
Profiles keep real sign-ins. The cookies in a profile stay signed in to the sites that the agent used. On macOS and Linux, only the user account that runs the HQ can read the profile folders under
~/.dash/gateway/browser-profiles/. But each person who signs in as that user can read them. Delete the profiles that you do not need.
Configuration
Two environment variables change how the browser works. Set them where the HQ runs. If Desktop manages the HQ for you, set them in the environment that starts Desktop. The HQ gets them from Desktop. Restart the HQ (or Desktop) to apply a change.
| Variable | Description |
|---|---|
BROWSER_MAX_SESSIONS | The number of browsers that the HQ can run at the same time, for all agents. A whole number from 1 to 10 (default: 3). The HQ ignores all other values, writes a warning to its log, and uses the default. |
BROWSER_EXECUTABLE_PATH | The full path of the browser to use. The HQ looks here first. If there is no browser at this path, or if the browser does not start, the HQ looks for Chrome or Edge as usual. |
If you have a problem, refer to Browser problems.