Forum Example

--- name: ubuntu-browser-session description: Use when a request needs a real Ubuntu Server browser session with durable site login reuse, bounded manual login recovery, or host-side page inspection for protected sites. metadata: version: 1.0.3 --- # Ubuntu Browser Session Use the Ubuntu host browser like a long-lived local browser for the agent. The primary model is site-centric: - each important site keeps one default primary identity - agent tasks reuse that default identity automatically - non-default identities are allowed only when the user explicitly names a different `session-key` - user help is only for first login, expired sessions, or unrecoverable challenges ## When To Use - Open, inspect, or interact with a protected site from the Ubuntu Server host - Reuse an already logged-in site session such as GitHub or Google - Recover an expired or challenge-blocked session with one bounded round of user help - Continue browser work on the host without re-authenticating every task Representative requests: - "Use the Ubuntu host browser and open this dashboard." - "Reuse the existing GitHub login on the server." - "Continue from the same Google session." - "Recover this protected page with the least user help." ## Do Not Use - General web research that only needs HTTP fetch or web search - Local desktop browsing outside the Ubuntu Server host - Cases where an API token or non-browser auth flow is enough ## Core Rules - Default to one primary identity per site - Do not guess between multiple accounts automatically - Only use a non-default identity when the user explicitly names a different `session-key` - Prefer local recovery before asking the user to take over - Treat wrong-page drift as recoverable: first navigate back to the requested site in the same profile, then ask for help only if the target still lands on a login wall or challenge - Finish the host-browser workflow before switching tools: if `open-protected-page.sh` returns `ready`, keep working from this skill's host browser context and do not switch to OpenClaw's generic `browser` profile or other browser tools unless this workflow is exhausted or the task explicitly requires a different browser stack ## Preferred Entry Point Use the wrapper first: ```bash {baseDir}/scripts/open-protected-page.sh --url 'https://target.example' --session-key default ``` The wrapper is responsible for: - site-centric profile lookup - session reuse - target-page verification - one automatic recovery attempt when the browser drifted to the wrong page - bounded escalation to noVNC when user help is actually needed - keeping work inside the host browser workflow by default instead of mixing in unrelated browser profiles ## Site Session Model Durable default identities are tracked by canonical site, not just exact origin. Examples: - `github.com` -> default GitHub browser identity - `google.com` -> default Google browser identity, including `myaccount.google.com` Persistent state is split across: - exact manifests for the live runtime and captured browser state - a site session registry for default per-site reuse - compatibility identity aliases for Google-family hosts Important files: - `~/.agent-browser/index/site-sessions.json` - `~/.agent-browser/sessions/...` - `~/.agent-browser/index/identity-profiles.json` ## Manual Login Recovery When the wrapper cannot recover locally, it starts the assisted browser overlay and returns: - a loopback noVNC URL for SSH tunnel usage - a LAN noVNC URL when the host IP is known and noVNC is exposed on `0.0.0.0` Typical outputs: ```text http://127.0.0.1:6084/vnc.html?autoconnect=1&resize=remote http://192.168.0.200:6084/vnc.html?autoconnect=1&resize=remote ``` Use the loopback URL if you are forwarding ports over SSH from Windows or another remote machine. Use the LAN URL only when the host firewall and network allow direct access. ## Typical Workflow 1. Try `open-protected-page.sh` with the requested URL and desired `session-key` 2. Let the wrapper resolve the default site identity 3. If the browser is already logged in but sitting on the wrong page, let the wrapper navigate back to the requested site automatically 4. If the target page is ready, continue the task from this host browser workflow first 5. If the target page still shows a login wall or challenge, open the returned noVNC URL 6. After the user finishes the login and leaves the final page loaded, run: ```bash {baseDir}/scripts/assisted-session.sh capture --origin 'https://target.example' --session-key default ``` That finalizes the wrapper-managed session state and updates the site session registry for later reuse. Do not use `assisted-session.sh start` as a normal entrypoint. Start with `open-protected-page.sh` so site-centric profile resolution and wrong-page recovery stay in effect. Only reach for unrelated browser tooling after this workflow fails to complete the task and you have a concrete reason the host browser workflow is insufficient. ## Environment Checks ```bash command -v python3 command -v curl command -v jq command -v Xvfb command -v x11vnc command -v websockify command -v google-chrome || command -v chromium || command -v chromium-browser ``` ## Key Files - `scripts/open-protected-page.sh`: main protected-site wrapper - `scripts/assisted-session.sh`: bounded manual takeover and capture - `scripts/site-session-registry.sh`: canonical per-site default session registry - `scripts/profile-resolution.sh`: site-first profile selection with compatibility fallback - `scripts/session-manifest.sh`: runtime manifest storage and verification support - `scripts/browser-runtime.sh`: browser runtime, target selection, and page checks - `scripts/cdp-eval.py`: CDP evaluation, page checks, and navigation - `scripts/cdp-snapshot.py`: structured page content extraction via CDP ## CDP Tools `cdp-eval.py` and `cdp-snapshot.py` stay generic by default. Forum/search-result helpers are optional enhancements; read `references/forum-enhancements.md` only when the current page is a forum topic list, category page, or search results page and you need structured topic/result links or text-based clicking. ### cdp-eval.py Evaluate page state or run JavaScript over the Chrome DevTools Protocol. ```bash # Check for challenge pages python3 {baseDir}/scripts/cdp-eval.py --port PORT --check challenge # Check for login walls python3 {baseDir}/scripts/cdp-eval.py --port PORT --check login-wall # Get page info (title, url, body snippet) python3 {baseDir}/scripts/cdp-eval.py --port PORT --check page-info # Run arbitrary JavaScript python3 {baseDir}/scripts/cdp-eval.py --port PORT --eval 'document.title' # Click the first visible link whose text matches or contains the given phrase python3 {baseDir}/scripts/cdp-eval.py --port PORT --click-link-text 'Pricing' # Navigate to a URL and wait for load python3 {baseDir}/scripts/cdp-eval.py --port PORT --navigate 'https://example.com' --wait-navigation # Navigate, wait for load, then check page state python3 {baseDir}/scripts/cdp-eval.py --port PORT --navigate 'https://example.com' --wait-navigation --check page-info # Navigate and wait for a specific element to appear (10s timeout) python3 {baseDir}/scripts/cdp-eval.py --port PORT --navigate 'https://example.com' --wait-for '#main-content' ``` Parameters: - `--port PORT` (required): CDP HTTP/WebSocket port - `--target-id ID`: specific target from `/json/list` - `--check {challenge,login-wall,page-info}`: built-in page state checks - `--eval EXPRESSION`: arbitrary JavaScript for `Runtime.evaluate` - `--navigate URL`: navigate to URL via `Page.navigate` - `--click-link-text TEXT`: click the first visible matching anchor; when combined with `--navigate`, pair it with `--wait-navigation` or `--wait-for` so the destination DOM is ready before clicking - `--wait-navigation`: wait for `Page.loadEventFired` after `--navigate` - `--wait-for SELECTOR`: poll for CSS selector to appear (timeout 10s) ### cdp-snapshot.py Capture structured page content in generic formats by default, plus an optional forum/search-result helper format. ```bash # Simplified markdown (default) python3 {baseDir}/scripts/cdp-snapshot.py --port PORT # Plain text extraction python3 {baseDir}/scripts/cdp-snapshot.py --port PORT --format text # Extract all links as title+href pairs python3 {baseDir}/scripts/cdp-snapshot.py --port PORT --format links # Extract topic/result links from the main content area (useful on forums/search pages) python3 {baseDir}/scripts/cdp-snapshot.py --port PORT --format topic-links # With character limit python3 {baseDir}/scripts/cdp-snapshot.py --port PORT --max-chars 4000 ``` Output: JSON to stdout `{"title": "...", "url": "...", "content": "..."}` Parameters: - `--port PORT` (required): CDP HTTP/WebSocket port - `--target-id ID`: specific target from `/json/list` - `--format {markdown,text,links,topic-links}`: output format (default: `markdown`) - `--max-chars N`: truncate content to N chars (default: 8000, 0=unlimited) For `topic-links`, `meta` is best-effort surrounding text truncated to 400 characters. See also: - `references/forum-enhancements.md` - `references/use-cases.md` - `references/session-manifest.md` - `references/assisted-session-flow.md` - `references/testing-matrix.md` - `references/manual-fallback.md` - `references/validation-findings.md` FILE:agents/openai.yaml interface: display_name: "Ubuntu Browser Session" short_description: "Protected-site browser workflow for Ubuntu Server" default_prompt: "Use $ubuntu-browser-session for protected-site browser tasks, session reuse, or recovery on Ubuntu Server." FILE:references/assisted-session-flow.md # Assisted Session Flow The assisted flow is a bounded manual takeover on top of the same durable browser profile the agent will later reuse. It is not meant to be the primary day-to-day interaction model. It is the exception path for: - first login - expired sessions - challenges that cannot be cleared automatically ## Current Flow 1. Wrapper resolves the site profile and starts or reuses the GUI runtime 2. Wrapper checks whether the current page is: - the requested target page - a wrong-page drift that can be auto-corrected - a real login wall or challenge 3. If local recovery is exhausted, `assisted-session.sh start` exposes the same live browser through noVNC 4. User completes the blocked step and leaves the final target page loaded 5. `assisted-session.sh capture` writes: - the exact manifest - the site session registry entry - compatibility identity aliases for Google-family hosts ## Access URLs `assisted-session.sh status` now reports both: - `novnc_url`: loopback URL for SSH tunnel use - `lan_novnc_url`: direct LAN URL when the host IP is known Examples: ```text http://127.0.0.1:6084/vnc.html?autoconnect=1&resize=remote http://192.168.0.200:6084/vnc.html?autoconnect=1&resize=remote ``` Use the loopback URL if you are forwarding ports over SSH from Windows or another remote machine. Use the LAN URL only when firewall and network policy allow direct access. ## Commands The wrapper is the supported entrypoint. Direct `assisted-session.sh` commands are for the bounded handoff after `open-protected-page.sh` has already selected the site profile and exposed noVNC. ```bash scripts/assisted-session.sh start --url 'https://target.example' --origin 'https://target.example' --session-key default scripts/assisted-session.sh status --origin 'https://target.example' --session-key default scripts/assisted-session.sh capture --origin 'https://target.example' --session-key default scripts/assisted-session.sh stop --origin 'https://target.example' --session-key default ``` ## Important Rule The user should not be asked to intervene merely because the browser is on the wrong page. The wrapper should first try to return the same logged-in profile to the requested target page automatically. Manual takeover should happen only when the target still lands on: - a login wall - a challenge page - another unrecoverable blocked state FILE:references/forum-enhancements.md # Forum/Search Result Enhancements These helpers are optional add-ons for pages that behave like forums, topic lists, or search result pages. Use them when the host browser is already on a page that lists many clickable topics or results and you want to: - extract the main topic/result links instead of every navigation link - click a result by visible text - work with forum pages without falling back to OCR or manual tab scraping They are generic heuristics for pages that expose topic/result links in repeated containers. ## Commands ### Extract topic/result links ```bash python3 {baseDir}/scripts/cdp-snapshot.py --port PORT --format topic-links ``` What it does: - prefers repeated content containers commonly seen on forums/search pages - prefers anchors that look like primary content links inside those containers - falls back to repeated containers under `main`/`article`/`[role=main]`, then to anchors in the main content area if no preferred containers are found Output is JSON with `title`, `url`, and `content`, where `content` is a JSON-encoded array like: ```json [ { "text": "Example topic", "href": "https://example.com/t/topic/123", "meta": "optional snippet from the surrounding container" } ] ``` `meta` is optional and truncated to 400 characters. ### Click a link by visible text ```bash python3 {baseDir}/scripts/cdp-eval.py --port PORT --click-link-text 'Example topic' ``` What it does: - finds visible anchors whose text exactly matches or contains the requested phrase - prefers exact matches over partial matches - scrolls the chosen anchor into view before clicking - returns candidate links when nothing matches - when combined with `--navigate`, pair it with `--wait-navigation` or `--wait-for` so the destination DOM has rendered before clicking ## Notes - `topic-links` is a best-effort heuristic, not a guaranteed universal schema. - Works best on Discourse-style forums, topic lists, category pages, and search result pages, but also falls back to generic main-content result lists. - On non-forum pages, prefer the generic `links` format or plain `--eval` JavaScript. - If a site renders results inside shadow DOM, canvas, or highly dynamic widgets, fall back to direct page-specific evaluation. FILE:references/manual-fallback.md # Manual Fallback Use this only when the wrapper and assisted flow are insufficient or when debugging the host browser stack itself. ## Manual Start Sequence ```bash mkdir -p "$HOME/.remote-browser-profile" "$HOME/.remote-browser-logs" Xvfb :77 -screen 0 1600x900x24 -ac +extension RANDR env DISPLAY=:77 x11vnc -display :77 -forever -shared -rfbport 5900 -localhost -nopw websockify --web=/usr/share/novnc 0.0.0.0:6080 localhost:5900 env DISPLAY=:77 google-chrome \ --no-first-run \ --no-default-browser-check \ --user-data-dir="$HOME/.remote-browser-profile" \ --remote-debugging-address=127.0.0.1 \ --remote-debugging-port=9222 \ --new-window 'https://example.com' ``` If `google-chrome` is unavailable, use the detected Chromium binary. ## Health Checks ```bash lsof -iTCP -sTCP:LISTEN -P -n | rg '5900|6080|9222' curl -I -s http://127.0.0.1:6080/vnc.html | sed -n '1,5p' curl -s http://127.0.0.1:9222/json/version curl -s http://127.0.0.1:9222/json/list ``` ## Remote Access Patterns Direct LAN access when allowed: ```text http://HOST_IP:6080/vnc.html?autoconnect=1&resize=remote ``` SSH tunnel when direct access is blocked: ```powershell ssh -L 6080:127.0.0.1:6080 -L 9222:127.0.0.1:9222 USER@HOST_IP ``` Then open: ```text http://127.0.0.1:6080/vnc.html?autoconnect=1&resize=remote http://127.0.0.1:9222/json ``` ## Common Problems ### `ERR_CONNECTION_TIMED_OUT` If the host itself can reach `HOST_IP:6080` but a Windows client on the LAN times out, the most likely cause is host firewall or network policy. Allow the port explicitly or use SSH port forwarding. ### Browser Logged In But On The Wrong Page Before asking the user to log in again, verify whether the profile is still valid and the browser only drifted to another page. The preferred fix is to navigate back to the requested target page in the same profile. ### Existing Listeners Remain After Shutdown Inspect active listeners and terminate leftover processes before restarting on the same ports. FILE:references/session-manifest.md # Session Storage The skill now uses two durable layers: 1. per-origin manifests for live runtime verification 2. a site session registry for default per-site identity reuse ## 1. Session Manifest `session-manifest.sh` stores exact runtime records under `~/.agent-browser/` by default. Current scoped layout: - runtime: `run///` - assisted overlay: `assist///` - profile: `profiles///` - logs: `logs///` - manifests: `sessions//.json` Typical manifest commands: ```bash scripts/session-manifest.sh list scripts/session-manifest.sh show --origin 'https://github.com' --session-key default scripts/session-manifest.sh write --origin 'https://github.com' --session-key default --state ready --browser-pid 123 scripts/session-manifest.sh mark-stale --origin 'https://github.com' --session-key default --reason 'browser exited' ``` Operational notes: - manifests track exact captured runtime details - they are still used for verification and compatibility fallback - they are not the primary product abstraction for default reuse anymore ## 2. Site Session Registry `site-session-registry.sh` stores the default durable identity by canonical site: - `github.com` - `google.com` - exact host for other sites unless an alias is intentionally added Storage file: - `index/site-sessions.json` Typical commands: ```bash scripts/site-session-registry.sh show --site github.com scripts/site-session-registry.sh resolve --site github.com --session-key default scripts/site-session-registry.sh write --site github.com --session-key default --profile-dir ~/.agent-browser/profiles/https___github_com/default --source-origin 'https://github.com' ``` Operational notes: - default reuse should prefer `site + session-key` - non-default identities are allowed, but only when the user explicitly chooses a different `session-key` - corrupt registry JSON is treated as empty and rewritten on the next successful capture ## Resolution Model Current preferred resolution order: 1. explicit CLI `--profile-dir` 2. site session registry 3. exact manifest `profile_dir` 4. compatibility fallback for legacy/scoped profile reuse 5. fresh derived path only as the last fallback FILE:references/testing-matrix.md # Testing Matrix ## Should Trigger - "Open GitHub settings on the Ubuntu host and reuse the existing login." - "Use the server browser and continue with the Google account." - "This protected site is asking me to sign in again. Recover it." - "The server browser is open somewhere else. Go back to the right page." - "Use the `work` session-key for this site." ## Should Not Trigger - "Search the web for the latest Ubuntu release notes." - "Fetch this public JSON API and summarize it." - "Open this site in my local desktop browser." - "Help me write a curl command for this endpoint." ## Critical Behaviors - Default site identity reuse: - GitHub default session opens GitHub without user help - Google default session opens Google without user help - Wrong-page recovery: - browser is logged in but currently on another site - wrapper should try to navigate back before asking for noVNC help - Explicit secondary identity: - non-default `session-key` must only activate when explicitly requested - agent must not guess between multiple identities automatically - Manual takeover: - output includes both loopback and LAN noVNC URLs when possible - capture updates site registry for later reuse FILE:references/use-cases.md # Use Cases ## 1. First Login On The Ubuntu Host User intent: establish a durable browser identity for an important site such as GitHub or Google on the Ubuntu Server host. Representative requests: - "Log into GitHub on the Ubuntu host and keep it for future tasks." - "Set up the Google account in the server browser once." Desired result: - one bounded round of user help through noVNC - successful capture of the final page - future tasks reuse the same site identity by default Preferred path: - `scripts/open-protected-page.sh --url ...` - user completes login in noVNC only if needed - `scripts/assisted-session.sh capture --origin ... --session-key default` Direct `assisted-session.sh start` usage is not the normal operator path. Use the wrapper first so the site registry selects the correct reusable profile before any manual takeover. ## 2. Reuse The Default Site Identity User intent: agent should use the already logged-in browser context for a site without asking again. Representative requests: - "Open GitHub settings on the server." - "Use the Google account session on the Ubuntu host." - "Continue from the same browser login as before." Desired result: - wrapper resolves the canonical site identity - browser opens the requested page in the same durable profile - no user prompt when the target page is still valid ## 3. Recover A Drifted But Still Logged-In Browser User intent: browser is open in the right profile but currently sitting on the wrong page. Representative requests: - "Go back to GitHub settings." - "Switch back to my Google account page." - "The browser is open somewhere else. Continue the task." Desired result: - wrapper first navigates the existing logged-in profile back to the requested site - user is not interrupted just because the browser drifted to another page ## 4. Recover An Expired Session User intent: site login really expired or the page returned to a challenge. Representative requests: - "This site is asking me to sign in again." - "Recover the old session with the least user help." - "Try the existing browser first, then tell me exactly what I need to do." Desired result: - local recovery paths happen first - user is asked to take over only when the target still lands on a login wall or challenge - successful capture updates the default site identity again ## 5. Use A Non-Default Identity Explicitly User intent: user wants a secondary browser identity for a site, but only by explicit request. Representative requests: - "Use the `work` GitHub identity." - "Open Google with `session-key work`." Desired result: - agent uses the explicitly named `session-key` - agent never guesses between accounts automatically FILE:references/validation-findings.md # Validation Findings Current fast validation: - `test_runtime_common.sh`: passes - `test_session_manifest.sh`: passes - `test_browser_runtime.sh`: passes - `test_open_protected_page.sh`: passes - `test_assisted_session.sh`: passes - `test_profile_resolution.sh`: passes - `test_identity_provider_reuse.sh`: passes - `test_site_session_registry.sh`: passes Important behavioral coverage now includes: - canonical site-key resolution - corruption-safe site registry handling - site-first profile resolution - capture writing both manifest and site registry - loopback plus LAN noVNC URL reporting - wrong-page recovery before escalating to user takeover Real-host validation on 2026-03-15: - GitHub default site identity captured successfully - Google default site identity captured successfully - `~/.agent-browser/index/site-sessions.json` rebuilt with: - `github.com` - `google.com` - OpenClaw `main` agent successfully reused both: - `https://github.com/settings/profile` - `https://myaccount.google.com/` Operational note: - direct LAN noVNC access may still require opening the corresponding host firewall port - when LAN access is blocked, the loopback noVNC URL is still valid through SSH port forwarding FILE:scripts/assisted-session.sh #!/usr/bin/env bash set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck disable=SC1091 source "$SCRIPT_DIR/runtime-common.sh" usage() { cat <<'EOF' Usage: assisted-session.sh start --url URL [options] assisted-session.sh status [options] assisted-session.sh capture --origin URL [options] assisted-session.sh stop [options] Options: --block-reason REASON --manifest-root DIR --novnc-port PORT --origin URL --profile-dir DIR --run-dir DIR --session-key KEY --url URL --vnc-port PORT EOF } log() { printf '[assisted-session] %s\n' "$*" } die() { printf '[assisted-session] ERROR: %s\n' "$*" >&2 exit 1 } require_arg() { local name="$1" local value="$2" [ -n "$value" ] || die "missing required argument: $name" } have_cmd() { command -v "$1" >/dev/null 2>&1 } runtime_helper() { "${AGENT_BROWSER_RUNTIME_HELPER:-$SCRIPT_DIR/browser-runtime.sh}" "$@" } select_target_helper() { "${AGENT_BROWSER_SELECT_TARGET_HELPER:-$SCRIPT_DIR/browser-runtime.sh}" "$@" } manifest_helper() { "${AGENT_BROWSER_MANIFEST_HELPER:-$SCRIPT_DIR/session-manifest.sh}" "$@" } profile_helper() { "${AGENT_BROWSER_PROFILE_HELPER:-$SCRIPT_DIR/profile-resolution.sh}" "$@" } manifest_field() { local field="$1" local payload="$2" python3 - "$field" "$payload" <<'PY' import json import sys field = sys.argv[1] payload = json.loads(sys.argv[2]) value = payload.get(field) if value is None: raise SystemExit(1) if isinstance(value, (dict, list)): print(json.dumps(value)) else: print(value) PY } site_registry_helper() { "${AGENT_BROWSER_SITE_REGISTRY_HELPER:-$SCRIPT_DIR/site-session-registry.sh}" "$@" } pid_file() { printf '%s/%s.pid\n' "$RUN_DIR" "$1" } read_pid() { local file file="$(pid_file "$1")" if [ -f "$file" ]; then cat "$file" fi } pid_running() { local pid="${1:-}" [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null } write_state() { mkdir -p "$RUN_DIR" cat >"$STATE_FILE" <"$logfile" setsid "$@" >>"$logfile" 2>&1 & local pid=$! printf '%s\n' "$pid" >"$(pid_file "$name")" sleep 1 pid_running "$pid" || die "$name failed to start; inspect $logfile" } stop_process() { local name="$1" local pid pid="$(read_pid "$name")" if ! pid_running "$pid"; then rm -f "$(pid_file "$name")" return 0 fi kill -TERM -- "-$pid" 2>/dev/null || kill -TERM "$pid" 2>/dev/null || true local _i for _i in 1 2 3 4 5; do if ! pid_running "$pid"; then rm -f "$(pid_file "$name")" return 0 fi sleep 1 done kill -KILL -- "-$pid" 2>/dev/null || kill -KILL "$pid" 2>/dev/null || true rm -f "$(pid_file "$name")" } runtime_status() { runtime_helper status \ --run-dir "$RUNTIME_RUN_DIR" \ --origin "$ORIGIN" \ --session-key "$SESSION_KEY" \ ${INITIAL_URL:+--url "$INITIAL_URL"} } runtime_value() { local key="$1" local payload="$2" python3 - "$key" "$payload" <<'PY' import sys target = sys.argv[1] for raw in sys.argv[2].splitlines(): if ":" not in raw: continue key, value = raw.split(":", 1) if key.strip() == target: print(value.strip()) break PY } identity_providers() { local page_json="${1:-}" local target_url="$ORIGIN" if [ -n "$page_json" ]; then target_url="$( python3 - "$page_json" "$ORIGIN" <<'PY' import json import sys payload, origin = sys.argv[1:] try: parsed = json.loads(payload) if payload else {} except json.JSONDecodeError: parsed = {} print(parsed.get("url") or origin) PY )" fi provider_aliases "$target_url" } write_identity_metadata() { local page_json="$1" local provider while IFS= read -r provider; do [ -n "$provider" ] || continue profile_helper write-identity \ --root "$BASE_ROOT" \ --provider "$provider" \ --profile-dir "$PROFILE_DIR" \ --source-origin "$ORIGIN" \ --source-session-key "$SESSION_KEY" >/dev/null done < <(identity_providers "$page_json") } write_site_session() { local site site="$(site_key "$ORIGIN")" site_registry_helper write \ --root "$BASE_ROOT" \ --site "$site" \ --session-key "$SESSION_KEY" \ --profile-dir "$PROFILE_DIR" \ --source-origin "$ORIGIN" >/dev/null } page_matches_target() { local page_json="$1" python3 - "$page_json" "${INITIAL_URL:-}" "$ORIGIN" <<'PY' import json import sys from urllib.parse import urlparse, urlunparse page_payload, initial_url, origin = sys.argv[1:] try: page_url = (json.loads(page_payload or "{}").get("url") or "").strip() except json.JSONDecodeError: page_url = "" def normalize(value: str) -> str: raw = (value or "").strip() if not raw: return "" parsed = urlparse(raw) if not parsed.scheme or not parsed.netloc: return raw.rstrip("/") path = parsed.path or "" if path not in ("", "/"): path = path.rstrip("/") else: path = "" return urlunparse(( parsed.scheme.lower(), parsed.netloc.lower(), path, parsed.params, parsed.query, "", )) page = normalize(page_url) initial = normalize(initial_url) origin_value = normalize(origin) if not page: raise SystemExit(1) if initial and initial != origin_value: raise SystemExit(0 if page == initial else 1) if origin_value and (page == origin_value or page.startswith(origin_value + "/")): raise SystemExit(0) raise SystemExit(1) PY } resolve_profile_dir() { local resolved_profile if resolved_profile="$( profile_helper resolve \ --root "$BASE_ROOT" \ --manifest-root "$MANIFEST_ROOT" \ --origin "$ORIGIN" \ --session-key "$SESSION_KEY" )"; then PROFILE_DIR="$(manifest_field profile_dir "$resolved_profile")" return 0 fi local resolve_status=$? if [ "$resolve_status" -eq 2 ]; then die "ambiguous reusable profiles for $ORIGIN" fi return "$resolve_status" } resolve_context() { BASE_ROOT="${HOME}/.agent-browser" MANIFEST_ROOT="${MANIFEST_ROOT:-$BASE_ROOT}" SESSION_KEY="${SESSION_KEY:-default}" if [ -z "$ORIGIN" ] && [ -n "$INITIAL_URL" ]; then ORIGIN="$(derive_origin "$INITIAL_URL")" fi if [ -z "$ORIGIN" ]; then ORIGIN="https://example.com" fi if [ -z "$INITIAL_URL" ]; then INITIAL_URL="$ORIGIN" fi if [ -z "$RUN_DIR" ]; then RUN_DIR="$(runtime_scoped_path "$BASE_ROOT" assist "$ORIGIN" "$SESSION_KEY")" fi STATE_FILE="$RUN_DIR/assist.env" load_state RUN_DIR="${CLI_RUN_DIR:-${RUN_DIR:-}}" MANIFEST_ROOT="${CLI_MANIFEST_ROOT:-${MANIFEST_ROOT:-$BASE_ROOT}}" INITIAL_URL="${CLI_INITIAL_URL:-${INITIAL_URL:-$ORIGIN}}" ORIGIN="${CLI_ORIGIN:-${ORIGIN:-$(derive_origin "$INITIAL_URL")}}" PROFILE_DIR="${CLI_PROFILE_DIR:-${PROFILE_DIR:-}}" NOVNC_PORT="${CLI_NOVNC_PORT:-${NOVNC_PORT:-6080}}" VNC_PORT="${CLI_VNC_PORT:-${VNC_PORT:-5900}}" SESSION_KEY="${CLI_SESSION_KEY:-${SESSION_KEY:-default}}" LOG_DIR="${LOG_DIR:-}" RUNTIME_RUN_DIR="${RUNTIME_RUN_DIR:-}" if [ -z "$RUN_DIR" ]; then RUN_DIR="$(runtime_scoped_path "$BASE_ROOT" assist "$ORIGIN" "$SESSION_KEY")" fi if [ -n "$INITIAL_URL" ] && [ "$(derive_origin "$INITIAL_URL")" != "$ORIGIN" ]; then INITIAL_URL="$ORIGIN" fi if [ -z "$RUNTIME_RUN_DIR" ]; then RUNTIME_RUN_DIR="$(runtime_scoped_path "$BASE_ROOT" run "$ORIGIN" "$SESSION_KEY")" fi if [ -z "$PROFILE_DIR" ]; then resolve_profile_dir fi if [ -z "$LOG_DIR" ]; then LOG_DIR="$(runtime_scoped_path "$BASE_ROOT" logs "$ORIGIN" "$SESSION_KEY")" fi STATE_FILE="$RUN_DIR/assist.env" } status_assisted() { local runtime local lan_host runtime="$(runtime_status)" printf 'assisted_session: %s\n' "$(pid_running "$(read_pid x11vnc)" && printf 'running' || printf 'stopped')" printf 'run_dir: %s\n' "$RUN_DIR" printf 'runtime_run_dir: %s\n' "$RUNTIME_RUN_DIR" printf 'novnc_url: http://127.0.0.1:%s/vnc.html?autoconnect=1&resize=remote\n' "$NOVNC_PORT" lan_host="$(primary_ipv4 || true)" if [ -n "$lan_host" ]; then printf 'lan_novnc_url: %s\n' "$(lan_novnc_url "$lan_host" "$NOVNC_PORT")" fi printf '%s\n' "$runtime" } ensure_runtime_for_assist() { local runtime runtime="$(runtime_status)" if ! printf '%s\n' "$runtime" | grep -q '^runtime: running$'; then runtime_helper start \ --run-dir "$RUNTIME_RUN_DIR" \ --url "$INITIAL_URL" \ --origin "$ORIGIN" \ --session-key "$SESSION_KEY" \ --mode gui \ ${PROFILE_DIR:+--profile-dir "$PROFILE_DIR"} >/dev/null runtime="$(runtime_status)" fi printf '%s\n' "$runtime" } ensure_overlay_deps() { local missing=() local dep local novnc_root="${AGENT_BROWSER_NOVNC_WEB_ROOT:-/usr/share/novnc}" for dep in x11vnc websockify; do if ! have_cmd "$dep"; then missing+=("$dep") fi done if [ "${#missing[@]}" -gt 0 ]; then die "missing dependencies: ${missing[*]}" fi if [ ! -d "$novnc_root" ]; then die "$novnc_root not found; install novnc or adjust the script for your distribution" fi } select_runtime_target() { local targets_json="$1" local target_url="" if [ -n "$INITIAL_URL" ] && [ "$(derive_origin "$INITIAL_URL")" = "$ORIGIN" ]; then target_url="$INITIAL_URL" fi select_target_helper select-target \ --origin "$ORIGIN" \ ${target_url:+--target-url "$target_url"} \ --targets-json "$targets_json" } start_assisted() { require_arg --url "$INITIAL_URL" local runtime display runtime_mode novnc_root target_id challenge login_wall targets_json runtime="$(ensure_runtime_for_assist)" runtime_mode="$(runtime_value mode "$runtime")" display="$(runtime_value display "$runtime")" PROFILE_DIR="$(runtime_value profile_dir "$runtime")" CDP_PORT="$(runtime_value cdp_port "$runtime")" [ "$runtime_mode" = "gui" ] || die "assisted flow requires runtime GUI mode" [ -n "$display" ] || die "runtime did not expose a display" ensure_overlay_deps novnc_root="${AGENT_BROWSER_NOVNC_WEB_ROOT:-/usr/share/novnc}" if ! pid_running "$(read_pid x11vnc)" && [ -z "${CLI_VNC_PORT:-}" ]; then VNC_PORT="$(pick_free_tcp_port "$VNC_PORT")" fi if ! pid_running "$(read_pid websockify)" && [ -z "${CLI_NOVNC_PORT:-}" ]; then NOVNC_PORT="$(pick_free_tcp_port "$NOVNC_PORT")" fi write_state if ! pid_running "$(read_pid x11vnc)"; then start_process x11vnc "$LOG_DIR/x11vnc.log" \ env DISPLAY="$display" \ x11vnc -display "$display" -forever -shared -rfbport "$VNC_PORT" -localhost -nopw fi if ! pid_running "$(read_pid websockify)"; then start_process websockify "$LOG_DIR/websockify.log" \ websockify --web="$novnc_root" "0.0.0.0:$NOVNC_PORT" "localhost:$VNC_PORT" fi targets_json="$(runtime_helper list-targets --run-dir "$RUNTIME_RUN_DIR" --origin "$ORIGIN" --session-key "$SESSION_KEY")" target_id="$(select_runtime_target "$targets_json")" challenge="" login_wall="" if [ -n "$target_id" ]; then challenge="$(runtime_helper check-page --run-dir "$RUNTIME_RUN_DIR" --origin "$ORIGIN" --session-key "$SESSION_KEY" --cdp-port "$CDP_PORT" --target-id "$target_id" --check challenge 2>/dev/null || true)" login_wall="$(runtime_helper check-page --run-dir "$RUNTIME_RUN_DIR" --origin "$ORIGIN" --session-key "$SESSION_KEY" --cdp-port "$CDP_PORT" --target-id "$target_id" --check login-wall 2>/dev/null || true)" fi log "noVNC URL: http://127.0.0.1:$NOVNC_PORT/vnc.html?autoconnect=1&resize=remote" if printf '%s' "$challenge" | grep -q '"hasChallenge": true'; then log "next action: open the noVNC URL, wait for the challenge to clear or complete it, and leave the protected page loaded" elif printf '%s' "$login_wall" | grep -q '"hasLoginWall": true'; then log "next action: open the noVNC URL, sign in, complete any MFA, and leave the final page loaded" else log "next action: open the noVNC URL only if local recovery did not already clear the block" fi } capture_session() { require_arg --origin "$ORIGIN" local runtime target_json target_id challenge_json login_json page_json browser_pid display cdp_port runtime="$(runtime_status)" if ! printf '%s\n' "$runtime" | grep -q '^runtime: running$'; then die "no verified browser runtime is available" fi browser_pid="$(runtime_value browser_pid "$runtime")" display="$(runtime_value display "$runtime")" cdp_port="$(runtime_value cdp_port "$runtime")" PROFILE_DIR="$(runtime_value profile_dir "$runtime")" require_arg browser_pid "$browser_pid" require_arg cdp_port "$cdp_port" target_json="$(runtime_helper list-targets --run-dir "$RUNTIME_RUN_DIR" --origin "$ORIGIN" --session-key "$SESSION_KEY")" target_id="$(select_runtime_target "$target_json")" [ -n "$target_id" ] || die "no page target is available for capture" challenge_json="$(runtime_helper check-page --run-dir "$RUNTIME_RUN_DIR" --origin "$ORIGIN" --session-key "$SESSION_KEY" --cdp-port "$cdp_port" --target-id "$target_id" --check challenge)" login_json="$(runtime_helper check-page --run-dir "$RUNTIME_RUN_DIR" --origin "$ORIGIN" --session-key "$SESSION_KEY" --cdp-port "$cdp_port" --target-id "$target_id" --check login-wall)" page_json="$(runtime_helper check-page --run-dir "$RUNTIME_RUN_DIR" --origin "$ORIGIN" --session-key "$SESSION_KEY" --cdp-port "$cdp_port" --target-id "$target_id" --check page-info)" if printf '%s' "$challenge_json" | grep -q '"hasChallenge": true'; then die "verification has not succeeded yet; challenge page still active" fi if printf '%s' "$login_json" | grep -q '"hasLoginWall": true'; then die "verification has not succeeded yet; login wall still active" fi if ! page_matches_target "$page_json"; then die "verification has not succeeded yet; browser is not on the requested target page" fi write_state manifest_helper write \ --root "$MANIFEST_ROOT" \ --origin "$ORIGIN" \ --session-key "$SESSION_KEY" \ --state ready \ --browser-pid "$browser_pid" \ --cdp-port "$cdp_port" \ --target-id "$target_id" \ --profile-dir "$PROFILE_DIR" \ --mode assisted-gui \ --display "$display" \ ${BLOCK_REASON:+--block-reason "$BLOCK_REASON"} >/dev/null write_site_session write_identity_metadata "$page_json" printf '%s\n' "$page_json" } stop_assisted() { stop_process websockify stop_process x11vnc log "assisted overlay stopped" } COMMAND="${1:-}" [ -n "$COMMAND" ] || { usage exit 1 } shift || true RUN_DIR="" RUNTIME_RUN_DIR="" MANIFEST_ROOT="" INITIAL_URL="" PROFILE_DIR="" NOVNC_PORT="" VNC_PORT="" ORIGIN="" SESSION_KEY="default" BLOCK_REASON="" CDP_PORT="" LOG_DIR="" while [ "$#" -gt 0 ]; do case "$1" in --run-dir) RUN_DIR="$2" shift 2 ;; --manifest-root) MANIFEST_ROOT="$2" shift 2 ;; --url) INITIAL_URL="$2" shift 2 ;; --profile-dir) PROFILE_DIR="$2" shift 2 ;; --novnc-port) NOVNC_PORT="$2" shift 2 ;; --vnc-port) VNC_PORT="$2" shift 2 ;; --origin) ORIGIN="$2" shift 2 ;; --session-key) SESSION_KEY="$2" shift 2 ;; --block-reason) BLOCK_REASON="$2" shift 2 ;; -h|--help) usage exit 0 ;; *) die "unknown argument: $1" ;; esac done CLI_RUN_DIR="$RUN_DIR" CLI_MANIFEST_ROOT="$MANIFEST_ROOT" CLI_INITIAL_URL="$INITIAL_URL" CLI_PROFILE_DIR="$PROFILE_DIR" CLI_NOVNC_PORT="$NOVNC_PORT" CLI_VNC_PORT="$VNC_PORT" CLI_ORIGIN="$ORIGIN" CLI_SESSION_KEY="$SESSION_KEY" resolve_context case "$COMMAND" in start) start_assisted ;; status) status_assisted ;; capture) capture_session ;; stop) stop_assisted ;; *) usage exit 1 ;; esac FILE:scripts/browser-runtime.sh #!/usr/bin/env bash set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck disable=SC1091 source "$SCRIPT_DIR/runtime-common.sh" usage() { cat <<'EOF' Usage: browser-runtime.sh start --url URL [--mode headless|gui] [options] browser-runtime.sh status [options] browser-runtime.sh list-targets [options] browser-runtime.sh check-page --check TYPE [options] browser-runtime.sh select-target [--origin URL] [--target-url URL] [--targets-json JSON] browser-runtime.sh attach --origin URL --session-key KEY [--manifest-root DIR] browser-runtime.sh verify --origin URL --session-key KEY [--manifest-root DIR] browser-runtime.sh stop [options] Options: --browser CMD --cdp-port PORT --display NUM --manifest-root DIR --mode MODE --origin URL --profile-dir DIR --run-dir DIR --session-key KEY --target-id ID --target-url URL --targets-json JSON --url URL EOF } log() { printf '[browser-runtime] %s\n' "$*" } die() { printf '[browser-runtime] ERROR: %s\n' "$*" >&2 exit 1 } have_cmd() { command -v "$1" >/dev/null 2>&1 } require_arg() { local name="$1" local value="$2" [ -n "$value" ] || die "missing required argument: $name" } detect_browser() { if [ -n "${BROWSER_CMD:-}" ]; then printf '%s\n' "$BROWSER_CMD" return 0 fi local candidate for candidate in google-chrome chromium chromium-browser; do if have_cmd "$candidate"; then printf '%s\n' "$candidate" return 0 fi done return 1 } pid_running() { local pid="${1:-}" [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null } cleanup_profile_locks() { local profile_dir="$1" [ -n "$profile_dir" ] || return 0 local lock="$profile_dir/SingletonLock" if [ -L "$lock" ]; then local target pid target="$(readlink "$lock" 2>/dev/null || true)" pid="${target##*-}" if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then return 0 fi fi rm -f \ "$lock" \ "$profile_dir/SingletonSocket" \ "$profile_dir/SingletonCookie" } pid_file() { printf '%s/%s.pid\n' "$RUN_DIR" "$1" } read_pid() { local file file="$(pid_file "$1")" if [ -f "$file" ]; then cat "$file" fi } write_state() { mkdir -p "$RUN_DIR" cat >"$STATE_FILE" <"$logfile" setsid "$@" >>"$logfile" 2>&1 & local pid=$! printf '%s\n' "$pid" >"$(pid_file "$name")" sleep 1 pid_running "$pid" || die "$name failed to start; inspect $logfile" } stop_process() { local name="$1" local pid pid="$(read_pid "$name")" if ! pid_running "$pid"; then rm -f "$(pid_file "$name")" return 0 fi kill -TERM -- "-$pid" 2>/dev/null || kill -TERM "$pid" 2>/dev/null || true local _i for _i in 1 2 3 4 5; do if ! pid_running "$pid"; then rm -f "$(pid_file "$name")" return 0 fi sleep 1 done kill -KILL -- "-$pid" 2>/dev/null || kill -KILL "$pid" 2>/dev/null || true rm -f "$(pid_file "$name")" } wait_for_display() { local socket_dir="${AGENT_BROWSER_X11_SOCKET_DIR:-/tmp/.X11-unix}" local socket="${socket_dir}/X${DISPLAY_NUM}" local _i for _i in 1 2 3 4 5 6 7 8 9 10; do if [ -e "$socket" ]; then if ! have_cmd xdpyinfo || DISPLAY=":$DISPLAY_NUM" xdpyinfo >/dev/null 2>&1; then return 0 fi fi sleep 1 done die "Xvfb did not become ready on :$DISPLAY_NUM" } wait_for_cdp() { local _i if ! have_cmd curl; then return 0 fi for _i in 1 2 3 4 5 6 7 8 9 10; do if curl -s --max-time 2 "http://127.0.0.1:${CDP_PORT}/json/version" >/dev/null 2>&1; then return 0 fi sleep 1 done die "CDP did not become ready on port $CDP_PORT" } runtime_browser_pid() { read_pid browser } runtime_xvfb_pid() { read_pid xvfb } runtime_running() { pid_running "$(runtime_browser_pid)" } manifest_helper() { "${AGENT_BROWSER_MANIFEST_HELPER:-$SCRIPT_DIR/session-manifest.sh}" "$@" } cdp_eval() { "${AGENT_BROWSER_CDP_EVAL:-$SCRIPT_DIR/cdp-eval.py}" "$@" } profile_helper() { "${AGENT_BROWSER_PROFILE_HELPER:-$SCRIPT_DIR/profile-resolution.sh}" "$@" } manifest_field() { local field="$1" local payload="$2" python3 - "$field" "$payload" <<'PY' import json import sys field = sys.argv[1] payload = json.loads(sys.argv[2]) value = payload.get(field) if value is None: raise SystemExit(1) if isinstance(value, (dict, list)): print(json.dumps(value)) else: print(value) PY } resolve_context() { BASE_ROOT="${HOME}/.agent-browser" MANIFEST_ROOT="${MANIFEST_ROOT:-$BASE_ROOT}" SESSION_KEY="${SESSION_KEY:-default}" if [ -z "$ORIGIN" ] && [ -n "$INITIAL_URL" ]; then ORIGIN="$(derive_origin "$INITIAL_URL")" fi if [ -z "$ORIGIN" ]; then ORIGIN="https://example.com" fi if [ -z "$INITIAL_URL" ]; then INITIAL_URL="$ORIGIN" fi if [ -z "$RUN_DIR" ]; then RUN_DIR="$(runtime_scoped_path "$BASE_ROOT" run "$ORIGIN" "$SESSION_KEY")" fi STATE_FILE="$RUN_DIR/runtime.env" load_state RUN_DIR="${CLI_RUN_DIR:-${RUN_DIR:-}}" MANIFEST_ROOT="${CLI_MANIFEST_ROOT:-${MANIFEST_ROOT:-$BASE_ROOT}}" MODE="${CLI_MODE:-${MODE:-headless}}" INITIAL_URL="${CLI_INITIAL_URL:-${INITIAL_URL:-$ORIGIN}}" ORIGIN="${CLI_ORIGIN:-${ORIGIN:-$(derive_origin "$INITIAL_URL")}}" SESSION_KEY="${CLI_SESSION_KEY:-${SESSION_KEY:-default}}" PROFILE_DIR="${CLI_PROFILE_DIR:-${PROFILE_DIR:-}}" LOG_DIR="${LOG_DIR:-}" CDP_PORT="${CLI_CDP_PORT:-${CDP_PORT:-19222}}" DISPLAY_NUM="${CLI_DISPLAY_NUM:-${DISPLAY_NUM:-88}}" BROWSER_CMD="${CLI_BROWSER_CMD:-${BROWSER_CMD:-}}" if [ -z "$RUN_DIR" ]; then RUN_DIR="$(runtime_scoped_path "$BASE_ROOT" run "$ORIGIN" "$SESSION_KEY")" fi if [ -z "$PROFILE_DIR" ]; then local resolved_profile if resolved_profile="$( profile_helper resolve \ --root "$BASE_ROOT" \ --manifest-root "$MANIFEST_ROOT" \ --origin "$ORIGIN" \ --session-key "$SESSION_KEY" )"; then PROFILE_DIR="$(manifest_field profile_dir "$resolved_profile")" else local resolve_status=$? if [ "$resolve_status" -eq 2 ]; then die "ambiguous reusable profiles for $ORIGIN" fi exit "$resolve_status" fi fi if [ -z "$LOG_DIR" ]; then LOG_DIR="$(runtime_scoped_path "$BASE_ROOT" logs "$ORIGIN" "$SESSION_KEY")" fi STATE_FILE="$RUN_DIR/runtime.env" } require_browser_deps() { if ! BROWSER_CMD="$(detect_browser)"; then die "missing browser dependency: google-chrome/chromium" fi if [ "$MODE" = "gui" ] && ! have_cmd Xvfb; then die "missing dependency: Xvfb" fi } select_target_from_json() { local payload="$1" python3 - "$ORIGIN" "$TARGET_URL" "$payload" <<'PY' import json import sys from urllib.parse import urlparse origin, target_url, payload = sys.argv[1:] targets = [target for target in json.loads(payload or "[]") if target.get("type") == "page"] def host(value: str) -> str: parsed = urlparse(value) return parsed.netloc def score(target): url = target.get("url", "") if target_url and url == target_url: return (0, url) if origin and url.startswith(origin): return (1, url) if origin and host(url) and host(url) == host(origin): return (2, url) return (9, url) if targets: print(sorted(targets, key=score)[0].get("id", "")) PY } start_runtime() { require_browser_deps if [ -z "${CLI_CDP_PORT:-}" ]; then CDP_PORT="$(pick_free_tcp_port "$CDP_PORT")" fi if [ "$MODE" = "gui" ] && [ -z "${CLI_DISPLAY_NUM:-}" ]; then DISPLAY_NUM="$(pick_free_display "$DISPLAY_NUM")" fi mkdir -p "$PROFILE_DIR" "$LOG_DIR" "$RUN_DIR" if runtime_running; then die "browser runtime already running; use stop or status first" fi cleanup_profile_locks "$PROFILE_DIR" write_state if [ "$MODE" = "gui" ]; then start_process xvfb "$LOG_DIR/xvfb.log" \ Xvfb ":$DISPLAY_NUM" -screen 0 1600x900x24 -ac +extension RANDR wait_for_display start_process browser "$LOG_DIR/browser.log" \ env DISPLAY=":$DISPLAY_NUM" \ "$BROWSER_CMD" \ --no-first-run \ --no-default-browser-check \ --user-data-dir="$PROFILE_DIR" \ --remote-debugging-address=127.0.0.1 \ --remote-debugging-port="$CDP_PORT" \ --new-window "$INITIAL_URL" else start_process browser "$LOG_DIR/browser.log" \ "$BROWSER_CMD" \ --headless=new \ --disable-gpu \ --no-first-run \ --no-default-browser-check \ --user-data-dir="$PROFILE_DIR" \ --remote-debugging-address=127.0.0.1 \ --remote-debugging-port="$CDP_PORT" \ "$INITIAL_URL" fi wait_for_cdp log "runtime started" log "mode: $MODE" log "profile dir: $PROFILE_DIR" log "DevTools URL on host: http://127.0.0.1:$CDP_PORT/json" } status_runtime() { printf 'runtime: %s\n' "$(runtime_running && printf 'running' || printf 'stopped')" printf 'mode: %s\n' "$MODE" printf 'origin: %s\n' "$ORIGIN" printf 'session_key: %s\n' "$SESSION_KEY" printf 'url: %s\n' "$INITIAL_URL" printf 'profile_dir: %s\n' "$PROFILE_DIR" printf 'run_dir: %s\n' "$RUN_DIR" printf 'cdp_port: %s\n' "$CDP_PORT" printf 'browser_pid: %s\n' "$(runtime_browser_pid || true)" if [ "$MODE" = "gui" ] || [ -n "$(runtime_xvfb_pid || true)" ]; then printf 'xvfb_pid: %s\n' "$(runtime_xvfb_pid || true)" printf 'display: :%s\n' "$DISPLAY_NUM" fi if runtime_running && have_cmd curl; then printf 'cdp_health: %s\n' "$(curl -s --max-time 3 "http://127.0.0.1:${CDP_PORT}/json/version" | head -c 160 || true)" fi } list_targets() { local port="${CDP_PORT}" if ! runtime_running && [ ! -f "$STATE_FILE" ]; then printf '[]\n' return 0 fi if ! have_cmd curl; then die "curl is required to list CDP targets" fi curl -s --max-time 3 "http://127.0.0.1:${port}/json/list" || printf '[]\n' } check_page() { require_arg --check "$CHECK_TYPE" require_arg --cdp-port "$CDP_PORT" cdp_eval --port "$CDP_PORT" ${TARGET_ID:+--target-id "$TARGET_ID"} --check "$CHECK_TYPE" } select_target() { if [ -z "$TARGETS_JSON" ]; then TARGETS_JSON="$(list_targets)" fi select_target_from_json "$TARGETS_JSON" } load_manifest() { manifest_helper show --root "$MANIFEST_ROOT" --origin "$ORIGIN" --session-key "$SESSION_KEY" } attach_session() { require_arg --origin "$ORIGIN" require_arg --session-key "$SESSION_KEY" local manifest browser_pid manifest="$(load_manifest)" || exit $? browser_pid="$(manifest_field browser_pid "$manifest" || true)" [ -n "$browser_pid" ] || die "manifest missing browser_pid" pid_running "$browser_pid" || die "browser is not running for manifest $SESSION_KEY" printf '%s\n' "$manifest" } verify_session() { require_arg --origin "$ORIGIN" require_arg --session-key "$SESSION_KEY" local manifest browser_pid cdp_port target_id targets manifest="$(load_manifest)" || exit $? browser_pid="$(manifest_field browser_pid "$manifest" || true)" cdp_port="$(manifest_field cdp_port "$manifest" || true)" target_id="$(manifest_field target_id "$manifest" || true)" if [ -z "$browser_pid" ] || ! pid_running "$browser_pid"; then manifest_helper mark-stale --root "$MANIFEST_ROOT" --origin "$ORIGIN" --session-key "$SESSION_KEY" --reason "browser process is not running" >/dev/null || true die "browser is not running for manifest $SESSION_KEY" fi if [ -n "$cdp_port" ] && have_cmd curl; then curl -s --max-time 3 "http://127.0.0.1:${cdp_port}/json/version" >/dev/null || die "CDP endpoint is unreachable for manifest $SESSION_KEY" if [ -n "$target_id" ]; then targets="$(curl -s --max-time 3 "http://127.0.0.1:${cdp_port}/json/list")" printf '%s' "$targets" | grep -q "\"id\":\"${target_id}\"" || die "target_id is no longer present for manifest $SESSION_KEY" fi fi printf '%s\n' "$manifest" } stop_runtime() { stop_process browser stop_process xvfb log "runtime stopped" } COMMAND="${1:-}" [ -n "$COMMAND" ] || { usage exit 1 } shift || true RUN_DIR="" MANIFEST_ROOT="" MODE="" INITIAL_URL="" PROFILE_DIR="" CDP_PORT="" DISPLAY_NUM="" BROWSER_CMD="" CHECK_TYPE="" TARGET_ID="" TARGET_URL="" TARGETS_JSON="" ORIGIN="" SESSION_KEY="" while [ "$#" -gt 0 ]; do case "$1" in --run-dir) RUN_DIR="$2" shift 2 ;; --manifest-root) MANIFEST_ROOT="$2" shift 2 ;; --mode) MODE="$2" shift 2 ;; --url) INITIAL_URL="$2" shift 2 ;; --profile-dir) PROFILE_DIR="$2" shift 2 ;; --cdp-port) CDP_PORT="$2" shift 2 ;; --display) DISPLAY_NUM="${2#:}" shift 2 ;; --browser) BROWSER_CMD="$2" shift 2 ;; --check) CHECK_TYPE="$2" shift 2 ;; --target-id) TARGET_ID="$2" shift 2 ;; --target-url) TARGET_URL="$2" shift 2 ;; --targets-json) TARGETS_JSON="$2" shift 2 ;; --origin) ORIGIN="$2" shift 2 ;; --session-key) SESSION_KEY="$2" shift 2 ;; -h|--help) usage exit 0 ;; *) die "unknown argument: $1" ;; esac done CLI_RUN_DIR="$RUN_DIR" CLI_MANIFEST_ROOT="$MANIFEST_ROOT" CLI_MODE="$MODE" CLI_INITIAL_URL="$INITIAL_URL" CLI_PROFILE_DIR="$PROFILE_DIR" CLI_CDP_PORT="$CDP_PORT" CLI_DISPLAY_NUM="$DISPLAY_NUM" CLI_BROWSER_CMD="$BROWSER_CMD" CLI_ORIGIN="$ORIGIN" CLI_SESSION_KEY="$SESSION_KEY" resolve_context case "$COMMAND" in start) require_arg --url "$INITIAL_URL" case "$MODE" in headless|gui) ;; *) die "--mode must be headless or gui" ;; esac start_runtime ;; status) status_runtime ;; list-targets) list_targets ;; check-page) check_page ;; select-target) select_target ;; attach) attach_session ;; verify) verify_session ;; stop) stop_runtime ;; *) usage exit 1 ;; esac FILE:scripts/cdp-eval.py #!/usr/bin/env python3 """Minimal CDP evaluation helper using only the Python standard library.""" from __future__ import annotations import argparse import base64 import hashlib import json import os import socket import struct import sys import time import urllib.error import urllib.parse import urllib.request TIMEOUT_SECONDS = 5 NAVIGATE_TIMEOUT_SECONDS = 15 WAIT_FOR_TIMEOUT_SECONDS = 10 USER_AGENT = "ubuntu-browser-session/cdp-eval" class CdpError(RuntimeError): pass def http_get_json(url: str) -> object: request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response: return json.loads(response.read().decode("utf-8")) def resolve_websocket_url(port: int, target_id: str | None) -> str: targets = http_get_json(f"http://127.0.0.1:{port}/json/list") if not isinstance(targets, list): raise CdpError("invalid target list response") page_targets = [target for target in targets if target.get("type") == "page"] if target_id: for target in page_targets: if target.get("id") == target_id: websocket_url = target.get("webSocketDebuggerUrl") if websocket_url: return websocket_url break raise CdpError("target-id not found") if page_targets: websocket_url = page_targets[0].get("webSocketDebuggerUrl") if websocket_url: return websocket_url version = http_get_json(f"http://127.0.0.1:{port}/json/version") websocket_url = version.get("webSocketDebuggerUrl") if isinstance(version, dict) else None if not websocket_url: raise CdpError("missing webSocketDebuggerUrl") return websocket_url def websocket_key() -> str: return base64.b64encode(os.urandom(16)).decode("ascii") def build_frame(payload: str) -> bytes: raw = payload.encode("utf-8") mask_key = os.urandom(4) length = len(raw) header = bytearray([0x81]) if length < 126: header.append(0x80 | length) elif length < (1 << 16): header.append(0x80 | 126) header.extend(struct.pack("!H", length)) else: header.append(0x80 | 127) header.extend(struct.pack("!Q", length)) masked = bytes(byte ^ mask_key[index % 4] for index, byte in enumerate(raw)) return bytes(header) + mask_key + masked def recv_exact(sock: socket.socket, size: int) -> bytes: chunks = [] remaining = size while remaining > 0: chunk = sock.recv(remaining) if not chunk: raise CdpError("unexpected websocket EOF") chunks.append(chunk) remaining -= len(chunk) return b"".join(chunks) def read_frame(sock: socket.socket) -> str: first_two = recv_exact(sock, 2) first, second = first_two[0], first_two[1] opcode = first & 0x0F masked = second & 0x80 length = second & 0x7F if length == 126: length = struct.unpack("!H", recv_exact(sock, 2))[0] elif length == 127: length = struct.unpack("!Q", recv_exact(sock, 8))[0] mask_key = recv_exact(sock, 4) if masked else b"" payload = recv_exact(sock, length) if masked: payload = bytes(byte ^ mask_key[index % 4] for index, byte in enumerate(payload)) if opcode == 0x8: raise CdpError("websocket closed by peer") if opcode == 0x9: sock.sendall(bytes([0x8A, 0x00])) return read_frame(sock) if opcode != 0x1: raise CdpError(f"unsupported websocket opcode: {opcode}") return payload.decode("utf-8") class CdpSession: """Persistent CDP websocket connection supporting multiple messages and events.""" def __init__(self, websocket_url: str, timeout: float = TIMEOUT_SECONDS): parsed = urllib.parse.urlparse(websocket_url) host = parsed.hostname or "127.0.0.1" port = parsed.port or (443 if parsed.scheme == "wss" else 80) if parsed.scheme != "ws": raise CdpError("only ws:// CDP endpoints are supported") key = websocket_key() request_path = parsed.path or "/" if parsed.query: request_path += f"?{parsed.query}" self._sock = socket.create_connection((host, port), timeout=timeout) self._sock.settimeout(timeout) handshake = ( f"GET {request_path} HTTP/1.1\r\n" f"Host: {host}:{port}\r\n" "Upgrade: websocket\r\n" "Connection: Upgrade\r\n" f"Sec-WebSocket-Key: {key}\r\n" "Sec-WebSocket-Version: 13\r\n\r\n" ) self._sock.sendall(handshake.encode("utf-8")) response = b"" while b"\r\n\r\n" not in response: chunk = self._sock.recv(4096) if not chunk: raise CdpError("incomplete websocket handshake") response += chunk header_blob = response.split(b"\r\n\r\n", 1)[0].decode("utf-8", errors="replace") if "101" not in header_blob.splitlines()[0]: raise CdpError(f"handshake failed: {header_blob.splitlines()[0]}") expected_accept = base64.b64encode( hashlib.sha1((key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").encode("ascii")).digest() ).decode("ascii") if f"Sec-WebSocket-Accept: {expected_accept}" not in header_blob: raise CdpError("invalid websocket accept header") self._next_id = 1 def __enter__(self): return self def __exit__(self, *args): self._sock.close() def send(self, method: str, params: dict | None = None) -> int: msg_id = self._next_id self._next_id += 1 message = {"id": msg_id, "method": method} if params: message["params"] = params self._sock.sendall(build_frame(json.dumps(message))) return msg_id def recv(self) -> dict: return json.loads(read_frame(self._sock)) def call(self, method: str, params: dict | None = None) -> dict: msg_id = self.send(method, params) while True: payload = self.recv() if payload.get("id") == msg_id: if "error" in payload: raise CdpError(str(payload["error"])) return payload def wait_for_event(self, event_method: str, timeout: float) -> dict: deadline = time.monotonic() + timeout old_timeout = self._sock.gettimeout() try: while True: remaining = deadline - time.monotonic() if remaining <= 0: raise CdpError(f"timeout waiting for {event_method}") self._sock.settimeout(remaining) payload = self.recv() if payload.get("method") == event_method: return payload finally: self._sock.settimeout(old_timeout) def set_timeout(self, timeout: float) -> None: self._sock.settimeout(timeout) def websocket_request(websocket_url: str, message: dict[str, object]) -> dict[str, object]: parsed = urllib.parse.urlparse(websocket_url) host = parsed.hostname or "127.0.0.1" port = parsed.port or (443 if parsed.scheme == "wss" else 80) if parsed.scheme != "ws": raise CdpError("only ws:// CDP endpoints are supported") key = websocket_key() request_path = parsed.path or "/" if parsed.query: request_path += f"?{parsed.query}" with socket.create_connection((host, port), timeout=TIMEOUT_SECONDS) as sock: sock.settimeout(TIMEOUT_SECONDS) handshake = ( f"GET {request_path} HTTP/1.1\r\n" f"Host: {host}:{port}\r\n" "Upgrade: websocket\r\n" "Connection: Upgrade\r\n" f"Sec-WebSocket-Key: {key}\r\n" "Sec-WebSocket-Version: 13\r\n\r\n" ) sock.sendall(handshake.encode("utf-8")) response = b"" while b"\r\n\r\n" not in response: chunk = sock.recv(4096) if not chunk: raise CdpError("incomplete websocket handshake") response += chunk header_blob = response.split(b"\r\n\r\n", 1)[0].decode("utf-8", errors="replace") if "101" not in header_blob.splitlines()[0]: raise CdpError(f"handshake failed: {header_blob.splitlines()[0]}") expected_accept = base64.b64encode( hashlib.sha1((key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").encode("ascii")).digest() ).decode("ascii") if f"Sec-WebSocket-Accept: {expected_accept}" not in header_blob: raise CdpError("invalid websocket accept header") sock.sendall(build_frame(json.dumps(message))) while True: payload = json.loads(read_frame(sock)) if payload.get("id") == message["id"]: return payload def evaluate(port: int, target_id: str | None, expression: str) -> dict[str, object]: websocket_url = resolve_websocket_url(port, target_id) response = websocket_request( websocket_url, { "id": 1, "method": "Runtime.evaluate", "params": { "expression": expression, "returnByValue": True, }, }, ) if "error" in response: raise CdpError(str(response["error"])) result = response.get("result", {}) if "exceptionDetails" in result: details = result["exceptionDetails"] description = details.get("exception", {}).get("description") or details.get("text") or "evaluation failed" raise CdpError(str(description)) payload = result.get("result", {}) return payload.get("value") if "value" in payload else payload def detect_challenge(page_info: dict[str, object]) -> dict[str, object]: indicators = [] haystack = " ".join( [ str(page_info.get("title", "")), str(page_info.get("bodySnippet", "")), str(page_info.get("htmlSnippet", "")), ] ).lower() for token in [ "请稍候", "just a moment", "checking your browser", "verify you are human", "cf-challenge", "challenge-platform", "turnstile", "captcha", ]: if token.lower() in haystack: indicators.append(token) return { "hasChallenge": bool(indicators), "indicators": indicators, "title": page_info.get("title", ""), "url": page_info.get("url", ""), } def detect_login_wall(page_info: dict[str, object]) -> dict[str, object]: login_hits = [] combined = " ".join([str(page_info.get("bodySnippet", "")), str(page_info.get("title", ""))]).lower() for token in ["sign in", "log in", "登录", "create your account", "sign up"]: if token.lower() in combined: login_hits.append(token) url = str(page_info.get("url", "")).lower() for token in ["/login", "/signin", "/auth", "/i/flow/login"]: if token in url and token not in login_hits: login_hits.append(token) return { "hasLoginWall": bool(login_hits), "loginHits": login_hits, "title": page_info.get("title", ""), "url": page_info.get("url", ""), } def gather_page_info(port: int, target_id: str | None) -> dict[str, object]: expression = """(() => { const title = document.title || ''; const url = location.href || ''; const bodyText = document.body ? (document.body.innerText || '') : ''; const html = document.documentElement ? (document.documentElement.outerHTML || '') : ''; return { title, url, bodySnippet: bodyText.slice(0, 2000), htmlSnippet: html.slice(0, 4000) }; })()""" value = evaluate(port, target_id, expression) if not isinstance(value, dict): raise CdpError("page-info did not return an object") return value def navigate_and_wait( port: int, target_id: str | None, url: str, wait_selector: str | None = None, wait_navigation: bool = False, ) -> dict[str, object]: """Navigate to *url* via CDP, optionally wait for load and/or a CSS selector.""" websocket_url = resolve_websocket_url(port, target_id) with CdpSession(websocket_url, timeout=NAVIGATE_TIMEOUT_SECONDS) as session: session.call("Page.enable") session.call("Page.navigate", {"url": url}) if wait_navigation: session.wait_for_event("Page.loadEventFired", timeout=NAVIGATE_TIMEOUT_SECONDS) if wait_selector: _wait_for_selector(session, wait_selector) # Return current page info after navigation. resp = session.call( "Runtime.evaluate", { "expression": "({title: document.title, url: location.href})", "returnByValue": True, }, ) result = resp.get("result", {}).get("result", {}) return result.get("value", result) def _wait_for_selector(session: CdpSession, selector: str) -> None: """Poll for a CSS selector to appear, up to WAIT_FOR_TIMEOUT_SECONDS.""" js = f"!!document.querySelector({json.dumps(selector)})" deadline = time.monotonic() + WAIT_FOR_TIMEOUT_SECONDS while True: resp = session.call("Runtime.evaluate", {"expression": js, "returnByValue": True}) value = resp.get("result", {}).get("result", {}).get("value") if value: return if time.monotonic() >= deadline: raise CdpError(f"timeout waiting for selector: {selector}") time.sleep(0.3) def click_link(port: int, target_id: str | None, text: str) -> dict[str, object]: expression = r"""((needle) => { const normalize = (value) => (value || '').replace(/\s+/g, ' ').trim(); const isVisible = (anchor) => { if (!anchor || !anchor.isConnected) return false; if (anchor.hidden || anchor.getAttribute('aria-hidden') === 'true') return false; const style = window.getComputedStyle(anchor); if (!style || style.display === 'none' || style.visibility === 'hidden') return false; if (Number(style.opacity || '1') === 0) return false; return anchor.getClientRects().length > 0; }; const wanted = normalize(needle).toLowerCase(); const anchors = Array.from(document.querySelectorAll('a[href]')); const ranked = anchors.map((anchor, index) => { const anchorText = normalize(anchor.innerText || anchor.textContent); return { index, anchor, text: anchorText, href: anchor.href || '', visible: isVisible(anchor), exact: anchorText.toLowerCase() === wanted, includes: wanted && anchorText.toLowerCase().includes(wanted), }; }).filter(item => item.text && item.href && item.visible); ranked.sort((a, b) => { const score = (item) => item.exact ? 0 : (item.includes ? 1 : 9); return score(a) - score(b) || a.text.length - b.text.length || a.index - b.index; }); const hit = ranked.find(item => item.exact || item.includes); if (!hit) { return { clicked: false, requestedText: needle, candidates: ranked.slice(0, 10).map(item => ({ text: item.text, href: item.href })) }; } hit.anchor.scrollIntoView({block: 'center', inline: 'center'}); hit.anchor.click(); return { clicked: true, requestedText: needle, text: hit.text, href: hit.href }; })(%s)""" % json.dumps(text) value = evaluate(port, target_id, expression) if not isinstance(value, dict): raise CdpError("click-link did not return an object") return value def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(prog="cdp-eval", description="Evaluate page state over the Chrome DevTools Protocol.") parser.add_argument("--port", type=int, required=True, help="CDP HTTP/WebSocket port") parser.add_argument("--target-id", help="Specific target id from /json/list") parser.add_argument("--check", choices=["challenge", "login-wall", "page-info"]) parser.add_argument("--eval", dest="expression", help="Arbitrary JavaScript expression for Runtime.evaluate") parser.add_argument("--navigate", metavar="URL", help="Navigate to URL before evaluating") parser.add_argument("--click-link-text", metavar="TEXT", help="Click the first anchor whose visible text matches or contains TEXT") parser.add_argument("--wait-for", metavar="SELECTOR", help="Wait for CSS selector to appear (timeout 10s)") parser.add_argument("--wait-navigation", action="store_true", help="Wait for page load event after --navigate") args = parser.parse_args() action_count = sum(bool(x) for x in [args.check, args.expression, args.click_link_text]) if args.navigate: pass # --navigate can be used alone or combined with one follow-up action elif action_count != 1: parser.error("provide exactly one of --check, --eval, or --click-link-text (or use --navigate)") if args.wait_for and not args.navigate and not args.expression and not args.check and not args.click_link_text: parser.error("--wait-for requires --navigate, --check, --eval, or --click-link-text") if args.wait_navigation and not args.navigate: parser.error("--wait-navigation requires --navigate") return args def main() -> int: args = parse_args() try: if args.navigate: nav_result = navigate_and_wait( args.port, args.target_id, args.navigate, wait_selector=args.wait_for, wait_navigation=args.wait_navigation, ) if not args.expression and not args.check: print(json.dumps(nav_result, ensure_ascii=False)) return 0 if args.expression: print(json.dumps(evaluate(args.port, args.target_id, args.expression), ensure_ascii=False)) return 0 if args.click_link_text: print(json.dumps(click_link(args.port, args.target_id, args.click_link_text), ensure_ascii=False)) return 0 page_info = gather_page_info(args.port, args.target_id) if args.check == "page-info": result = { "title": page_info.get("title", ""), "url": page_info.get("url", ""), "bodySnippet": page_info.get("bodySnippet", ""), } elif args.check == "challenge": result = detect_challenge(page_info) else: result = detect_login_wall(page_info) print(json.dumps(result, ensure_ascii=False)) return 0 except (OSError, urllib.error.URLError, socket.timeout) as exc: print(json.dumps({"error": str(exc)})) return 1 except CdpError as exc: print(json.dumps({"error": str(exc)})) return 2 if __name__ == "__main__": sys.exit(main()) FILE:scripts/cdp-snapshot.py #!/usr/bin/env python3 """Capture a structured snapshot of a browser page via CDP. Reuses the low-level CDP helpers from cdp-eval.py (same directory) so there is no duplicated websocket code. """ from __future__ import annotations import argparse import importlib.util import json import os import sys # Import shared CDP helpers from the sibling module (hyphenated filename). _here = os.path.dirname(os.path.abspath(__file__)) _spec = importlib.util.spec_from_file_location("cdp_eval", os.path.join(_here, "cdp-eval.py")) _mod = importlib.util.module_from_spec(_spec) _spec.loader.exec_module(_mod) CdpError = _mod.CdpError evaluate = _mod.evaluate # --------------------------------------------------------------------------- # JavaScript extraction snippets # --------------------------------------------------------------------------- _JS_MARKDOWN = r"""(() => { const lines = []; const title = document.title || ''; if (title) lines.push('# ' + title, ''); const url = location.href || ''; if (url) lines.push('URL: ' + url, ''); function walkNode(node, depth) { if (node.nodeType === Node.TEXT_NODE) { const t = node.textContent.trim(); if (t) lines.push(t); return; } if (node.nodeType !== Node.ELEMENT_NODE) return; const tag = node.tagName; if (['SCRIPT', 'STYLE', 'NOSCRIPT', 'SVG'].includes(tag)) return; if (tag === 'H1') lines.push('', '# ' + (node.innerText || '').trim(), ''); else if (tag === 'H2') lines.push('', '## ' + (node.innerText || '').trim(), ''); else if (tag === 'H3') lines.push('', '### ' + (node.innerText || '').trim(), ''); else if (tag === 'H4') lines.push('', '#### ' + (node.innerText || '').trim(), ''); else if (tag === 'H5') lines.push('', '##### ' + (node.innerText || '').trim(), ''); else if (tag === 'H6') lines.push('', '###### ' + (node.innerText || '').trim(), ''); else if (tag === 'A') { const href = node.getAttribute('href') || ''; const text = (node.innerText || '').trim(); if (text && href) lines.push('[' + text + '](' + href + ')'); else if (text) lines.push(text); return; // don't recurse into links } else if (tag === 'LI') { lines.push('- ' + (node.innerText || '').trim()); return; } else if (tag === 'P' || tag === 'DIV' || tag === 'SECTION' || tag === 'ARTICLE') { // recurse into block elements for (const child of node.childNodes) walkNode(child, depth + 1); lines.push(''); return; } else if (tag === 'BR') { lines.push(''); return; } else if (tag === 'IMG') { const alt = node.getAttribute('alt') || ''; if (alt) lines.push('[image: ' + alt + ']'); return; } else { for (const child of node.childNodes) walkNode(child, depth + 1); return; } } if (document.body) walkNode(document.body, 0); // collapse multiple blank lines const collapsed = []; let lastBlank = false; for (const line of lines) { if (line === '') { if (!lastBlank) collapsed.push(''); lastBlank = true; } else { collapsed.push(line); lastBlank = false; } } return { title, url, content: collapsed.join('\n') }; })()""" _JS_TEXT = r"""(() => { const title = document.title || ''; const url = location.href || ''; const text = document.body ? (document.body.innerText || '') : ''; return { title, url, content: text }; })()""" _JS_LINKS = r"""(() => { const title = document.title || ''; const url = location.href || ''; const anchors = Array.from(document.querySelectorAll('a[href]')); const links = anchors.map(a => ({ text: (a.innerText || '').trim(), href: a.href })).filter(l => l.text && l.href); return { title, url, content: JSON.stringify(links) }; })()""" _JS_TOPIC_LINKS = r"""(() => { const title = document.title || ''; const url = location.href || ''; const normalizeText = (value) => (value || '').replace(/\s+/g, ' ').trim(); const toAbsoluteHref = (href) => { if (!href) return ''; try { const u = new URL(href, location.href); if (!['http:', 'https:', 'file:'].includes(u.protocol)) return ''; return u.href; } catch { return ''; } }; const isPreferredTopicHref = (href) => { const absolute = toAbsoluteHref(href); if (!absolute) return false; try { const u = new URL(absolute); return ( /^\/t\//.test(u.pathname) || /\/(topic|thread|discussion|comment|comments|question|questions|item)s?(\/|$)/i.test(u.pathname) || (/viewtopic\.php$/i.test(u.pathname) && u.searchParams.has('t')) ); } catch { return false; } }; const topicRoots = Array.from(new Set([ document.querySelector('main'), document.querySelector('article'), document.querySelector('[role=main]') ].filter(Boolean))); const chooseAnchor = (anchors) => { const ranked = anchors.map((anchor, index) => { const text = normalizeText(anchor.innerText || anchor.textContent); const href = toAbsoluteHref(anchor.getAttribute('href') || anchor.href || ''); if (!text || !href) return null; let score = 0; if (isPreferredTopicHref(href)) score += 4; if (anchor.closest('h1, h2, h3, h4')) score += 2; const className = normalizeText([ anchor.className || '', anchor.parentElement ? anchor.parentElement.className || '' : '' ].join(' ')).toLowerCase(); if (/(^| )(title|topic|question|result|headline|subject)( |$)/.test(className)) score += 1; score += Math.min(text.length, 120) / 120; return { anchor, href, index, score, text }; }).filter(Boolean); ranked.sort((a, b) => b.score - a.score || b.text.length - a.text.length || a.index - b.index); return ranked[0] || null; }; const seen = new Set(); const results = []; const collectFromContainer = (container) => { const chosen = chooseAnchor(Array.from(container.querySelectorAll('a[href]'))); if (!chosen || seen.has(chosen.href)) return; seen.add(chosen.href); const item = { text: chosen.text, href: chosen.href, }; const meta = normalizeText(container.innerText || ''); if (meta) item.meta = meta.slice(0, 400); const topicId = container.getAttribute('data-topic-id'); if (topicId) item.topicId = topicId; results.push(item); }; const containers = Array.from(document.querySelectorAll([ '.search-results .fps-result', '.topic-list-item', '.latest-topic-list-item', '.category-topic-link', 'tr.topic-list-item', '[data-topic-id]' ].join(','))); for (const container of containers) { collectFromContainer(container); } if (!results.length) { const repeatedContainers = []; for (const root of topicRoots) { for (const child of Array.from(root.children || [])) { if (child.querySelector('a[href]')) repeatedContainers.push(child); } } if (repeatedContainers.length >= 2) { for (const container of repeatedContainers) { collectFromContainer(container); } } } if (!results.length) { const roots = topicRoots.length ? topicRoots : [document.body].filter(Boolean); for (const root of roots) { for (const anchor of Array.from(root.querySelectorAll('a[href]'))) { const text = normalizeText(anchor.innerText || anchor.textContent); const href = toAbsoluteHref(anchor.getAttribute('href') || anchor.href || ''); if (!text || !href || seen.has(href)) continue; seen.add(href); results.push({ text, href }); } } } return { title, url, content: JSON.stringify(results) }; })()""" _FORMAT_JS = { "markdown": _JS_MARKDOWN, "text": _JS_TEXT, "links": _JS_LINKS, "topic-links": _JS_TOPIC_LINKS, } def snapshot(port: int, target_id: str | None, fmt: str, max_chars: int) -> dict[str, str]: js = _FORMAT_JS[fmt] value = evaluate(port, target_id, js) if not isinstance(value, dict): raise CdpError("snapshot did not return an object") content = str(value.get("content", "")) if max_chars and len(content) > max_chars: content = content[:max_chars] + "\n[truncated]" return { "title": str(value.get("title", "")), "url": str(value.get("url", "")), "content": content, } def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( prog="cdp-snapshot", description="Capture a structured page snapshot via CDP.", ) parser.add_argument("--port", type=int, required=True, help="CDP HTTP/WebSocket port") parser.add_argument("--target-id", help="Specific target id from /json/list") parser.add_argument( "--format", choices=["markdown", "text", "links", "topic-links"], default="markdown", help="Output format (default: markdown)", ) parser.add_argument( "--max-chars", type=int, default=8000, help="Truncate content to N chars (default: 8000, 0=unlimited)", ) return parser.parse_args() def main() -> int: args = parse_args() try: result = snapshot(args.port, args.target_id, args.format, args.max_chars) print(json.dumps(result, ensure_ascii=False)) return 0 except (OSError, CdpError) as exc: print(json.dumps({"error": str(exc)})) return 1 if __name__ == "__main__": sys.exit(main()) FILE:scripts/open-protected-page.sh #!/usr/bin/env bash set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck disable=SC1091 source "$SCRIPT_DIR/runtime-common.sh" usage() { cat <<'EOF' Usage: open-protected-page.sh --url URL [options] Options: --manifest-root DIR --origin URL --session-key KEY --url URL EOF } die() { printf '[open-protected-page] ERROR: %s\n' "$*" >&2 exit 1 } require_arg() { local name="$1" local value="$2" [ -n "$value" ] || die "missing required argument: $name" } manifest_helper() { "${AGENT_BROWSER_MANIFEST_HELPER:-$SCRIPT_DIR/session-manifest.sh}" "$@" } runtime_helper() { "${AGENT_BROWSER_RUNTIME_HELPER:-$SCRIPT_DIR/browser-runtime.sh}" "$@" } assisted_helper() { "${AGENT_BROWSER_ASSISTED_HELPER:-$SCRIPT_DIR/assisted-session.sh}" "$@" } profile_helper() { "${AGENT_BROWSER_PROFILE_HELPER:-$SCRIPT_DIR/profile-resolution.sh}" "$@" } manifest_field() { local field="$1" local payload="$2" python3 - "$field" "$payload" <<'PY' import json import sys payload = json.loads(sys.argv[2]) value = payload.get(sys.argv[1], "") if isinstance(value, (dict, list)): import json as _json print(_json.dumps(value)) else: print(value) PY } status_value() { local key="$1" local payload="$2" python3 - "$key" "$payload" <<'PY' import sys target = sys.argv[1] for raw in sys.argv[2].splitlines(): if ":" not in raw: continue key, value = raw.split(":", 1) if key.strip() == target: print(value.strip()) break PY } emit_result() { local kind="$1" local action="$2" local target_id="$3" local page_json="$4" local assisted_status="$5" local reason="$6" local cdp_port="$7" python3 - "$kind" "$action" "$target_id" "$page_json" "$assisted_status" "$reason" "$cdp_port" <<'PY' import json import sys kind, action, target_id, page_json, assisted_status, reason, cdp_port = sys.argv[1:] payload = {"status": kind, "action": action} if target_id: payload["targetId"] = target_id if cdp_port: try: payload["cdpPort"] = int(cdp_port) except ValueError: pass if reason: payload["reason"] = reason if page_json: page = json.loads(page_json) if page.get("url"): payload["url"] = page["url"] if page.get("title"): payload["title"] = page["title"] if page.get("bodySnippet"): payload["bodySnippet"] = page["bodySnippet"] if assisted_status: for raw in assisted_status.splitlines(): if ":" not in raw: continue key, value = raw.split(":", 1) if key.strip() == "novnc_url": payload["novncUrl"] = value.strip() if key.strip() == "lan_novnc_url": payload["lanNovncUrl"] = value.strip() print(json.dumps(payload, ensure_ascii=False)) PY } page_field() { local payload="$1" local field="$2" python3 - "$payload" "$field" <<'PY' import json import sys payload = json.loads(sys.argv[1]) value = payload.get(sys.argv[2], "") print(value if isinstance(value, str) else "") PY } page_matches_target() { local page_url="${1:-}" python3 - "$page_url" "${INITIAL_URL:-}" "${ORIGIN:-}" <<'PY' import sys from urllib.parse import urlparse, urlunparse page_url, initial_url, origin = sys.argv[1:] def normalize(value: str) -> str: raw = (value or "").strip() if not raw: return "" parsed = urlparse(raw) if not parsed.scheme or not parsed.netloc: return raw.rstrip("/") path = parsed.path or "" if path not in ("", "/"): path = path.rstrip("/") else: path = "" return urlunparse(( parsed.scheme.lower(), parsed.netloc.lower(), path, parsed.params, parsed.query, "", )) page = normalize(page_url) initial = normalize(initial_url) origin_value = normalize(origin) if initial and initial != origin_value: raise SystemExit(0 if page == initial else 1) if origin_value and (page == origin_value or page.startswith(origin_value + "/")): raise SystemExit(0) raise SystemExit(1) PY } page_ready() { local payload="$1" local page_url page_url="$(page_field "$payload" url)" page_loaded "$payload" || return 1 page_matches_target "$page_url" } page_loaded() { local payload="$1" local page_url page_body page_url="$(page_field "$payload" url)" page_body="$(page_field "$payload" bodySnippet)" [ -n "$page_body" ] || return 1 [ "$page_url" != "about:blank" ] || return 1 [ -n "$page_url" ] || return 1 return 0 } page_mismatch() { local payload="$1" page_loaded "$payload" || return 1 page_ready "$payload" && return 1 return 0 } resolve_cdp_port() { local status_output status_output="$(runtime_helper status --origin "$ORIGIN" --session-key "$SESSION_KEY" 2>/dev/null || true)" status_value cdp_port "$status_output" } select_existing_session() { local manifest selected_key if manifest="$(manifest_helper show --root "$MANIFEST_ROOT" --origin "$ORIGIN" --session-key "$SESSION_KEY" 2>/dev/null)"; then selected_key="$(manifest_field session_key "$manifest" || true)" if [ -n "$selected_key" ]; then SESSION_KEY="$selected_key" return 0 fi fi if manifest="$(manifest_helper select --root "$MANIFEST_ROOT" --origin "$ORIGIN" 2>/dev/null)"; then selected_key="$(manifest_field session_key "$manifest" || true)" if [ -n "$selected_key" ]; then SESSION_KEY="$selected_key" return 0 fi fi return 1 } resolve_profile() { local resolved profile_dir resolved="$( profile_helper resolve \ --root "$HOME/.agent-browser" \ --manifest-root "$MANIFEST_ROOT" \ --origin "$ORIGIN" \ --session-key "$SESSION_KEY" )" || return $? profile_dir="$(manifest_field profile_dir "$resolved" || true)" [ -n "$profile_dir" ] || die "profile resolver returned no profile_dir" PROFILE_DIR="$profile_dir" } ensure_runtime() { local runtime_status_output if runtime_helper verify --manifest-root "$MANIFEST_ROOT" --origin "$ORIGIN" --session-key "$SESSION_KEY" >/dev/null 2>&1; then return 0 fi runtime_status_output="$(runtime_helper status --origin "$ORIGIN" --session-key "$SESSION_KEY" 2>/dev/null || true)" if printf '%s\n' "$runtime_status_output" | grep -q '^runtime: running$'; then return 0 fi resolve_profile || return $? runtime_helper start \ --url "$INITIAL_URL" \ --origin "$ORIGIN" \ --session-key "$SESSION_KEY" \ --profile-dir "$PROFILE_DIR" \ --mode gui >/dev/null } recover_target_mismatch() { if [ -z "${PROFILE_DIR:-}" ]; then resolve_profile || return $? fi runtime_helper stop --origin "$ORIGIN" --session-key "$SESSION_KEY" >/dev/null 2>&1 || true runtime_helper start \ --url "$INITIAL_URL" \ --origin "$ORIGIN" \ --session-key "$SESSION_KEY" \ --profile-dir "$PROFILE_DIR" \ --mode gui >/dev/null } COMMAND_URL="" ORIGIN="" MANIFEST_ROOT="" SESSION_KEY="default" PROFILE_DIR="" while [ "$#" -gt 0 ]; do case "$1" in --url) COMMAND_URL="$2" shift 2 ;; --origin) ORIGIN="$2" shift 2 ;; --manifest-root) MANIFEST_ROOT="$2" shift 2 ;; --session-key) SESSION_KEY="$2" shift 2 ;; -h|--help) usage exit 0 ;; *) die "unknown argument: $1" ;; esac done require_arg --url "$COMMAND_URL" INITIAL_URL="$COMMAND_URL" ORIGIN="${ORIGIN:-$(derive_origin "$INITIAL_URL")}" MANIFEST_ROOT="${MANIFEST_ROOT:-$HOME/.agent-browser}" select_existing_session || true if ! ensure_runtime; then emit_result "needs-user" "open-novnc" "" "" "" "ambiguous-profile" "" exit 0 fi CDP_PORT="$(resolve_cdp_port)" RECOVERY_ATTEMPTED=0 for _attempt in 1 2 3 4 5; do TARGETS_JSON="$(runtime_helper list-targets --origin "$ORIGIN" --session-key "$SESSION_KEY")" TARGET_ID="$( runtime_helper select-target \ --origin "$ORIGIN" \ --target-url "$INITIAL_URL" \ --targets-json "$TARGETS_JSON" )" CHALLENGE_JSON="$(runtime_helper check-page --origin "$ORIGIN" --session-key "$SESSION_KEY" --target-id "$TARGET_ID" --check challenge)" LOGIN_JSON="$(runtime_helper check-page --origin "$ORIGIN" --session-key "$SESSION_KEY" --target-id "$TARGET_ID" --check login-wall)" PAGE_JSON="$(runtime_helper check-page --origin "$ORIGIN" --session-key "$SESSION_KEY" --target-id "$TARGET_ID" --check page-info)" if [ "$RECOVERY_ATTEMPTED" -eq 0 ] && page_mismatch "$PAGE_JSON"; then if recover_target_mismatch; then RECOVERY_ATTEMPTED=1 sleep 1 continue fi fi if printf '%s' "$CHALLENGE_JSON" | grep -q '"hasChallenge": *true'; then break fi if printf '%s' "$LOGIN_JSON" | grep -q '"hasLoginWall": *true'; then break fi if page_ready "$PAGE_JSON"; then break fi sleep 1 done if printf '%s' "$CHALLENGE_JSON" | grep -q '"hasChallenge": *true'; then assisted_helper start --url "$INITIAL_URL" --origin "$ORIGIN" --session-key "$SESSION_KEY" >/dev/null ASSISTED_STATUS="$(assisted_helper status --url "$INITIAL_URL" --origin "$ORIGIN" --session-key "$SESSION_KEY")" emit_result "needs-user" "open-novnc" "$TARGET_ID" "" "$ASSISTED_STATUS" "challenge" "$CDP_PORT" exit 0 fi if printf '%s' "$LOGIN_JSON" | grep -q '"hasLoginWall": *true'; then assisted_helper start --url "$INITIAL_URL" --origin "$ORIGIN" --session-key "$SESSION_KEY" >/dev/null ASSISTED_STATUS="$(assisted_helper status --url "$INITIAL_URL" --origin "$ORIGIN" --session-key "$SESSION_KEY")" emit_result "needs-user" "open-novnc" "$TARGET_ID" "" "$ASSISTED_STATUS" "login-wall" "$CDP_PORT" exit 0 fi if ! page_ready "$PAGE_JSON"; then assisted_helper start --url "$INITIAL_URL" --origin "$ORIGIN" --session-key "$SESSION_KEY" >/dev/null ASSISTED_STATUS="$(assisted_helper status --url "$INITIAL_URL" --origin "$ORIGIN" --session-key "$SESSION_KEY")" emit_result "needs-user" "open-novnc" "$TARGET_ID" "$PAGE_JSON" "$ASSISTED_STATUS" "target-mismatch" "$CDP_PORT" exit 0 fi emit_result "ready" "report-page" "$TARGET_ID" "$PAGE_JSON" "" "" "$CDP_PORT" FILE:scripts/profile-resolution.sh #!/usr/bin/env bash set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck disable=SC1091 source "$SCRIPT_DIR/runtime-common.sh" usage() { cat <<'EOF' Usage: profile-resolution.sh resolve --origin URL [options] profile-resolution.sh show-identity --provider HOST [options] profile-resolution.sh write-identity --provider HOST --profile-dir DIR [options] Options: --manifest-root DIR --origin URL --profile-dir DIR --provider HOST --root DIR --session-key KEY --source-origin URL --source-session-key KEY EOF } die() { printf '[profile-resolution] ERROR: %s\n' "$*" >&2 exit 1 } require_arg() { local name="$1" local value="$2" [ -n "$value" ] || die "missing required argument: $name" } timestamp() { date -u +"%Y-%m-%dT%H:%M:%SZ" } identity_index_path() { printf '%s/index/identity-profiles.json\n' "$ROOT_DIR" } site_registry_helper() { "${AGENT_BROWSER_SITE_REGISTRY_HELPER:-$SCRIPT_DIR/site-session-registry.sh}" "$@" } provider_aliases_json() { local value="$1" local aliases=() mapfile -t aliases < <(provider_aliases "$value") python3 - "${aliases[@]}" <<'PY' import json import sys print(json.dumps(sys.argv[1:])) PY } profile_resolution_json() { python3 - <<'PY' import json import os from urllib.parse import urlparse origin = os.environ["ORIGIN"] session_key = os.environ.get("SESSION_KEY", "default") root_dir = os.environ["ROOT_DIR"] explicit_profile = os.environ.get("PROFILE_DIR", "") manifest_payload = os.environ.get("MANIFEST_PAYLOAD", "") aliases = json.loads(os.environ.get("ALIASES_JSON", "[]")) def host_from_origin(value: str) -> str: parsed = urlparse(value) return (parsed.netloc or value).lower() def evidence_score(path: str) -> int: score = 0 if os.path.exists(os.path.join(path, "Default", "Cookies")): score += 1 if os.path.exists(os.path.join(path, "Default", "Login Data")): score += 1 if os.path.exists(os.path.join(path, "Local State")): score += 1 return score if explicit_profile: print(json.dumps({"profile_dir": explicit_profile, "source": "explicit"})) raise SystemExit(0) if manifest_payload: manifest = json.loads(manifest_payload) profile_dir = manifest.get("profile_dir") or "" if profile_dir: print(json.dumps({"profile_dir": profile_dir, "source": "manifest"})) raise SystemExit(0) identity_path = os.path.join(root_dir, "index", "identity-profiles.json") if aliases and os.path.exists(identity_path): try: with open(identity_path, "r", encoding="utf-8") as handle: payload = json.load(handle) except (json.JSONDecodeError, OSError): payload = {"providers": {}} providers = payload.get("providers", {}) for alias in aliases: entry = providers.get(alias) or {} profile_dir = entry.get("profile_dir") or "" if profile_dir: print(json.dumps({ "profile_dir": profile_dir, "source": "identity-index", "provider": alias, "source_origin": entry.get("source_origin"), "source_session_key": entry.get("source_session_key"), })) raise SystemExit(0) host = host_from_origin(origin) legacy_profile = os.path.join(root_dir, "profiles", host) scoped_profile = os.path.join(root_dir, "profiles", os.environ["ORIGIN_KEY"], session_key) legacy_exists = os.path.isdir(legacy_profile) scoped_exists = os.path.isdir(scoped_profile) if legacy_exists and scoped_exists: legacy_score = evidence_score(legacy_profile) scoped_score = evidence_score(scoped_profile) if legacy_score > scoped_score: print(json.dumps({"profile_dir": legacy_profile, "source": "legacy"})) raise SystemExit(0) if scoped_score > legacy_score: print(json.dumps({"profile_dir": scoped_profile, "source": "scoped"})) raise SystemExit(0) if legacy_score > 0: print(json.dumps({ "error": "ambiguous-profile", "legacy_profile": legacy_profile, "scoped_profile": scoped_profile, })) raise SystemExit(2) print(json.dumps({"profile_dir": legacy_profile, "source": "legacy"})) raise SystemExit(0) if legacy_exists: print(json.dumps({"profile_dir": legacy_profile, "source": "legacy"})) raise SystemExit(0) print(json.dumps({"profile_dir": scoped_profile, "source": "scoped"})) PY } cmd_resolve() { require_arg --origin "$ORIGIN" ORIGIN_KEY="$(origin_slug "$ORIGIN")" SITE_KEY="$(site_key "$ORIGIN")" ALIASES_JSON="$(provider_aliases_json "$ORIGIN")" MANIFEST_PAYLOAD="$( "$SCRIPT_DIR/session-manifest.sh" show --root "$MANIFEST_ROOT" --origin "$ORIGIN" --session-key "$SESSION_KEY" 2>/dev/null || true )" SITE_PAYLOAD="$( site_registry_helper resolve --root "$ROOT_DIR" --site "$SITE_KEY" --session-key "$SESSION_KEY" 2>/dev/null || true )" if [ -n "$SITE_PAYLOAD" ]; then python3 - "$SITE_PAYLOAD" <<'PY' import json import sys payload = json.loads(sys.argv[1]) print(json.dumps({ "profile_dir": payload["profile_dir"], "site": payload.get("site"), "session_key": payload.get("session_key"), "source": "site-registry", })) PY return 0 fi export ORIGIN SESSION_KEY ROOT_DIR PROFILE_DIR MANIFEST_PAYLOAD ALIASES_JSON ORIGIN_KEY profile_resolution_json } cmd_show_identity() { require_arg --provider "$PROVIDER" local path path="$(identity_index_path)" [ -f "$path" ] || exit 1 python3 - "$path" "$PROVIDER" <<'PY' import json import sys with open(sys.argv[1], "r", encoding="utf-8") as handle: try: payload = json.load(handle) except json.JSONDecodeError: raise SystemExit(1) entry = payload.get("providers", {}).get(sys.argv[2], {}) if not entry: raise SystemExit(1) print(json.dumps(entry, sort_keys=True)) PY } cmd_write_identity() { require_arg --provider "$PROVIDER" require_arg --profile-dir "$PROFILE_DIR" require_arg --source-origin "$SOURCE_ORIGIN" require_arg --source-session-key "$SOURCE_SESSION_KEY" local path now path="$(identity_index_path)" now="$(timestamp)" mkdir -p "$(dirname "$path")" python3 - "$path" "$PROVIDER" "$PROFILE_DIR" "$SOURCE_ORIGIN" "$SOURCE_SESSION_KEY" "$now" <<'PY' import json import os import sys path, provider, profile_dir, source_origin, source_session_key, now = sys.argv[1:] payload = {"providers": {}} if os.path.exists(path): try: with open(path, "r", encoding="utf-8") as handle: payload = json.load(handle) except (json.JSONDecodeError, OSError): payload = {"providers": {}} providers = payload.setdefault("providers", {}) providers[provider] = { "profile_dir": profile_dir, "source_origin": source_origin, "source_session_key": source_session_key, "updated_at": now, } with open(path, "w", encoding="utf-8") as handle: json.dump(payload, handle, indent=2, sort_keys=True) handle.write("\n") PY } COMMAND="${1:-}" [ -n "$COMMAND" ] || { usage exit 1 } shift || true ROOT_DIR="${ROOT_DIR:-$HOME/.agent-browser}" MANIFEST_ROOT="${MANIFEST_ROOT:-$ROOT_DIR}" ORIGIN="" SESSION_KEY="default" PROFILE_DIR="" PROVIDER="" SOURCE_ORIGIN="" SOURCE_SESSION_KEY="" while [ "$#" -gt 0 ]; do case "$1" in --root) ROOT_DIR="$2" shift 2 ;; --manifest-root) MANIFEST_ROOT="$2" shift 2 ;; --origin) ORIGIN="$2" shift 2 ;; --session-key) SESSION_KEY="$2" shift 2 ;; --profile-dir) PROFILE_DIR="$2" shift 2 ;; --provider) PROVIDER="$2" shift 2 ;; --source-origin) SOURCE_ORIGIN="$2" shift 2 ;; --source-session-key) SOURCE_SESSION_KEY="$2" shift 2 ;; -h|--help) usage exit 0 ;; *) die "unknown argument: $1" ;; esac done case "$COMMAND" in resolve) cmd_resolve ;; show-identity) cmd_show_identity ;; write-identity) cmd_write_identity ;; *) usage exit 1 ;; esac FILE:scripts/runtime-common.sh #!/usr/bin/env bash derive_origin() { python3 - "${1:-}" <<'PY' import sys from urllib.parse import urlparse raw = sys.argv[1].strip() if not raw: print("") raise SystemExit(0) parsed = urlparse(raw) if parsed.scheme and parsed.netloc: print(f"{parsed.scheme}://{parsed.netloc}") else: print(raw) PY } origin_slug() { python3 - "$1" <<'PY' import sys value = sys.argv[1].strip().lower() for old, new in ( ("://", "___"), ("/", "_"), (":", "_"), ("?", "_"), ("&", "_"), ("=", "_"), (".", "_"), ): value = value.replace(old, new) print(value.strip("_") or "default") PY } runtime_scoped_path() { local base_root="$1" local category="$2" local origin="$3" local session_key="${4:-default}" local slug slug="$(origin_slug "$origin")" printf '%s/%s/%s/%s\n' "$base_root" "$category" "$slug" "$session_key" } site_key() { python3 - "${1:-}" <<'PY' import sys from urllib.parse import urlparse raw = (sys.argv[1] or "").strip().lower() host = urlparse(raw).netloc.lower() if "://" in raw else raw google_hosts = { "google.com", "www.google.com", "myaccount.google.com", "accounts.google.com", } if host.endswith("github.com"): print("github.com") elif host in google_hosts: print("google.com") else: print(host or "default") PY } provider_aliases() { python3 - "${1:-}" <<'PY' import sys from urllib.parse import urlparse raw = (sys.argv[1] or "").strip().lower() host = urlparse(raw).netloc.lower() if "://" in raw else raw aliases = [] if host: aliases.append(host) if host.endswith("github.com"): aliases.append("github.com") if host.endswith("google.com"): aliases.extend(["accounts.google.com", "myaccount.google.com", "google.com"]) seen = [] for item in aliases: if item and item not in seen: seen.append(item) print("\n".join(seen)) PY } lan_novnc_url() { local host="$1" local port="$2" printf 'http://%s:%s/vnc.html?autoconnect=1&resize=remote\n' "$host" "$port" } primary_ipv4() { local host="${AGENT_BROWSER_NOVNC_PUBLIC_HOST:-}" if [ -n "$host" ]; then printf '%s\n' "$host" return 0 fi if command -v ip >/dev/null 2>&1; then ip route get 1.1.1.1 2>/dev/null | awk '/src/ {for (i = 1; i <= NF; i++) if ($i == "src") {print $(i + 1); exit}}' return 0 fi return 1 } pick_free_tcp_port() { python3 - "$1" <<'PY' import socket import sys start = int(sys.argv[1]) for port in range(start, start + 50): with socket.socket() as sock: sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) try: sock.bind(("127.0.0.1", port)) except OSError: continue print(port) raise SystemExit(0) raise SystemExit(1) PY } pick_free_display() { local start="${1:-88}" local socket_dir="${AGENT_BROWSER_X11_SOCKET_DIR:-/tmp/.X11-unix}" local candidate socket_path for candidate in $(seq "$start" $((start + 50))); do socket_path="$socket_dir/X${candidate}" if [ -e "$socket_path" ]; then continue fi if ps -eo command= 2>/dev/null | grep -Eq "(^|[[:space:]])(:${candidate})([[:space:]]|$)"; then continue fi printf '%s\n' "$candidate" return 0 done return 1 } FILE:scripts/session-manifest.sh #!/usr/bin/env bash set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck disable=SC1091 source "$SCRIPT_DIR/runtime-common.sh" usage() { cat <<'EOF' Usage: session-manifest.sh list [--root DIR] session-manifest.sh path --origin URL [--session-key KEY] [--root DIR] session-manifest.sh write --origin URL --session-key KEY --state STATE --browser-pid PID [options] session-manifest.sh mark-stale --origin URL [--session-key KEY] --reason TEXT [--root DIR] session-manifest.sh show --origin URL [--session-key KEY] [--root DIR] session-manifest.sh index-show --origin URL [--root DIR] session-manifest.sh select --origin URL [--account-hint TEXT] [--task-scope TEXT] [--root DIR] Options: --account-hint TEXT --block-reason TEXT --root DIR --session-key KEY --task-scope TEXT EOF } die() { printf '[session-manifest] ERROR: %s\n' "$*" >&2 exit 1 } timestamp() { date -u +"%Y-%m-%dT%H:%M:%SZ" } require_arg() { local name="$1" local value="$2" [ -n "$value" ] || die "missing required argument: $name" } init_paths() { MANIFEST_ROOT="${MANIFEST_ROOT:-$HOME/.agent-browser}" ROOT_DIR="${ROOT_DIR:-$MANIFEST_ROOT}" MANIFEST_DIR="$ROOT_DIR/sessions" INDEX_DIR="$ROOT_DIR/index" mkdir -p "$MANIFEST_DIR" "$INDEX_DIR" } origin_key() { origin_slug "$1" } origin_dir() { printf '%s/%s\n' "$MANIFEST_DIR" "$(origin_key "$1")" } index_path() { printf '%s/%s.json\n' "$INDEX_DIR" "$(origin_key "$1")" } manifest_path() { local origin="$1" local session_key="$2" printf '%s/%s.json\n' "$(origin_dir "$origin")" "$session_key" } ensure_origin_dir() { mkdir -p "$(origin_dir "$1")" } rebuild_index() { local origin="$1" local origin_path local index_file origin_path="$(origin_dir "$origin")" index_file="$(index_path "$origin")" python3 - "$origin_path" "$index_file" <<'PY' import json import os import sys origin_dir = sys.argv[1] index_file = sys.argv[2] items = [] if os.path.isdir(origin_dir): for name in sorted(os.listdir(origin_dir)): if not name.endswith(".json"): continue path = os.path.join(origin_dir, name) try: with open(path, "r", encoding="utf-8") as handle: data = json.load(handle) except FileNotFoundError: continue items.append({ "session_key": data.get("session_key"), "account_hint": data.get("account_hint"), "task_scope": data.get("task_scope"), "state": data.get("state"), "browser_pid": data.get("browser_pid"), "last_verified_at": data.get("last_verified_at"), "path": path, }) with open(index_file, "w", encoding="utf-8") as handle: json.dump({"sessions": items}, handle, indent=2, sort_keys=True) handle.write("\n") PY } cmd_list() { if [ ! -d "$MANIFEST_DIR" ]; then exit 0 fi find "$MANIFEST_DIR" -mindepth 2 -maxdepth 2 -type f -name '*.json' | sort } cmd_path() { require_arg --origin "$ORIGIN" if [ -n "$SESSION_KEY" ]; then printf '%s\n' "$(manifest_path "$ORIGIN" "$SESSION_KEY")" return 0 fi printf '%s\n' "$(origin_dir "$ORIGIN")" } cmd_write() { require_arg --origin "$ORIGIN" require_arg --session-key "$SESSION_KEY" require_arg --state "$STATE" require_arg --browser-pid "$BROWSER_PID" ensure_origin_dir "$ORIGIN" local path now created_at path="$(manifest_path "$ORIGIN" "$SESSION_KEY")" now="$(timestamp)" created_at="$now" if [ -f "$path" ]; then created_at="$(python3 - "$path" <<'PY' import json import sys with open(sys.argv[1], "r", encoding="utf-8") as handle: data = json.load(handle) print(data.get("created_at", "")) PY )" created_at="${created_at:-$now}" fi python3 - "$path" <<'PY' import json import os import sys path = sys.argv[1] payload = { "origin": os.environ["ORIGIN"], "session_key": os.environ["SESSION_KEY"], "account_hint": os.environ.get("ACCOUNT_HINT") or None, "task_scope": os.environ.get("TASK_SCOPE") or None, "state": os.environ["STATE"], "browser_pid": int(os.environ["BROWSER_PID"]), "created_at": os.environ["CREATED_AT"], "last_verified_at": os.environ["NOW"], } optional_keys = [ "BLOCK_REASON", "CDP_PORT", "CDP_URL", "WEBSOCKET_DEBUGGER_URL", "TARGET_ID", "PROFILE_DIR", "MODE", "DISPLAY_VALUE", "XVFB_DISPLAY", "XVFB_PID", "NOVNC_PORT", ] field_map = { "BLOCK_REASON": "block_reason", "CDP_PORT": "cdp_port", "CDP_URL": "cdp_url", "WEBSOCKET_DEBUGGER_URL": "websocket_debugger_url", "TARGET_ID": "target_id", "PROFILE_DIR": "profile_dir", "MODE": "mode", "DISPLAY_VALUE": "display", "XVFB_DISPLAY": "xvfb_display", "XVFB_PID": "xvfb_pid", "NOVNC_PORT": "novnc_port", } int_fields = {"CDP_PORT", "XVFB_PID", "NOVNC_PORT"} for key in optional_keys: value = os.environ.get(key) if value in (None, ""): continue payload[field_map[key]] = int(value) if key in int_fields else value with open(path, "w", encoding="utf-8") as handle: json.dump(payload, handle, indent=2, sort_keys=True) handle.write("\n") PY rebuild_index "$ORIGIN" cat "$path" } resolve_single_path() { require_arg --origin "$ORIGIN" local path if [ -n "$SESSION_KEY" ]; then path="$(manifest_path "$ORIGIN" "$SESSION_KEY")" [ -f "$path" ] || die "manifest not found for origin/session-key" printf '%s\n' "$path" return 0 fi mapfile -t matches < <(find "$(origin_dir "$ORIGIN")" -maxdepth 1 -type f -name '*.json' 2>/dev/null | sort) case "${#matches[@]}" in 1) printf '%s\n' "${matches[0]}" ;; 0) die "no manifests found for origin" ;; *) die "multiple manifests found; provide --session-key or use select" ;; esac } cmd_show() { local path path="$(resolve_single_path)" cat "$path" } cmd_index_show() { require_arg --origin "$ORIGIN" local file file="$(index_path "$ORIGIN")" if [ ! -f "$file" ]; then rebuild_index "$ORIGIN" fi cat "$file" } cmd_mark_stale() { require_arg --origin "$ORIGIN" require_arg --reason "$REASON" local path path="$(resolve_single_path)" python3 - "$path" <<'PY' import json import os import sys path = sys.argv[1] with open(path, "r", encoding="utf-8") as handle: data = json.load(handle) data["state"] = "stale" data["stale_reason"] = os.environ["REASON"] data["last_verified_at"] = os.environ["NOW"] with open(path, "w", encoding="utf-8") as handle: json.dump(data, handle, indent=2, sort_keys=True) handle.write("\n") PY rebuild_index "$ORIGIN" cat "$path" } filter_candidates() { python3 - <<'PY' import json import os import sys index_file = os.environ["INDEX_FILE"] account_hint = os.environ.get("ACCOUNT_HINT") or "" task_scope = os.environ.get("TASK_SCOPE") or "" with open(index_file, "r", encoding="utf-8") as handle: payload = json.load(handle) candidates = payload.get("sessions", []) if account_hint: candidates = [item for item in candidates if item.get("account_hint") == account_hint] if task_scope: candidates = [item for item in candidates if item.get("task_scope") == task_scope] for item in candidates: print(json.dumps(item, sort_keys=True)) PY } cmd_select() { require_arg --origin "$ORIGIN" local file selected_count line selected_path file="$(index_path "$ORIGIN")" if [ ! -f "$file" ]; then rebuild_index "$ORIGIN" fi mapfile -t selected < <(INDEX_FILE="$file" ACCOUNT_HINT="$ACCOUNT_HINT" TASK_SCOPE="$TASK_SCOPE" filter_candidates) selected_count="${#selected[@]}" if [ "$selected_count" -eq 0 ]; then die "no matching sessions found" fi if [ "$selected_count" -gt 1 ]; then die "multiple matching sessions found; provide --account-hint or --task-scope" fi selected_path="$(python3 - "${selected[0]}" <<'PY' import json import sys print(json.loads(sys.argv[1])["path"]) PY )" cat "$selected_path" } COMMAND="${1:-}" [ -n "$COMMAND" ] || { usage exit 1 } shift || true ROOT_DIR="" ORIGIN="" SESSION_KEY="" ACCOUNT_HINT="" TASK_SCOPE="" STATE="" BROWSER_PID="" REASON="" BLOCK_REASON="" CDP_PORT="" CDP_URL="" WEBSOCKET_DEBUGGER_URL="" TARGET_ID="" PROFILE_DIR="" MODE="" DISPLAY_VALUE="" XVFB_DISPLAY="" XVFB_PID="" NOVNC_PORT="" while [ "$#" -gt 0 ]; do case "$1" in --root) ROOT_DIR="$2" shift 2 ;; --origin) ORIGIN="$2" shift 2 ;; --session-key) SESSION_KEY="$2" shift 2 ;; --account-hint) ACCOUNT_HINT="$2" shift 2 ;; --task-scope) TASK_SCOPE="$2" shift 2 ;; --state) STATE="$2" shift 2 ;; --browser-pid) BROWSER_PID="$2" shift 2 ;; --reason) REASON="$2" shift 2 ;; --block-reason) BLOCK_REASON="$2" shift 2 ;; --cdp-port) CDP_PORT="$2" shift 2 ;; --cdp-url) CDP_URL="$2" shift 2 ;; --websocket-debugger-url) WEBSOCKET_DEBUGGER_URL="$2" shift 2 ;; --target-id) TARGET_ID="$2" shift 2 ;; --profile-dir) PROFILE_DIR="$2" shift 2 ;; --mode) MODE="$2" shift 2 ;; --display) DISPLAY_VALUE="$2" shift 2 ;; --xvfb-display) XVFB_DISPLAY="$2" shift 2 ;; --xvfb-pid) XVFB_PID="$2" shift 2 ;; --novnc-port) NOVNC_PORT="$2" shift 2 ;; -h|--help) usage exit 0 ;; *) die "unknown argument: $1" ;; esac done init_paths NOW="$(timestamp)" CREATED_AT="$NOW" export ORIGIN SESSION_KEY ACCOUNT_HINT TASK_SCOPE STATE BROWSER_PID REASON BLOCK_REASON export CDP_PORT CDP_URL WEBSOCKET_DEBUGGER_URL TARGET_ID PROFILE_DIR MODE DISPLAY_VALUE export XVFB_DISPLAY XVFB_PID NOVNC_PORT NOW CREATED_AT case "$COMMAND" in list) cmd_list ;; path) cmd_path ;; write) cmd_write ;; mark-stale) cmd_mark_stale ;; show) cmd_show ;; index-show) cmd_index_show ;; select) cmd_select ;; *) usage exit 1 ;; esac FILE:scripts/site-session-registry.sh #!/usr/bin/env bash set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck disable=SC1091 source "$SCRIPT_DIR/runtime-common.sh" usage() { cat <<'EOF' Usage: site-session-registry.sh show --site SITE [options] site-session-registry.sh resolve --site SITE [--session-key KEY] [options] site-session-registry.sh write --site SITE --session-key KEY --profile-dir DIR --source-origin URL [options] Options: --profile-dir DIR --root DIR --session-key KEY --site SITE --source-origin URL EOF } die() { printf '[site-session-registry] ERROR: %s\n' "$*" >&2 exit 1 } require_arg() { local name="$1" local value="$2" [ -n "$value" ] || die "missing required argument: $name" } timestamp() { date -u +"%Y-%m-%dT%H:%M:%SZ" } registry_path() { printf '%s/index/site-sessions.json\n' "$ROOT_DIR" } safe_payload() { python3 - "$1" <<'PY' import json import os import sys path = sys.argv[1] payload = {"sites": {}} if os.path.exists(path): try: with open(path, "r", encoding="utf-8") as handle: payload = json.load(handle) except (json.JSONDecodeError, OSError): payload = {"sites": {}} print(json.dumps(payload)) PY } cmd_show() { require_arg --site "$SITE" local path payload path="$(registry_path)" payload="$(safe_payload "$path")" python3 - "$payload" "$SITE" <<'PY' import json import sys payload = json.loads(sys.argv[1]) site = sys.argv[2] entry = payload.get("sites", {}).get(site) if not entry: raise SystemExit(1) print(json.dumps(entry, indent=2, sort_keys=True)) PY } cmd_resolve() { require_arg --site "$SITE" local path payload path="$(registry_path)" payload="$(safe_payload "$path")" python3 - "$payload" "$SITE" "${SESSION_KEY:-}" <<'PY' import json import sys payload = json.loads(sys.argv[1]) site = sys.argv[2] session_key = sys.argv[3] or "" entry = payload.get("sites", {}).get(site) if not entry: raise SystemExit(1) sessions = entry.get("sessions", {}) resolved_key = session_key or entry.get("default_session") or "" session = sessions.get(resolved_key) if not session: raise SystemExit(1) result = dict(session) result["site"] = site result["session_key"] = resolved_key result["default_session"] = entry.get("default_session") print(json.dumps(result, sort_keys=True)) PY } cmd_write() { require_arg --site "$SITE" require_arg --session-key "$SESSION_KEY" require_arg --profile-dir "$PROFILE_DIR" require_arg --source-origin "$SOURCE_ORIGIN" local path now payload path="$(registry_path)" now="$(timestamp)" mkdir -p "$(dirname "$path")" payload="$(safe_payload "$path")" python3 - "$path" "$payload" "$SITE" "$SESSION_KEY" "$PROFILE_DIR" "$SOURCE_ORIGIN" "$now" <<'PY' import json import sys path, payload_raw, site, session_key, profile_dir, source_origin, now = sys.argv[1:] payload = json.loads(payload_raw) sites = payload.setdefault("sites", {}) entry = sites.setdefault(site, {"default_session": session_key, "sessions": {}}) entry.setdefault("default_session", session_key) sessions = entry.setdefault("sessions", {}) sessions[session_key] = { "profile_dir": profile_dir, "source_origin": source_origin, "updated_at": now, } if not entry.get("default_session"): entry["default_session"] = session_key with open(path, "w", encoding="utf-8") as handle: json.dump(payload, handle, indent=2, sort_keys=True) handle.write("\n") PY } COMMAND="${1:-}" [ -n "$COMMAND" ] || { usage exit 1 } shift || true ROOT_DIR="${ROOT_DIR:-$HOME/.agent-browser}" SITE="" SESSION_KEY="" PROFILE_DIR="" SOURCE_ORIGIN="" while [ "$#" -gt 0 ]; do case "$1" in --root) ROOT_DIR="$2" shift 2 ;; --site) SITE="$2" shift 2 ;; --session-key) SESSION_KEY="$2" shift 2 ;; --profile-dir) PROFILE_DIR="$2" shift 2 ;; --source-origin) SOURCE_ORIGIN="$2" shift 2 ;; -h|--help) usage exit 0 ;; *) die "unknown argument: $1" ;; esac done case "$COMMAND" in show) cmd_show ;; resolve) cmd_resolve ;; write) cmd_write ;; *) usage exit 1 ;; esac FILE:scripts/test_assisted_session.sh #!/usr/bin/env bash set -euo pipefail BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT mkdir -p "$TMP_DIR/home" export HOME="$TMP_DIR/home" cat >"$TMP_DIR/runtime-stub.sh" <<'EOF' #!/usr/bin/env bash set -euo pipefail STATE_DIR="${RUNTIME_STUB_DIR:?}" TARGETS_JSON="${RUNTIME_STUB_TARGETS_JSON:-}" PAGE_URL="${RUNTIME_STUB_PAGE_URL:-https://example.com}" PAGE_TITLE="${RUNTIME_STUB_PAGE_TITLE:-Example}" PAGE_BODY="${RUNTIME_STUB_PAGE_BODY:-hello}" LOG_FILE="${RUNTIME_STUB_LOG_FILE:-}" if [ -z "$TARGETS_JSON" ]; then TARGETS_JSON='[{"id":"page-1","type":"page","url":"https://example.com"}]' fi command="${1:-}" shift || true if [ -n "$LOG_FILE" ]; then printf '%s %s\n' "$command" "$*" >>"$LOG_FILE" fi case "$command" in start) touch "$STATE_DIR/running" cat <&2 exit 1 ;; esac EOF chmod +x "$TMP_DIR/runtime-stub.sh" cat >"$TMP_DIR/profile-resolution-stub.sh" <<'EOF' #!/usr/bin/env bash set -euo pipefail case "${1:-}" in resolve) printf '%s\n' "$*" >>"${PROFILE_STUB_LOG:?}" if [ "${PROFILE_STUB_MODE:-ok}" = "ambiguous" ]; then exit 2 fi printf '{"profile_dir":"%s","source":"site-registry"}\n' "${PROFILE_STUB_RESOLVED_PROFILE_DIR:?}" ;; write-identity) provider="" profile_dir="" source_origin="" source_session_key="" while [ "$#" -gt 0 ]; do case "$1" in --provider) provider="$2" shift 2 ;; --profile-dir) profile_dir="$2" shift 2 ;; --source-origin) source_origin="$2" shift 2 ;; --source-session-key) source_session_key="$2" shift 2 ;; *) shift ;; esac done python3 - "${PROFILE_STUB_IDENTITY_FILE:?}" "$provider" "$profile_dir" "$source_origin" "$source_session_key" <<'PY' import json import os import sys path, provider, profile_dir, source_origin, source_session_key = sys.argv[1:] payload = {"providers": {}} if os.path.exists(path): with open(path, "r", encoding="utf-8") as handle: payload = json.load(handle) providers = payload.setdefault("providers", {}) providers[provider] = { "profile_dir": profile_dir, "source_origin": source_origin, "source_session_key": source_session_key, } with open(path, "w", encoding="utf-8") as handle: json.dump(payload, handle, indent=2, sort_keys=True) handle.write("\n") PY ;; *) echo "unexpected profile command: $1" >&2 exit 1 ;; esac EOF chmod +x "$TMP_DIR/profile-resolution-stub.sh" mkdir -p "$TMP_DIR/runtime" RUNTIME_STUB_DIR="$TMP_DIR/runtime" AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ "$BASE_DIR/assisted-session.sh" status --run-dir "$TMP_DIR" >/dev/null PROFILE_STUB_LOG="$TMP_DIR/profile.log" \ PROFILE_STUB_RESOLVED_PROFILE_DIR="$TMP_DIR/resolved-profile" \ RUNTIME_STUB_DIR="$TMP_DIR/runtime" \ RUNTIME_STUB_LOG_FILE="$TMP_DIR/runtime.log" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ "$BASE_DIR/assisted-session.sh" start \ --run-dir "$TMP_DIR/direct-assist" \ --manifest-root "$TMP_DIR/direct-manifests" \ --url "https://github.com/settings/profile" \ --origin "https://github.com" \ --session-key default >/dev/null grep -q '^resolve --root '"$TMP_DIR"'/home/.agent-browser --manifest-root '"$TMP_DIR"'/direct-manifests --origin https://github.com --session-key default$' "$TMP_DIR/profile.log" grep -q '^start --run-dir '"$TMP_DIR"'/home/.agent-browser/run/https___github_com/default --url https://github.com/settings/profile --origin https://github.com --session-key default --mode gui --profile-dir '"$TMP_DIR"'/resolved-profile$' "$TMP_DIR/runtime.log" assist_status="$( RUNTIME_STUB_DIR="$TMP_DIR/runtime" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ "$BASE_DIR/assisted-session.sh" status \ --url "https://foxcode.rjj.cc/api-keys" \ --origin "https://foxcode.rjj.cc" \ --session-key "foxcode-main" )" printf '%s\n' "$assist_status" | grep -q "run_dir: $TMP_DIR/home/.agent-browser/assist/https___foxcode_rjj_cc/foxcode-main" if RUNTIME_STUB_DIR="$TMP_DIR/runtime" AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ "$BASE_DIR/assisted-session.sh" capture --run-dir "$TMP_DIR" --manifest-root "$TMP_DIR/manifests" --origin "https://example.com" >/dev/null 2>&1; then echo "expected capture without a verified browser to fail" exit 1 fi mkdir -p "$TMP_DIR/runtime" touch "$TMP_DIR/runtime/running" "$TMP_DIR/runtime/verified" RUNTIME_STUB_DIR="$TMP_DIR/runtime" AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ "$BASE_DIR/assisted-session.sh" capture --run-dir "$TMP_DIR" --manifest-root "$TMP_DIR/manifests" --origin "https://example.com" --block-reason login-wall >/dev/null "$BASE_DIR/session-manifest.sh" show --root "$TMP_DIR/manifests" --origin "https://example.com" --session-key default | grep -q '"block_reason": "login-wall"' RUNTIME_STUB_TARGETS_JSON='[ {"id":"page-login","type":"page","url":"https://example.com/login"}, {"id":"page-foxcode","type":"page","url":"https://foxcode.rjj.cc/api-keys"}, {"id":"page-other","type":"page","url":"https://news.ycombinator.com"} ]' \ RUNTIME_STUB_PAGE_URL='https://foxcode.rjj.cc/api-keys' \ RUNTIME_STUB_PAGE_TITLE='Foxcode API Keys' \ RUNTIME_STUB_PAGE_BODY='keys' \ RUNTIME_STUB_DIR="$TMP_DIR/runtime" AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ "$BASE_DIR/assisted-session.sh" capture --run-dir "$TMP_DIR" --manifest-root "$TMP_DIR/fox-manifests" --origin "https://foxcode.rjj.cc" >/dev/null "$BASE_DIR/session-manifest.sh" show --root "$TMP_DIR/fox-manifests" --origin "https://foxcode.rjj.cc" --session-key default | grep -q '"target_id": "page-foxcode"' PROFILE_STUB_IDENTITY_FILE="$TMP_DIR/identity-profiles.json" \ RUNTIME_STUB_TARGETS_JSON='[ {"id":"page-github","type":"page","url":"https://github.com/settings/profile"} ]' \ RUNTIME_STUB_PAGE_URL='https://github.com/settings/profile' \ RUNTIME_STUB_PAGE_TITLE='GitHub Settings' \ RUNTIME_STUB_PAGE_BODY='settings' \ RUNTIME_STUB_DIR="$TMP_DIR/runtime" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ "$BASE_DIR/assisted-session.sh" capture --run-dir "$TMP_DIR" --manifest-root "$TMP_DIR/github-manifests" --origin "https://github.com" >/dev/null grep -q '"github.com"' "$TMP_DIR/identity-profiles.json" grep -q '"profile_dir": "'"$TMP_DIR"'/runtime/profile"' "$TMP_DIR/identity-profiles.json" "$BASE_DIR/site-session-registry.sh" resolve --root "$TMP_DIR/home/.agent-browser" --site github.com --session-key default | grep -q '"profile_dir": "'"$TMP_DIR"'/runtime/profile"' cat >"$TMP_DIR/identity-profiles.json" </dev/null grep -q '"source_session_key": "updated"' "$TMP_DIR/identity-profiles.json" rm -f "$TMP_DIR/identity-profiles.json" "$TMP_DIR/runtime/verified" if PROFILE_STUB_IDENTITY_FILE="$TMP_DIR/identity-profiles.json" \ RUNTIME_STUB_DIR="$TMP_DIR/runtime" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ "$BASE_DIR/assisted-session.sh" capture --run-dir "$TMP_DIR" --manifest-root "$TMP_DIR/github-manifests" --origin "https://github.com" >/dev/null 2>&1; then echo "expected capture with login wall to fail before writing identity metadata" exit 1 fi test ! -f "$TMP_DIR/identity-profiles.json" touch "$TMP_DIR/runtime/verified" if RUNTIME_STUB_TARGETS_JSON='[ {"id":"page-google","type":"page","url":"https://myaccount.google.com/"} ]' \ RUNTIME_STUB_PAGE_URL='https://myaccount.google.com/' \ RUNTIME_STUB_PAGE_TITLE='Google Account' \ RUNTIME_STUB_PAGE_BODY='signed in' \ RUNTIME_STUB_DIR="$TMP_DIR/runtime" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ "$BASE_DIR/assisted-session.sh" capture --run-dir "$TMP_DIR" --manifest-root "$TMP_DIR/drift-manifests" --origin "https://github.com" >/dev/null 2>&1; then echo "expected capture on an off-origin page to fail" exit 1 fi if "$BASE_DIR/site-session-registry.sh" resolve --root "$TMP_DIR/home/.agent-browser" --site google.com --session-key default >/dev/null 2>&1; then echo "capture should not register an off-origin site session" exit 1 fi mkdir -p "$TMP_DIR/override-run" cat >"$TMP_DIR/override-run/assist.env" </dev/null "$BASE_DIR/session-manifest.sh" show --root "$TMP_DIR/override-manifests" --origin "https://override.example.com" --session-key override-check | grep -q '"session_key": "override-check"' RUNTIME_STUB_TARGETS_JSON='[ {"id":"page-github","type":"page","url":"https://github.com/settings/profile"} ]' \ RUNTIME_STUB_PAGE_URL='https://github.com/settings/profile' \ RUNTIME_STUB_PAGE_TITLE='GitHub Settings' \ RUNTIME_STUB_PAGE_BODY='settings' \ RUNTIME_STUB_DIR="$TMP_DIR/runtime" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ "$BASE_DIR/assisted-session.sh" capture --run-dir "$TMP_DIR/override-run" --manifest-root "$TMP_DIR/override-manifests" --origin "https://github.com" --session-key root-check >/dev/null test -f "$TMP_DIR/home/.agent-browser/index/identity-profiles.json" test ! -f "$TMP_DIR/override-manifests/index/identity-profiles.json" "$BASE_DIR/site-session-registry.sh" resolve --root "$TMP_DIR/home/.agent-browser" --site github.com --session-key root-check | grep -q '"session_key": "root-check"' mkdir -p "$TMP_DIR/bin" "$TMP_DIR/runtime" "$TMP_DIR/novnc" cat >"$TMP_DIR/bin/x11vnc" <<'EOF' #!/usr/bin/env bash set -euo pipefail printf '%s\n' "$*" >"${ASSIST_STUB_STATE_DIR:?}/x11vnc.args" sleep 30 EOF cat >"$TMP_DIR/bin/websockify" <<'EOF' #!/usr/bin/env bash set -euo pipefail printf '%s\n' "$*" >"${ASSIST_STUB_STATE_DIR:?}/websockify.args" sleep 30 EOF chmod +x "$TMP_DIR/bin/x11vnc" "$TMP_DIR/bin/websockify" python3 -m http.server 5900 --bind 127.0.0.1 >/dev/null 2>&1 & PORT_A_PID=$! python3 -m http.server 6080 --bind 127.0.0.1 >/dev/null 2>&1 & PORT_B_PID=$! trap 'kill "$PORT_A_PID" "$PORT_B_PID" 2>/dev/null || true; rm -rf "$TMP_DIR"' EXIT ASSIST_STUB_STATE_DIR="$TMP_DIR" \ AGENT_BROWSER_NOVNC_WEB_ROOT="$TMP_DIR/novnc" \ PATH="$TMP_DIR/bin:$PATH" \ RUNTIME_STUB_DIR="$TMP_DIR/runtime" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/runtime-stub.sh" \ AGENT_BROWSER_SELECT_TARGET_HELPER="$TMP_DIR/runtime-stub.sh" \ "$BASE_DIR/assisted-session.sh" start --run-dir "$TMP_DIR/assist-run" --url "https://foxcode.rjj.cc/api-keys" >/dev/null grep -Eq '(^| )-rfbport (590[1-9]|59[1-9][0-9]|6[0-9]{3,})( |$)' "$TMP_DIR/x11vnc.args" grep -Eq '0.0.0.0:(608[1-9]|60[89][0-9]|6[1-9][0-9]{2,})' "$TMP_DIR/websockify.args" "$BASE_DIR/assisted-session.sh" stop --run-dir "$TMP_DIR/assist-run" >/dev/null FILE:scripts/test_browser_runtime.sh #!/usr/bin/env bash set -euo pipefail BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT cat >"$TMP_DIR/cdp-stub.py" <<'EOF' #!/usr/bin/env python3 import json import sys args = sys.argv[1:] check = args[args.index("--check") + 1] if check == "challenge": print(json.dumps({"hasChallenge": True, "indicators": ["turnstile"], "title": "Verify you are human", "url": "https://example.com"})) elif check == "login-wall": print(json.dumps({"hasLoginWall": True, "loginHits": ["Sign in"], "title": "Sign in", "url": "https://example.com/login"})) else: print(json.dumps({"title": "Example", "url": "https://example.com", "bodySnippet": "hello"})) EOF chmod +x "$TMP_DIR/cdp-stub.py" status_output="$("$BASE_DIR/browser-runtime.sh" status --run-dir "$TMP_DIR")" printf '%s\n' "$status_output" | grep -q "stopped" mkdir -p "$TMP_DIR/home" isolated_status="$( HOME="$TMP_DIR/home" "$BASE_DIR/browser-runtime.sh" status \ --url "https://foxcode.rjj.cc/api-keys" \ --session-key "foxcode-main" )" printf '%s\n' "$isolated_status" | grep -q "run_dir: $TMP_DIR/home/.agent-browser/run/https___foxcode_rjj_cc/foxcode-main" printf '%s\n' "$isolated_status" | grep -q "profile_dir: $TMP_DIR/home/.agent-browser/profiles/https___foxcode_rjj_cc/foxcode-main" list_output="$("$BASE_DIR/browser-runtime.sh" list-targets --run-dir "$TMP_DIR")" printf '%s\n' "$list_output" | grep -q '^\[\]$' selected_target="$( "$BASE_DIR/browser-runtime.sh" select-target \ --origin "https://foxcode.rjj.cc" \ --target-url "https://foxcode.rjj.cc/api-keys" \ --targets-json '[ {"id":"page-login","type":"page","url":"https://example.com/login"}, {"id":"page-foxcode","type":"page","url":"https://foxcode.rjj.cc/api-keys"} ]' )" printf '%s\n' "$selected_target" | grep -q '^page-foxcode$' if "$BASE_DIR/browser-runtime.sh" attach --run-dir "$TMP_DIR" --origin "https://example.com" --session-key "missing" >/dev/null 2>&1; then echo "expected attach with missing session to fail" exit 1 fi "$BASE_DIR/session-manifest.sh" write \ --root "$TMP_DIR/manifests" \ --origin "https://example.com" \ --session-key "stale" \ --state ready \ --browser-pid 999999 >/dev/null if "$BASE_DIR/browser-runtime.sh" verify --run-dir "$TMP_DIR" --manifest-root "$TMP_DIR/manifests" --origin "https://example.com" --session-key "stale" >/dev/null 2>&1; then echo "expected verify with dead browser to fail" exit 1 fi challenge_output="$(AGENT_BROWSER_CDP_EVAL="$TMP_DIR/cdp-stub.py" \ "$BASE_DIR/browser-runtime.sh" check-page --run-dir "$TMP_DIR" --cdp-port 19222 --target-id TARGET_ID --check challenge)" printf '%s\n' "$challenge_output" | grep -q '"hasChallenge": true' login_output="$(AGENT_BROWSER_CDP_EVAL="$TMP_DIR/cdp-stub.py" \ "$BASE_DIR/browser-runtime.sh" check-page --run-dir "$TMP_DIR" --cdp-port 19222 --target-id TARGET_ID --check login-wall)" printf '%s\n' "$login_output" | grep -q '"hasLoginWall": true' mkdir -p "$TMP_DIR/override-run" cat >"$TMP_DIR/override-run/runtime.env" <"$TMP_DIR/bin/curl" <<'EOF' #!/usr/bin/env bash printf '{"Browser":"stub"}\n' EOF chmod +x "$TMP_DIR/bin/curl" cat >"$TMP_DIR/bin/browser-stub" <<'EOF' #!/usr/bin/env bash sleep 30 EOF chmod +x "$TMP_DIR/bin/browser-stub" ( cd "$TMP_DIR/lock-profile-live" ln -s "stub-$$" SingletonLock ) PATH="$TMP_DIR/bin:$PATH" "$BASE_DIR/browser-runtime.sh" start \ --run-dir "$TMP_DIR/live-run" \ --profile-dir "$TMP_DIR/lock-profile-live" \ --url "https://example.com" \ --origin "https://example.com" \ --session-key "live" \ --mode headless \ --cdp-port 24555 \ --browser "$TMP_DIR/bin/browser-stub" test -L "$TMP_DIR/lock-profile-live/SingletonLock" "$BASE_DIR/browser-runtime.sh" stop --run-dir "$TMP_DIR/live-run" ( cd "$TMP_DIR/lock-profile-stale" ln -s "stub-999999" SingletonLock ) PATH="$TMP_DIR/bin:$PATH" "$BASE_DIR/browser-runtime.sh" start \ --run-dir "$TMP_DIR/stale-run" \ --profile-dir "$TMP_DIR/lock-profile-stale" \ --url "https://example.com" \ --origin "https://example.com" \ --session-key "stale" \ --mode headless \ --cdp-port 24556 \ --browser "$TMP_DIR/bin/browser-stub" test ! -e "$TMP_DIR/lock-profile-stale/SingletonLock" "$BASE_DIR/browser-runtime.sh" stop --run-dir "$TMP_DIR/stale-run" FILE:scripts/test_cdp_eval.sh #!/usr/bin/env bash set -euo pipefail BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" python3 -c "import py_compile; py_compile.compile('$BASE_DIR/cdp-eval.py', doraise=True)" python3 "$BASE_DIR/cdp-eval.py" --help | grep -q "cdp-eval" FILE:scripts/test_cdp_helpers.py #!/usr/bin/env python3 import json import re import socket import subprocess import tempfile from pathlib import Path BASE_DIR = Path(__file__).resolve().parent CDP_EVAL = BASE_DIR / 'cdp-eval.py' CDP_SNAPSHOT = BASE_DIR / 'cdp-snapshot.py' HTML = ''' Forum Example

First Topic

Snippet one

Second Topic

Snippet two

''' def run(*args: str) -> str: return subprocess.check_output(args, text=True).strip() def pick_free_port() -> int: with socket.socket() as sock: sock.bind(('127.0.0.1', 0)) return int(sock.getsockname()[1]) def runtime_port(run_dir: Path) -> int: status = run(str(BASE_DIR / 'browser-runtime.sh'), 'status', '--run-dir', str(run_dir)) match = re.search(r'^cdp_port: (\d+)$', status, re.MULTILINE) assert match, status return int(match.group(1)) def main() -> int: with tempfile.TemporaryDirectory() as td: html_path = Path(td) / 'index.html' html_path.write_text(HTML, encoding='utf-8') profile_dir = Path(td) / 'profile' run_dir = Path(td) / 'run' requested_port = pick_free_port() subprocess.check_call([ str(BASE_DIR / 'browser-runtime.sh'), 'start', '--url', html_path.as_uri(), '--origin', 'https://forum.example', '--profile-dir', str(profile_dir), '--run-dir', str(run_dir), '--mode', 'headless', '--session-key', 'test', '--cdp-port', str(requested_port), ]) try: port = runtime_port(run_dir) assert port == requested_port, port snapshot_raw = run('python3', str(CDP_SNAPSHOT), '--port', str(port), '--format', 'topic-links') snapshot = json.loads(snapshot_raw) links = json.loads(snapshot['content']) assert len(links) == 2, links assert links[0]['text'] == 'First Topic', links assert links[0]['href'] == 'https://forum.example/questions/101', links assert links[0]['topicId'] == '101', links assert 'Snippet one' in links[0]['meta'], links click_raw = run('python3', str(CDP_EVAL), '--port', str(port), '--click-link-text', 'Second Topic') click = json.loads(click_raw) assert click['clicked'] is True, click assert click['href'] == 'https://forum.example/questions/102', click finally: subprocess.call([ str(BASE_DIR / 'browser-runtime.sh'), 'stop', '--run-dir', str(run_dir), ]) print('ok') return 0 if __name__ == '__main__': raise SystemExit(main()) FILE:scripts/test_identity_provider_reuse.sh #!/usr/bin/env bash set -euo pipefail BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT cat >"$TMP_DIR/browser-runtime-stub.sh" <<'EOF' #!/usr/bin/env bash set -euo pipefail STATE_DIR="${RUNTIME_STUB_DIR:?}" TARGET_URL="${RUNTIME_STUB_TARGET_URL:?}" EXPECTED_PROFILE="${RUNTIME_STUB_EXPECTED_PROFILE:-}" record_profile() { local next="" while [ "$#" -gt 0 ]; do case "$1" in --profile-dir) next="$2" shift 2 ;; *) shift ;; esac done printf '%s\n' "$next" >"$STATE_DIR/profile-dir" if [ -n "$EXPECTED_PROFILE" ] && [ "$next" = "$EXPECTED_PROFILE" ]; then printf '%s\n' "mapped" >"$STATE_DIR/mode" else printf '%s\n' "fresh" >"$STATE_DIR/mode" fi } mode_value() { if [ -f "$STATE_DIR/mode" ]; then cat "$STATE_DIR/mode" else printf '%s\n' "fresh" fi } case "${1:-}" in verify) exit 1 ;; status) cat <&2 exit 1 ;; esac EOF cat >"$TMP_DIR/assisted-stub.sh" <<'EOF' #!/usr/bin/env bash set -euo pipefail case "${1:-}" in start) echo '[assisted-session] noVNC URL: http://127.0.0.1:6081/vnc.html?autoconnect=1&resize=remote' ;; status) cat <&2 exit 1 ;; esac EOF chmod +x "$TMP_DIR/browser-runtime-stub.sh" "$TMP_DIR/assisted-stub.sh" mkdir -p "$TMP_DIR/home/.agent-browser/index" "$TMP_DIR/runtime" cat >"$TMP_DIR/home/.agent-browser/index/identity-profiles.json" <"$TMP_DIR/home/.agent-browser/index/identity-profiles.json" <"$TMP_DIR/home/.agent-browser/index/identity-profiles.json" rm -f "$TMP_DIR/runtime/profile-dir" "$TMP_DIR/runtime/mode" corrupt_output="$( HOME="$TMP_DIR/home" \ RUNTIME_STUB_DIR="$TMP_DIR/runtime" \ RUNTIME_STUB_TARGET_URL="https://github.com/settings/profile" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/browser-runtime-stub.sh" \ AGENT_BROWSER_ASSISTED_HELPER="$TMP_DIR/assisted-stub.sh" \ "$BASE_DIR/open-protected-page.sh" \ --url 'https://github.com/settings/profile' \ --origin 'https://github.com' \ --session-key default )" printf '%s\n' "$corrupt_output" | grep -q '"status": "needs-user"' printf '%s\n' "$corrupt_output" | grep -q '"reason": "login-wall"' FILE:scripts/test_open_protected_page.sh #!/usr/bin/env bash set -euo pipefail BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT cat >"$TMP_DIR/browser-runtime-stub.sh" <<'EOF' #!/usr/bin/env bash set -euo pipefail MODE="${RUNTIME_STUB_MODE:-ready}" STATE_FILE="${RUNTIME_STUB_STATE_FILE:-}" LOG_FILE="${RUNTIME_STUB_LOG_FILE:-}" if [ -n "$LOG_FILE" ]; then printf '%s\n' "$*" >>"$LOG_FILE" fi mode_value() { if { [ "$MODE" = "recover-mismatch" ] || [ "$MODE" = "recover-wrong-login" ]; } && [ -n "$STATE_FILE" ] && [ -f "$STATE_FILE" ]; then cat "$STATE_FILE" return 0 fi printf '%s\n' "$MODE" } case "${1:-}" in verify) exit 1 ;; status) if [ "$MODE" = "verify-fails-running" ] || { { [ "$MODE" = "recover-mismatch" ] || [ "$MODE" = "recover-wrong-login" ]; } && [ "$(mode_value)" != "stopped" ]; }; then cat <&2 exit 1 fi if { [ "$MODE" = "recover-mismatch" ] || [ "$MODE" = "recover-wrong-login" ]; } && [ -n "$STATE_FILE" ]; then printf '%s\n' "recovered" >"$STATE_FILE" fi echo "runtime started" ;; stop) if { [ "$MODE" = "recover-mismatch" ] || [ "$MODE" = "recover-wrong-login" ]; } && [ -n "$STATE_FILE" ]; then printf '%s\n' "stopped" >"$STATE_FILE" fi echo "runtime stopped" ;; list-targets) if [ "$MODE" = "recover-mismatch" ] && [ "$(mode_value)" = "recovered" ]; then echo '[{"id":"page-foxcode","type":"page","url":"https://foxcode.rjj.cc/api-keys"}]' else echo '[{"id":"page-foxcode","type":"page","url":"https://foxcode.rjj.cc/api-keys"}]' fi ;; select-target) echo "page-foxcode" ;; check-page) check="" while [ "$#" -gt 0 ]; do case "$1" in --check) check="$2" shift 2 ;; *) shift ;; esac done case "$(mode_value):$check" in ready:challenge|login-wall:challenge) echo '{"hasChallenge":false}' ;; transient-login:challenge) echo '{"hasChallenge":false}' ;; recover-mismatch:challenge) echo '{"hasChallenge":false}' ;; recover-wrong-login:challenge) echo '{"hasChallenge":false}' ;; ready:login-wall) echo '{"hasLoginWall":false}' ;; login-wall:login-wall) echo '{"hasLoginWall":true}' ;; transient-login:login-wall) if [ -n "$STATE_FILE" ]; then count=0 if [ -f "$STATE_FILE" ]; then count="$(cat "$STATE_FILE")" fi count=$((count + 1)) printf '%s\n' "$count" >"$STATE_FILE" if [ "$count" -ge 2 ]; then echo '{"hasLoginWall":true}' else echo '{"hasLoginWall":false}' fi else echo '{"hasLoginWall":false}' fi ;; recover-mismatch:login-wall) echo '{"hasLoginWall":false}' ;; recover-wrong-login:login-wall) if [ "$(mode_value)" = "recovered" ]; then echo '{"hasLoginWall":false}' else echo '{"hasLoginWall":true}' fi ;; *:page-info) if [ "$MODE" = "transient-login" ] && [ -n "$STATE_FILE" ] && [ -f "$STATE_FILE" ] && [ "$(cat "$STATE_FILE")" -lt 2 ]; then echo '{"title":"","url":"about:blank","bodySnippet":""}' elif [ "$(mode_value)" = "wrong-page" ] || [ "$(mode_value)" = "recover-mismatch" ]; then echo '{"title":"Google Account","url":"https://myaccount.google.com/","bodySnippet":"signed in"}' elif [ "$(mode_value)" = "same-origin-wrong-page" ]; then echo '{"title":"Foxcode Settings","url":"https://foxcode.rjj.cc/settings","bodySnippet":"settings"}' elif [ "$(mode_value)" = "recover-wrong-login" ]; then if [ -n "$STATE_FILE" ] && [ -f "$STATE_FILE" ] && [ "$(cat "$STATE_FILE")" = "recovered" ]; then echo '{"title":"API密钥管理 - NEW CLI","url":"https://foxcode.rjj.cc/api-keys","bodySnippet":"余额基数"}' else echo '{"title":"Sign in to GitHub · GitHub","url":"https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsettings%2Fprofile","bodySnippet":"Sign in to GitHub"}' fi else echo '{"title":"API密钥管理 - NEW CLI","url":"https://foxcode.rjj.cc/api-keys","bodySnippet":"余额基数"}' fi ;; *) echo '{"hasChallenge":false,"hasLoginWall":false}' ;; esac ;; *) echo "unexpected runtime command: $1" >&2 exit 1 ;; esac EOF cat >"$TMP_DIR/assisted-stub.sh" <<'EOF' #!/usr/bin/env bash set -euo pipefail printf '%s\n' "$*" >>"${ASSISTED_STUB_LOG:?}" case "${1:-}" in start) echo '[assisted-session] noVNC URL: http://127.0.0.1:6081/vnc.html?autoconnect=1&resize=remote' ;; status) cat <&2 exit 1 ;; esac EOF chmod +x "$TMP_DIR/browser-runtime-stub.sh" "$TMP_DIR/assisted-stub.sh" cat >"$TMP_DIR/profile-resolution-stub.sh" <<'EOF' #!/usr/bin/env bash set -euo pipefail printf '%s\n' "$*" >>"${PROFILE_STUB_LOG:?}" case "${1:-}" in resolve) if [ "${PROFILE_STUB_MODE:-ok}" = "ambiguous" ]; then exit 2 fi printf '{"profile_dir":"%s","source":"identity-index"}\n' "${PROFILE_STUB_PROFILE_DIR:?}" ;; *) echo "unexpected profile command: $1" >&2 exit 1 ;; esac EOF chmod +x "$TMP_DIR/profile-resolution-stub.sh" ready_output="$( AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/browser-runtime-stub.sh" \ AGENT_BROWSER_ASSISTED_HELPER="$TMP_DIR/assisted-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ ASSISTED_STUB_LOG="$TMP_DIR/assisted.log" \ PROFILE_STUB_LOG="$TMP_DIR/profile.log" \ PROFILE_STUB_PROFILE_DIR="/tmp/resolved-profile" \ RUNTIME_STUB_LOG_FILE="$TMP_DIR/runtime.log" \ "$BASE_DIR/open-protected-page.sh" \ --url 'https://foxcode.rjj.cc/api-keys' \ --origin 'https://foxcode.rjj.cc' \ --session-key foxcode-main )" printf '%s\n' "$ready_output" | grep -q '"status": "ready"' printf '%s\n' "$ready_output" | grep -q '"targetId": "page-foxcode"' if [ -f "$TMP_DIR/assisted.log" ]; then echo "assisted flow should not start for ready pages" exit 1 fi login_output="$( RUNTIME_STUB_MODE="login-wall" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/browser-runtime-stub.sh" \ AGENT_BROWSER_ASSISTED_HELPER="$TMP_DIR/assisted-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ ASSISTED_STUB_LOG="$TMP_DIR/assisted.log" \ PROFILE_STUB_LOG="$TMP_DIR/profile.log" \ PROFILE_STUB_PROFILE_DIR="/tmp/resolved-profile" \ RUNTIME_STUB_LOG_FILE="$TMP_DIR/runtime.log" \ "$BASE_DIR/open-protected-page.sh" \ --url 'https://foxcode.rjj.cc/api-keys' \ --origin 'https://foxcode.rjj.cc' \ --session-key foxcode-main )" printf '%s\n' "$login_output" | grep -q '"status": "needs-user"' printf '%s\n' "$login_output" | grep -q 'http://127.0.0.1:6081/vnc.html' printf '%s\n' "$login_output" | grep -q '"lanNovncUrl": "http://192.168.0.200:6081/vnc.html?autoconnect=1&resize=remote"' grep -q '^start ' "$TMP_DIR/assisted.log" rm -f "$TMP_DIR/assisted.log" "$TMP_DIR/runtime-state" transient_output="$( RUNTIME_STUB_MODE="transient-login" \ RUNTIME_STUB_STATE_FILE="$TMP_DIR/runtime-state" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/browser-runtime-stub.sh" \ AGENT_BROWSER_ASSISTED_HELPER="$TMP_DIR/assisted-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ ASSISTED_STUB_LOG="$TMP_DIR/assisted.log" \ PROFILE_STUB_LOG="$TMP_DIR/profile.log" \ PROFILE_STUB_PROFILE_DIR="/tmp/resolved-profile" \ RUNTIME_STUB_LOG_FILE="$TMP_DIR/runtime.log" \ "$BASE_DIR/open-protected-page.sh" \ --url 'https://foxcode.rjj.cc/api-keys' \ --origin 'https://foxcode.rjj.cc' \ --session-key foxcode-main )" printf '%s\n' "$transient_output" | grep -q '"status": "needs-user"' printf '%s\n' "$transient_output" | grep -q '"reason": "login-wall"' wrong_page_output="$( RUNTIME_STUB_MODE="wrong-page" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/browser-runtime-stub.sh" \ AGENT_BROWSER_ASSISTED_HELPER="$TMP_DIR/assisted-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ ASSISTED_STUB_LOG="$TMP_DIR/assisted.log" \ PROFILE_STUB_LOG="$TMP_DIR/profile.log" \ PROFILE_STUB_PROFILE_DIR="/tmp/resolved-profile" \ RUNTIME_STUB_LOG_FILE="$TMP_DIR/runtime.log" \ "$BASE_DIR/open-protected-page.sh" \ --url 'https://foxcode.rjj.cc/api-keys' \ --origin 'https://foxcode.rjj.cc' \ --session-key foxcode-main )" printf '%s\n' "$wrong_page_output" | grep -q '"status": "needs-user"' same_origin_wrong_page_output="$( RUNTIME_STUB_MODE="same-origin-wrong-page" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/browser-runtime-stub.sh" \ AGENT_BROWSER_ASSISTED_HELPER="$TMP_DIR/assisted-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ ASSISTED_STUB_LOG="$TMP_DIR/assisted.log" \ PROFILE_STUB_LOG="$TMP_DIR/profile.log" \ PROFILE_STUB_PROFILE_DIR="/tmp/resolved-profile" \ RUNTIME_STUB_LOG_FILE="$TMP_DIR/runtime.log" \ "$BASE_DIR/open-protected-page.sh" \ --url 'https://foxcode.rjj.cc/api-keys' \ --origin 'https://foxcode.rjj.cc' \ --session-key foxcode-main )" printf '%s\n' "$same_origin_wrong_page_output" | grep -q '"status": "needs-user"' printf '%s\n' "$same_origin_wrong_page_output" | grep -q '"reason": "target-mismatch"' recover_output="$( RUNTIME_STUB_MODE="recover-mismatch" \ RUNTIME_STUB_STATE_FILE="$TMP_DIR/recover-state" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/browser-runtime-stub.sh" \ AGENT_BROWSER_ASSISTED_HELPER="$TMP_DIR/assisted-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ ASSISTED_STUB_LOG="$TMP_DIR/assisted.log" \ PROFILE_STUB_LOG="$TMP_DIR/profile.log" \ PROFILE_STUB_PROFILE_DIR="/tmp/resolved-profile" \ RUNTIME_STUB_LOG_FILE="$TMP_DIR/runtime.log" \ "$BASE_DIR/open-protected-page.sh" \ --url 'https://foxcode.rjj.cc/api-keys' \ --origin 'https://foxcode.rjj.cc' \ --session-key foxcode-main )" printf '%s\n' "$recover_output" | grep -q '"status": "ready"' grep -q '^stop --origin https://foxcode.rjj.cc --session-key foxcode-main$' "$TMP_DIR/runtime.log" recover_login_output="$( RUNTIME_STUB_MODE="recover-wrong-login" \ RUNTIME_STUB_STATE_FILE="$TMP_DIR/recover-login-state" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/browser-runtime-stub.sh" \ AGENT_BROWSER_ASSISTED_HELPER="$TMP_DIR/assisted-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ ASSISTED_STUB_LOG="$TMP_DIR/assisted.log" \ PROFILE_STUB_LOG="$TMP_DIR/profile.log" \ PROFILE_STUB_PROFILE_DIR="/tmp/resolved-profile" \ RUNTIME_STUB_LOG_FILE="$TMP_DIR/runtime.log" \ "$BASE_DIR/open-protected-page.sh" \ --url 'https://foxcode.rjj.cc/api-keys' \ --origin 'https://foxcode.rjj.cc' \ --session-key foxcode-main )" printf '%s\n' "$recover_login_output" | grep -q '"status": "ready"' running_output="$( RUNTIME_STUB_MODE="verify-fails-running" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/browser-runtime-stub.sh" \ AGENT_BROWSER_ASSISTED_HELPER="$TMP_DIR/assisted-stub.sh" \ AGENT_BROWSER_PROFILE_HELPER="$TMP_DIR/profile-resolution-stub.sh" \ ASSISTED_STUB_LOG="$TMP_DIR/assisted.log" \ PROFILE_STUB_LOG="$TMP_DIR/profile.log" \ PROFILE_STUB_PROFILE_DIR="/tmp/resolved-profile" \ RUNTIME_STUB_LOG_FILE="$TMP_DIR/runtime.log" \ "$BASE_DIR/open-protected-page.sh" \ --url 'https://foxcode.rjj.cc/api-keys' \ --origin 'https://foxcode.rjj.cc' \ --session-key foxcode-main )" printf '%s\n' "$running_output" | grep -q '"status": "ready"' grep -q '^resolve ' "$TMP_DIR/profile.log" grep -q '^start --url https://foxcode.rjj.cc/api-keys --origin https://foxcode.rjj.cc --session-key foxcode-main --profile-dir /tmp/resolved-profile --mode gui$' "$TMP_DIR/runtime.log" mkdir -p "$TMP_DIR/home/.agent-browser/profiles/foxcode.rjj.cc/Default" mkdir -p "$TMP_DIR/home/.agent-browser/profiles/https___foxcode_rjj_cc/foxcode-main/Default" touch "$TMP_DIR/home/.agent-browser/profiles/foxcode.rjj.cc/Default/Cookies" touch "$TMP_DIR/home/.agent-browser/profiles/https___foxcode_rjj_cc/foxcode-main/Default/Login Data" ambiguous_output="$( HOME="$TMP_DIR/home" \ AGENT_BROWSER_RUNTIME_HELPER="$TMP_DIR/browser-runtime-stub.sh" \ AGENT_BROWSER_ASSISTED_HELPER="$TMP_DIR/assisted-stub.sh" \ ASSISTED_STUB_LOG="$TMP_DIR/assisted.log" \ RUNTIME_STUB_LOG_FILE="$TMP_DIR/runtime.log" \ "$BASE_DIR/open-protected-page.sh" \ --url 'https://foxcode.rjj.cc/api-keys' \ --origin 'https://foxcode.rjj.cc' \ --session-key foxcode-main )" printf '%s\n' "$ambiguous_output" | grep -q '"reason": "ambiguous-profile"' FILE:scripts/test_profile_resolution.sh #!/usr/bin/env bash set -euo pipefail BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT mkdir -p "$TMP_DIR/home/.agent-browser/profiles/x.com/Default" touch "$TMP_DIR/home/.agent-browser/profiles/x.com/Default/Cookies" "$BASE_DIR/site-session-registry.sh" write \ --root "$TMP_DIR/home/.agent-browser" \ --site "github.com" \ --session-key default \ --profile-dir "$TMP_DIR/home/.agent-browser/profiles/sites/github/default" \ --source-origin "https://github.com" >/dev/null registry_status="$( HOME="$TMP_DIR/home" \ "$BASE_DIR/browser-runtime.sh" status \ --origin "https://github.com" \ --session-key default )" printf '%s\n' "$registry_status" | grep -q "profile_dir: $TMP_DIR/home/.agent-browser/profiles/sites/github/default" "$BASE_DIR/session-manifest.sh" write \ --root "$TMP_DIR/manifests" \ --origin "https://x.com" \ --session-key default \ --state ready \ --browser-pid 123 \ --profile-dir "$TMP_DIR/home/.agent-browser/profiles/x.com" >/dev/null status_output="$( HOME="$TMP_DIR/home" \ "$BASE_DIR/browser-runtime.sh" status \ --manifest-root "$TMP_DIR/manifests" \ --origin "https://x.com" \ --session-key default )" printf '%s\n' "$status_output" | grep -q "profile_dir: $TMP_DIR/home/.agent-browser/profiles/x.com" legacy_status="$( HOME="$TMP_DIR/home" \ "$BASE_DIR/browser-runtime.sh" status \ --origin "https://x.com" \ --session-key default )" printf '%s\n' "$legacy_status" | grep -q "profile_dir: $TMP_DIR/home/.agent-browser/profiles/x.com" mkdir -p "$TMP_DIR/home/.agent-browser/profiles/https___x_com/default/Default" touch "$TMP_DIR/home/.agent-browser/profiles/https___x_com/default/Default/Login Data" touch "$TMP_DIR/home/.agent-browser/profiles/https___x_com/default/Local State" populated_status="$( HOME="$TMP_DIR/home" \ "$BASE_DIR/browser-runtime.sh" status \ --origin "https://x.com" \ --session-key default )" printf '%s\n' "$populated_status" | grep -q "profile_dir: $TMP_DIR/home/.agent-browser/profiles/https___x_com/default" runtime_key="$( HOME="$TMP_DIR/home" \ "$BASE_DIR/browser-runtime.sh" status \ --origin "HTTPS://X.COM" \ --session-key default | sed -n 's#.*run_dir: .*/run/$[^/]*$/.*#\1#p' )" manifest_key="$( "$BASE_DIR/session-manifest.sh" path \ --root "$TMP_DIR/manifests" \ --origin "HTTPS://X.COM" | sed -n 's#.*/sessions/$[^/]*$$#\1#p' )" [ "$runtime_key" = "$manifest_key" ] mkdir -p "$TMP_DIR/home/.agent-browser/index" printf '{invalid json\n' >"$TMP_DIR/home/.agent-browser/index/identity-profiles.json" write_output="$( HOME="$TMP_DIR/home" \ "$BASE_DIR/profile-resolution.sh" write-identity \ --root "$TMP_DIR/home/.agent-browser" \ --provider "github.com" \ --profile-dir "$TMP_DIR/home/.agent-browser/profiles/x.com" \ --source-origin "https://x.com" \ --source-session-key default )" [ -z "$write_output" ] grep -q '"github.com"' "$TMP_DIR/home/.agent-browser/index/identity-profiles.json" FILE:scripts/test_runtime_common.sh #!/usr/bin/env bash set -euo pipefail BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT python3 -m http.server 5900 --bind 127.0.0.1 >/dev/null 2>&1 & PORT_A_PID=$! python3 -m http.server 6080 --bind 127.0.0.1 >/dev/null 2>&1 & PORT_B_PID=$! trap 'kill "$PORT_A_PID" "$PORT_B_PID" 2>/dev/null || true; rm -rf "$TMP_DIR"' EXIT mkdir -p "$TMP_DIR/x11" touch "$TMP_DIR/x11/X88" # shellcheck disable=SC1091 source "$BASE_DIR/runtime-common.sh" slug="$(origin_slug 'https://foxcode.rjj.cc')" [ "$slug" = "https___foxcode_rjj_cc" ] [ "$(derive_origin 'https://foxcode.rjj.cc/api-keys')" = "https://foxcode.rjj.cc" ] [ "$(site_key 'https://github.com/settings/profile')" = "github.com" ] [ "$(site_key 'https://myaccount.google.com/')" = "google.com" ] [ "$(site_key 'https://accounts.google.com/')" = "google.com" ] [ "$(provider_aliases 'https://myaccount.google.com/' | tr '\n' ' ')" = "myaccount.google.com accounts.google.com google.com " ] [ "$(provider_aliases 'https://github.com/settings/profile' | tr '\n' ' ')" = "github.com " ] [ "$(AGENT_BROWSER_NOVNC_PUBLIC_HOST='192.168.0.200' primary_ipv4)" = "192.168.0.200" ] [ "$(lan_novnc_url '192.168.0.200' '6084')" = "http://192.168.0.200:6084/vnc.html?autoconnect=1&resize=remote" ] vnc_port="$(pick_free_tcp_port 5900)" novnc_port="$(pick_free_tcp_port 6080)" display_num="$(AGENT_BROWSER_X11_SOCKET_DIR="$TMP_DIR/x11" pick_free_display 88)" [ "$vnc_port" != "5900" ] [ "$novnc_port" != "6080" ] [ "$display_num" != "88" ] FILE:scripts/test_session_manifest.sh #!/usr/bin/env bash set -euo pipefail BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT "$BASE_DIR/session-manifest.sh" list --root "$TMP_DIR" >/dev/null "$BASE_DIR/session-manifest.sh" write \ --root "$TMP_DIR" \ --origin "https://example.com" \ --session-key "acct-a-main" \ --account-hint "acct-a" \ --state ready \ --browser-pid 123 >/dev/null "$BASE_DIR/session-manifest.sh" write \ --root "$TMP_DIR" \ --origin "https://example.com" \ --session-key "acct-b-main" \ --account-hint "acct-b" \ --state ready \ --browser-pid 456 >/dev/null test -f "$TMP_DIR/sessions/https___example_com/acct-a-main.json" "$BASE_DIR/session-manifest.sh" index-show --root "$TMP_DIR" --origin "https://example.com" | grep -q "acct-a-main" "$BASE_DIR/session-manifest.sh" select --root "$TMP_DIR" --origin "https://example.com" --account-hint "acct-a" | grep -q "acct-a-main" if "$BASE_DIR/session-manifest.sh" select --root "$TMP_DIR" --origin "https://example.com" >/dev/null 2>&1; then echo "expected ambiguous selection to fail" exit 1 fi FILE:scripts/test_site_session_registry.sh #!/usr/bin/env bash set -euo pipefail BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT "$BASE_DIR/site-session-registry.sh" write \ --root "$TMP_DIR/root" \ --site github.com \ --session-key default \ --profile-dir "$TMP_DIR/profiles/github-default" \ --source-origin "https://github.com" >/dev/null "$BASE_DIR/site-session-registry.sh" resolve \ --root "$TMP_DIR/root" \ --site github.com \ --session-key default | grep -q '"profile_dir": "'"$TMP_DIR"'/profiles/github-default"' "$BASE_DIR/site-session-registry.sh" show \ --root "$TMP_DIR/root" \ --site github.com | grep -q '"default_session": "default"' printf '{broken json\n' >"$TMP_DIR/root/index/site-sessions.json" "$BASE_DIR/site-session-registry.sh" write \ --root "$TMP_DIR/root" \ --site google.com \ --session-key default \ --profile-dir "$TMP_DIR/profiles/google-default" \ --source-origin "https://myaccount.google.com/" >/dev/null "$BASE_DIR/site-session-registry.sh" resolve \ --root "$TMP_DIR/root" \ --site google.com \ --session-key default | grep -q '"source_origin": "https://myaccount.google.com/"' FILE:scripts/test_skill_scope.sh #!/usr/bin/env bash set -euo pipefail BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SKILL_DIR="$(cd "$BASE_DIR/.." && pwd)" test ! -e "$BASE_DIR/protected-page-followup.sh" test ! -e "$BASE_DIR/test_protected_page_followup.sh" ! rg -n "Ready-Page Follow-Up|only valid follow-up context|rediscover a tab|protected-page-followup\\.sh" "$SKILL_DIR/SKILL.md" >/dev/null ! rg -n 'Wrapper Already Returned `targetId`|protected-page-followup\.sh' "$SKILL_DIR/references/manual-fallback.md" >/dev/null

Ubuntu Browser Session

Comments (0)