E2E manual testing of PRs/branches using docker compose, agent-browser, and API calls. TRIGGER when user asks to manually test a PR, test a feature end-to-end,
复制下面这句话，粘贴给 Claude Code、Codex、Cursor 等 AI 编程工具，它会读取安装说明并在你确认后完成安装。
请阅读 https://ai.atlankj.com/install/asset/gh-pr-test-2fada3752f52 ，按照其中的说明把「pr-test」安装到你（当前 AI 工具）中。执行前先告诉我将运行的命令和写入的位置，等我确认。
查看 AI 将读取的安装说明正在读取 GitHub 原文…
内容来自 GitHub 原始文件，由原作者维护。在 GitHub 查看
Test a PR/branch end-to-end by building the full platform, interacting via browser and API, capturing screenshots, and reporting results.
Changelog 2.2.0 — auth flow updated for Better Auth (Supabase signup is
gone), env-setup gaps closed, a proven Playwright fallback for agent-browser,
a billing-test trap that produces false passes, safer process cleanup, and a
mock-provider pattern for deterministic $0 testing. Learned on
#14206 — see the
evidence comment.
2.2.1 — corrects two claims from 2.2.0 that didn't survive live-stack
verification (JWKS does not rotate on a frontend restart; the local
Postgres port is 5432, not 54322) and hardens the auth setup (explicit
password-length/allowlist/rate-limit failure modes, fail-fast on an empty
token, password kept out of process args).
2.3.0 — screenshots are posted as GitHub comment attachments
(gh pr comment --attach, needs gh >= 2.99.0) instead of being pushed to a
test-screenshots/* branch, which accumulated one branch per tested PR and
made every old report's images depend on that branch surviving.
These are NON-NEGOTIABLE. Every test run MUST satisfy ALL the following:
{NN}-{action}-{state}.png (e.g., 01-credits-before.png, 02-credits-after.png)gh pr comment --attach (see Step 7). Never push image files to a repo branch — attachments live in GitHub's own asset store and leave nothing behind in the repocredits_before=100, credits_after=95)Each test scenario in the report MUST have:
Billing-test trap — this one produces a false pass, not a visible failure.
LLM block cost filters key on the platform-owned credential id. A run made
with the test user's own API key bills nothing by design, so a credits
before/after assertion silently passes on zero deltas either way. Any test
that verifies credit reconciliation MUST use the system credential, not a
user-supplied key. Related: Ollama block entries are configured with an
explicit $0 run-based cost (BlockCostType.RUN, cost_amount=0 in
block_cost_config.py), not a token-metered one — so pre/post-flight cost
deltas are always 0 for them regardless of credential. Never use an Ollama
model to test credit reconciliation; pick any hosted model billed through the
system credential instead.
When testing features that depend on specific states (rate limits, credits, quotas):
Use Redis CLI to set counters directly:
# Find the Redis container
REDIS_CONTAINER=$(docker ps --format '{{.Names}}' | grep redis | head -1)
# Set a key with expiry
docker exec $REDIS_CONTAINER redis-cli SET key value EX ttl
# Example: Set rate limit counter to near-limit
docker exec $REDIS_CONTAINER redis-cli SET "rate_limit:user:$PR_TEST_USER_EMAIL" 99 EX 3600
# Example: Check current value
docker exec $REDIS_CONTAINER redis-cli GET "rate_limit:user:$PR_TEST_USER_EMAIL"
Use API calls to check before/after state:
# BEFORE: Record current state
BEFORE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
echo "Credits BEFORE: $BEFORE"
# Perform the action...
# AFTER: Record new state and compare
AFTER=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
echo "Credits AFTER: $AFTER"
echo "Delta: $(( BEFORE - AFTER ))"
Take screenshots BEFORE and AFTER state changes — the UI must reflect the backend state change
Never rely on mocked/injected browser state — always use real backend state. Do NOT use agent-browser eval to fake UI state. The backend must be the source of truth.
Use direct DB queries when needed:
# Query via Supabase's PostgREST or docker exec into the DB
docker exec supabase-db psql -U supabase_admin -d postgres -c "SELECT credits FROM user_credits WHERE user_id = '...';"
After every API test, verify the state change actually persisted:
# Example: After a credits purchase, verify DB matches API
API_CREDITS=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
DB_CREDITS=$(docker exec supabase-db psql -U supabase_admin -d postgres -t -c "SELECT credits FROM user_credits WHERE user_id = '...';" | tr -d ' ')
[ "$API_CREDITS" = "$DB_CREDITS" ] && echo "CONSISTENT" || echo "MISMATCH: API=$API_CREDITS DB=$DB_CREDITS"
$ARGUMENTS — worktree path (e.g. $REPO_ROOT) or PR number--fix flag is present, auto-fix bugs found and push fixes (like pr-address loop)# If argument is a PR number, find its worktree
gh pr view {N} --json headRefName --jq '.headRefName'
# If argument is a path, use it directly
Determine:
REPO_ROOT — the root repo directory: git -C "$WORKTREE_PATH" worktree list | head -1 | awk '{print $1}' (or git rev-parse --show-toplevel if not a worktree)WORKTREE_PATH — the worktree directoryPLATFORM_DIR — $WORKTREE_PATH/autogpt_platformBACKEND_DIR — $PLATFORM_DIR/backendFRONTEND_DIR — $PLATFORM_DIR/frontendPR_NUMBER — the PR number (from gh pr list --head $(git branch --show-current))PR_TITLE — the PR title, slugified (e.g. "Add copilot permissions" → "add-copilot-permissions")RESULTS_DIR — $REPO_ROOT/test-results/PR-{PR_NUMBER}-{slugified-title}Create the results directory:
PR_NUMBER=$(cd $WORKTREE_PATH && gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT --json number --jq '.[0].number')
PR_TITLE=$(cd $WORKTREE_PATH && gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT --json title --jq '.[0].title' | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//' | head -c 50)
RESULTS_DIR="$REPO_ROOT/test-results/PR-${PR_NUMBER}-${PR_TITLE}"
mkdir -p $RESULTS_DIR
Test user credentials — required to log into the UI or call authenticated APIs.
NEVER hardcode these in this SKILL, a PR comment, a screenshot, or any committed file. Sources, in priority order:
$PR_TEST_USER_EMAIL + $PR_TEST_USER_PASSWORD. If both are set, use them.Acquire the variables — env first, prompt if missing — and only then lock them in:
# 1. Prefer env vars (CI / preconfigured shell). Prompt only for the
# specific var that is unset so an already-exported credential is
# not overwritten by the prompt when only the other one is missing.
if [ -z "${PR_TEST_USER_EMAIL:-}" ] || [ -z "${PR_TEST_USER_PASSWORD:-}" ]; then
echo "Test user credentials required for this run."
if [ -z "${PR_TEST_USER_EMAIL:-}" ]; then
read -r -p "Email: " PR_TEST_USER_EMAIL
fi
if [ -z "${PR_TEST_USER_PASSWORD:-}" ]; then
read -r -s -p "Password: " PR_TEST_USER_PASSWORD
echo
fi
export PR_TEST_USER_EMAIL PR_TEST_USER_PASSWORD
fi
# 2. Lock them in — fail loudly if either is STILL unset (e.g. the user
# pressed Enter on an empty prompt). The error message names the var so
# the agent / operator knows what to fix.
: "${PR_TEST_USER_EMAIL:?PR_TEST_USER_EMAIL is empty after env+prompt — supply a value before re-running}"
: "${PR_TEST_USER_PASSWORD:?PR_TEST_USER_PASSWORD is empty after env+prompt — supply a value before re-running}"
For local docker-compose runs, a fresh dev user is created on first call to the signup snippet below. For dev-preview runs, the test user lives in the project's hosted auth backend — ask the user for the current valid credentials each session (the previously-shared test@test.com test account was disabled on 2026-05-23 after its credentials leaked into this very SKILL — do NOT re-introduce a default). PR_TEST_USER_PASSWORD should always be a throwaway/test-only credential, never a real account's password — the auth requests in 3h go over plain HTTP on localhost:3000 for local runs, which has no transport encryption. If a dev-preview run's target isn't on localhost, confirm it's HTTPS before sending credentials to it.
Before testing, understand what changed:
cd $WORKTREE_PATH
# Read PR description to understand the WHY
gh pr view {N} --json body --jq '.body'
git log --oneline dev..HEAD | head -20
git diff dev --stat
Read the PR description (Why / What / How) and changed files to understand: 0. Why does this PR exist? What problem does it solve?
Based on the PR analysis, write a test plan to $RESULTS_DIR/test-plan.md:
# Test Plan: PR #{N} — {title}
## Scenarios
1. [Scenario name] — [what to verify]
2. ...
## API Tests (if applicable)
1. [Endpoint] — [expected behavior]
- Before state: [what to check before]
- After state: [what to verify changed]
## UI Tests (if applicable)
1. [Page/component] — [interaction to test]
- Screenshot before: [what to capture]
- Screenshot after: [what to capture]
## Negative Tests (REQUIRED — at least one per feature)
1. [What should NOT happen] — [how to trigger it]
- Expected error: [what error message/code]
- State unchanged: [what to verify did NOT change]
Be critical — include edge cases, error paths, and security checks. Every scenario MUST specify what screenshots to take and what state to verify.
Multiple worktrees share the same host — Docker infra (postgres, redis, clamav), app ports (3000/8006/…), and the test user. Two agents running /pr-test concurrently will corrupt each other's state (connection-pool exhaustion, port binds failing silently, cross-test assertions). Use the root-worktree lock file to take turns.
Path (always the root worktree so all siblings see it): $REPO_ROOT/.ign.testing.lock
Body (one key=value per line):
holder=<pr-XXXXX-purpose>
pid=<pid-or-"self">
started=<iso8601>
heartbeat=<iso8601, updated every ~2 min>
worktree=<full path>
branch=<branch name>
intent=<one-line description + rough duration>
LOCK=$REPO_ROOT/.ign.testing.lock
NOW=$(date -u +%Y-%m-%dT%H:%MZ)
STALE_AFTER_MIN=5
if [ -f "$LOCK" ]; then
HB=$(grep '^heartbeat=' "$LOCK" | cut -d= -f2)
HB_EPOCH=$(date -j -f '%Y-%m-%dT%H:%MZ' "$HB" +%s 2>/dev/null || date -d "$HB" +%s 2>/dev/null || echo 0)
AGE_MIN=$(( ( $(date -u +%s) - HB_EPOCH ) / 60 ))
if [ "$AGE_MIN" -gt "$STALE_AFTER_MIN" ]; then
echo "WARN: stale lock (${AGE_MIN}m old) — reclaiming"
cat "$LOCK" | sed 's/^/ stale: /'
else
echo "Another agent holds the lock:"; cat "$LOCK"
echo "Wait until released or resume after $((STALE_AFTER_MIN - AGE_MIN))m."
exit 1
fi
fi
cat > "$LOCK" <<EOF
holder=pr-${PR_NUMBER}-e2e
pid=self
started=$NOW
heartbeat=$NOW
worktree=$WORKTREE_PATH
branch=$(cd $WORKTREE_PATH && git branch --show-current)
intent=E2E test PR #${PR_NUMBER}, native mode, ~60min
EOF
echo "Lock claimed"
Without a heartbeat a crashed agent keeps the lock forever. Run this as a background process right after claim:
(while true; do
sleep 120
[ -f "$LOCK" ] || exit 0 # lock released → exit heartbeat
perl -i -pe "s/^heartbeat=.*/heartbeat=$(date -u +%Y-%m-%dT%H:%MZ)/" "$LOCK"
done) &
HEARTBEAT_PID=$!
echo "$HEARTBEAT_PID" > /tmp/pr-test-heartbeat.pid
kill "$HEARTBEAT_PID" 2>/dev/null
rm -f "$LOCK" /tmp/pr-test-heartbeat.pid
echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] released lock" \
>> $REPO_ROOT/.ign.testing.log
Use a trap so release runs even on exit 1:
trap 'kill "$HEARTBEAT_PID" 2>/dev/null; rm -f "$LOCK"' EXIT INT TERM
The lock guards test execution, not app lifecycle. Once Step 5 (record results) and Step 6 (post PR comment) are complete, release the lock IMMEDIATELY — even if:
poetry run app / pnpm dev processes are still running so the user can keep poking at the app manually.Keeping the lock held past the test run is the single most common way /pr-test stalls other agents. The app staying up is orthogonal to the lock; don't conflate them. Sibling worktrees running their own /pr-test will kill the stray processes and free the ports themselves (Step 3c/3e-native handle that) — they just need the lock file gone.
Concretely, the sequence at the end of every /pr-test run (success or failure) is:
# 1. Write the final report + post PR comment — done above in Step 5/6.
# 2. Release the lock right now, even if the app is still up.
kill "$HEARTBEAT_PID" 2>/dev/null
rm -f "$LOCK" /tmp/pr-test-heartbeat.pid
echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] released lock (app may still be running)" \
>> $REPO_ROOT/.ign.testing.log
# 3. Optionally leave the app running and note it so the user knows:
echo "Native stack still running on :3000 / :8006 for manual poking. Kill with:"
echo " pkill -9 -f 'poetry run app'; pkill -9 -f 'next-server|next dev'"
If a sibling agent's /pr-test needs to take over, it'll do the kill+rebuild dance from Step 3c/3e-native on its own — your only job is to not hold the lock file past the end of your test.
$REPO_ROOT/.ign.testing.log is an append-only channel any agent can read/write. Use it for "I'm waiting", "I'm done, resources free", or post-run notes:
echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] <message>" \
>> $REPO_ROOT/.ign.testing.log
The root worktree ($REPO_ROOT) has the canonical .env files with all API keys. Copy them to the target worktree:
# CRITICAL: .env files are NOT checked into git. They must be copied manually.
cp $REPO_ROOT/autogpt_platform/.env $PLATFORM_DIR/.env
cp $REPO_ROOT/autogpt_platform/backend/.env $BACKEND_DIR/.env
cp $REPO_ROOT/autogpt_platform/frontend/.env $FRONTEND_DIR/.env
A copy from the root worktree is no longer sufficient on a recent dev —
auth moved from Supabase to Better Auth (see 3h) and two vars are easy to
miss because nothing fails loudly without them, it just 401s later:
$BACKEND_DIR/.env needs JWT_JWKS_URL — the Better Auth JWKS endpoint the
backend verifies tokens against. The localhost:3000 value below is for
native mode only. In docker mode it's harmless to have it in .env
because docker-compose.platform.yml overrides it with the
Compose-reachable http://frontend:3000/api/auth/jwks — but if you ever run
the backend against this .env value directly (bypassing Compose), a
localhost value inside a container resolves to itself, not the frontend,
and every call 401s with no other symptom.$FRONTEND_DIR/.env needs its own DATABASE_URL — Better Auth runs
inside the Next.js app and talks to Postgres directly, it does not go
through the backend. Derive it from $BACKEND_DIR/.env's DB_USER /
DB_PASS / DB_PORT / DB_NAME rather than copying
frontend/.env.default's placeholder verbatim — the placeholder happens
to match the stock local defaults, but if backend/.env's DB_PASS was
ever customized (rotated secret, non-default port), copying the placeholder
silently points Better Auth at the wrong database instead of the one the
rest of the stack actually uses.# [ -n ... ], not grep -q alone — a present-but-empty JWT_JWKS_URL= would
# otherwise be treated as "already set" and skip the fallback, leaving the
# backend without a JWKS endpoint to verify tokens against.
[ -n "$(grep '^JWT_JWKS_URL=' $BACKEND_DIR/.env | cut -d= -f2-)" ] || echo "JWT_JWKS_URL=http://localhost:3000/api/auth/jwks" >> $BACKEND_DIR/.env # native mode only — see note above
if [ -n "$(grep '^DATABASE_URL=' $FRONTEND_DIR/.env | cut -d= -f2-)" ]; then
# grep -q alone matches a present-but-empty DATABASE_URL= too, which would
# otherwise skip derivation and leave Better Auth pointed at nothing.
echo "Frontend DATABASE_URL: already set (not touching it)"
else
# cut -f2 (not -f2-) truncates any value containing '=' (base64 secrets do); -f2- keeps the rest.
DB_USER=$(grep '^DB_USER=' $BACKEND_DIR/.env | cut -d= -f2-)
DB_PASS=$(grep '^DB_PASS=' $BACKEND_DIR/.env | cut -d= -f2-)
DB_PORT=$(grep '^DB_PORT=' $BACKEND_DIR/.env | cut -d= -f2-)
DB_NAME=$(grep '^DB_NAME=' $BACKEND_DIR/.env | cut -d= -f2-)
: "${DB_USER:?}" "${DB_PASS:?}" "${DB_PORT:?}" "${DB_NAME:?}" # fail loudly, not with a silently-empty URL
# Percent-encode user/pass — a raw '@', '#', '?', '%', or ':' in either would
# otherwise be misparsed as URL structure instead of credential content.
# Via env vars, not `jq --arg`, which would put DB_PASS in the process arglist.
DB_USER_ENC=$(DB_USER_VAL="$DB_USER" jq -rn '$ENV.DB_USER_VAL|@uri')
DB_PASS_ENC=$(DB_PASS_VAL="$DB_PASS" jq -rn '$ENV.DB_PASS_VAL|@uri')
echo "DATABASE_URL=postgresql://${DB_USER_ENC}:${DB_PASS_ENC}@localhost:${DB_PORT}/${DB_NAME}" >> $FRONTEND_DIR/.env
# Reconstructed, not regex-redacted — a password containing '@' would otherwise
# leak its tail past a naive "redact up to the first @" pattern.
echo "Frontend DATABASE_URL: postgresql://${DB_USER_ENC}:***@localhost:${DB_PORT}/${DB_NAME}"
fi
The copilot needs an LLM API to function. Two approaches (try subscription first):
The claude_agent_sdk Python package bundles its own Claude CLI binary — no need to install @anthropic-ai/claude-code via npm. The backend auto-provisions credentials from environment variables on startup.
Run the helper script to extract tokens from your host and auto-update backend/.env (works on macOS, Linux, and Windows/WSL):
# Extracts OAuth tokens and writes CLAUDE_CODE_OAUTH_TOKEN + CLAUDE_CODE_REFRESH_TOKEN into .env
bash $BACKEND_DIR/scripts/refresh_claude_token.sh --env-file $BACKEND_DIR/.env
How it works: The script reads the OAuth token from:
"Claude Code-credentials")~/.claude/.credentials.json%APPDATA%/claude/.credentials.jsonIt sets CLAUDE_CODE_OAUTH_TOKEN, CLAUDE_CODE_REFRESH_TOKEN, and CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true in the .env file. On container startup, the backend auto-provisions ~/.claude/.credentials.json inside the container from these env vars. The SDK's bundled CLI then authenticates using that file. No claude login, no npm install needed.
Note: The OAuth token expires (~24h). If copilot returns auth errors, re-run the script and restart: $BACKEND_DIR/scripts/refresh_claude_token.sh --env-file $BACKEND_DIR/.env && docker compose up -d copilot_executor
If subscription mode doesn't work, switch to API key mode using OpenRouter:
# In $BACKEND_DIR/.env, ensure these are set:
CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=false
CHAT_API_KEY=<value of OPEN_ROUTER_API_KEY from the same .env>
CHAT_BASE_URL=https://openrouter.ai/api/v1
CHAT_USE_CLAUDE_AGENT_SDK=true
Use sed to update these values:
ORKEY=$(grep "^OPEN_ROUTER_API_KEY=" $BACKEND_DIR/.env | cut -d= -f2)
[ -n "$ORKEY" ] || { echo "ERROR: OPEN_ROUTER_API_KEY is missing in $BACKEND_DIR/.env"; exit 1; }
perl -i -pe 's/CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true/CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=false/' $BACKEND_DIR/.env
# Add or update CHAT_API_KEY and CHAT_BASE_URL
grep -q "^CHAT_API_KEY=" $BACKEND_DIR/.env && perl -i -pe "s|^CHAT_API_KEY=.*|CHAT_API_KEY=$ORKEY|" $BACKEND_DIR/.env || echo "CHAT_API_KEY=$ORKEY" >> $BACKEND_DIR/.env
grep -q "^CHAT_BASE_URL=" $BACKEND_DIR/.env && perl -i -pe 's|^CHAT_BASE_URL=.*|CHAT_BASE_URL=https://openrouter.ai/api/v1|' $BACKEND_DIR/.env || echo "CHAT_BASE_URL=https://openrouter.ai/api/v1" >> $BACKEND_DIR/.env
# Stop any running app containers (keep infra: supabase, redis, rabbitmq, clamav)
docker ps --format "{{.Names}}" | grep -E "rest_server|executor|copilot|websocket|database_manager|scheduler|notification|frontend|migrate" | while read name; do
docker stop "$name" 2>/dev/null
done
Native mode also: when running the app natively (see 3e-native), kill any stray host processes and free the app ports before starting — otherwise poetry run app and pnpm dev will fail to bind.
Kill by port, not by broad process pattern. A pattern-based
pkill -f "python.*backend" (or anything matching by worktree cwd) is too
coarse on a host running several worktrees — it has taken out the frontend
and a mock server sitting on other ports along with the intended backend
process. Target the pid actually holding each port instead — this only kills
whoever is bound to that specific port, which is narrower than a pattern
match, but it is not worktree isolation: if a sibling worktree's own dev
server happens to be using one of these ports (e.g. its frontend also on
:3000), this kills that too. lsof tells you who holds the port, not who
owns it — a docker-proxy pid there means a compose stack owns it.
# Free app ports one at a time — errors per port are ignored (port may simply
# be unused). `xargs -r` is GNU-only (macOS xargs rejects -r); the `[ -n ]`
# guard below is the portable equivalent.
for port in 3000 8006 8001 8002 8005 8008; do
pids=$(lsof -ti :$port -sTCP:LISTEN 2>/dev/null)
[ -n "$pids" ] && kill -9 $pids 2>/dev/null || true
done
Native mode runs infra (postgres, supabase, redis, rabbitmq, clamav) in docker but runs the backend and frontend directly on the host. This avoids the 3-8 minute docker compose build cycle on every backend change — code edits are picked up on process restart (seconds) instead of a full image rebuild.
When to prefer native mode (default for this skill):
poetry run app in a couple of secondsWhen to prefer docker mode (3e fallback):
Dockerfile, docker-compose.yml, or base imagesNote on 3b (copilot auth): no npm install anywhere. poetry install pulls in claude_agent_sdk, which ships its own Claude CLI binary — available on PATH whenever you run commands via poetry run (native) OR whenever the copilot_executor container is built from its Poetry lockfile (docker). The OAuth token extraction still applies (same refresh_claude_token.sh call).
Preamble: before starting native, run the kill-stray + free-ports block from 3c's "Native mode also" subsection.
1. Start infra only (one-time per session):
cd $PLATFORM_DIR && docker compose --profile local up deps --detach --remove-orphans --build
This brings up postgres/supabase/redis/rabbitmq/clamav and skips all app services.
2. Start the backend natively:
cd $BACKEND_DIR && (poetry run app 2>&1 | tee .ign.application.logs) &
poetry run app spawns all app subprocesses — rest_server, executor, copilot_executor, websocket, scheduler, notification_server, database_manager — inside ONE parent process. No separate containers, no separate terminals. The .ign.application.logs prefix is already gitignored.
3. Wait for the backend on :8006 BEFORE starting the frontend. This ordering matters — the frontend's pnpm dev startup invokes generate-api-queries, which fetches /openapi.json from the backend. If the backend isn't listening yet, pnpm dev fails immediately.
for i in $(seq 1 60); do
if [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8006/docs 2>/dev/null)" = "200" ]; then
echo "Backend ready"
break
fi
sleep 2
done
4. Start the frontend natively:
cd $FRONTEND_DIR && (pnpm dev 2>&1 | tee .ign.frontend.logs) &
5. Wait for the frontend on :3000:
for i in $(seq 1 60); do
if [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:3000 2>/dev/null)" = "200" ]; then
echo "Frontend ready"
break
fi
sleep 2
done
Once both are up, skip 3e/3f and go straight to 3g/3h (feature flags / test user creation).
cd $PLATFORM_DIR && docker compose build --no-cache 2>&1 | tail -20
if [ ${PIPESTATUS[0]} -ne 0 ]; then echo "ERROR: Docker build failed"; exit 1; fi
cd $PLATFORM_DIR && docker compose up -d 2>&1 | tail -20
if [ ${PIPESTATUS[0]} -ne 0 ]; then echo "ERROR: Docker compose up failed"; exit 1; fi
Note: If the container appears to be running old code (e.g. missing PR changes), use docker compose build --no-cache to force a full rebuild. Docker BuildKit may sometimes reuse cached COPY layers from a previous build on a different branch.
Expected time: 3-8 minutes for build, 5-10 minutes with --no-cache.
# Poll until backend and frontend respond
for i in $(seq 1 60); do
BACKEND=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8006/docs 2>/dev/null)
FRONTEND=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 2>/dev/null)
if [ "$BACKEND" = "200" ] && [ "$FRONTEND" = "200" ]; then
echo "Services ready"
break
fi
sleep 5
done
The platform moved off Supabase auth to Better Auth, embedded in the
Next.js app at /api/auth/*. Signup and sign-in both go through the frontend
now, not Kong on :8000 — and /api/auth/token mints a backend-API JWT from a
session cookie, it does not accept credentials directly, so sign-in has to
happen first to get that cookie.
Better Auth's default minimum password length is 12 characters — shorter
values fail signup with PASSWORD_TOO_SHORT and every step below degrades
silently into an empty token unless you check for it.
COOKIE_JAR=$(mktemp)
trap 'rm -f "$COOKIE_JAR"' EXIT # cleans up on early exit too, not just the happy path
# Via env vars, not `jq --arg`, which would put the password in the process arglist.
AUTH_PAYLOAD=$(PR_TEST_USER_EMAIL="$PR_TEST_USER_EMAIL" PR_TEST_USER_PASSWORD="$PR_TEST_USER_PASSWORD" \
jq -nc '{email:$ENV.PR_TEST_USER_EMAIL,password:$ENV.PR_TEST_USER_PASSWORD,name:"PR Test User"}')
# Signup (idempotent — a real error body means "already exists" only if you
# check; -d passes the payload as an argument, which leaks the password into
# process listings, so pipe it through stdin with --data-binary @- instead.
# --noproxy guards against an inherited proxy env var routing the password
# through a proxy. --max-time bounds it — without one, a server that accepts
# the connection but never responds hangs setup indefinitely instead of
# reaching the empty-$TOKEN failure check below).
SIGNUP_RESULT=$(printf '%s' "$AUTH_PAYLOAD" | curl -s --max-time 15 --noproxy localhost,127.0.0.1,::1 -X POST 'http://localhost:3000/api/auth/sign-up/email' \
-H 'Content-Type: application/json' --data-binary @-)
echo "$SIGNUP_RESULT" | grep -qi '"code"' && echo "Signup: $SIGNUP_RESULT" # log it — "already exists" and "password too short" look identical downstream otherwise
# Sign in — sets the better-auth.session_token cookie in $COOKIE_JAR.
# Capture the body: a failure here (e.g. account exists with a different
# password) otherwise only shows up as an empty $TOKEN with no explanation.
SIGNIN_RESULT=$(printf '%s' "$AUTH_PAYLOAD" | curl -s --max-time 15 --noproxy localhost,127.0.0.1,::1 -c "$COOKIE_JAR" -X POST 'http://localhost:3000/api/auth/sign-in/email' \
-H 'Content-Type: application/json' --data-binary @-)
echo "$SIGNIN_RESULT" | grep -qi '"code"' && echo "Sign-in: $SIGNIN_RESULT"
# Mint a backend-API JWT from the session cookie
TOKEN=$(curl -s -b "$COOKIE_JAR" 'http://localhost:3000/api/auth/token' | jq -r '.token // ""')
[ -n "$TOKEN" ] || { echo "ERROR: auth setup failed — TOKEN is empty. Check password length (min 12 chars), AUTH_ALLOW_NEW_ACCOUNTS, AUTH_SIGNUP_ALLOWLIST, and rate limiting on /api/auth/*."; exit 1; }
Use this token for ALL API calls:
curl -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/...
The frontend redirects to /onboarding when the ONBOARDING_COMPLETE step is not in completedSteps.
Mark it complete via the backend API so every browser test lands on the real feature UI:
ONBOARDING_RESULT=$(curl -s --max-time 30 -X POST \
"http://localhost:8006/api/onboarding/step?step=ONBOARDING_COMPLETE" \
-H "Authorization: Bearer $TOKEN")
echo "Onboarding bypass: $ONBOARDING_RESULT"
# Verify it took effect
ONBOARDING_STATUS=$(curl -s --max-time 30 \
"http://localhost:8006/api/onboarding/completed" \
-H "Authorization: Bearer $TOKEN" | jq -r '.is_completed')
echo "Onboarding completed: $ONBOARDING_STATUS"
if [ "$ONBOARDING_STATUS" != "true" ]; then
echo "ERROR: onboarding bypass failed — browser tests will hit /onboarding instead of the target feature. Investigate before proceeding."
exit 1
fi
| Service | Port | URL |
|---|---|---|
| Frontend | 3000 | http://localhost:3000 |
| Backend REST | 8006 | http://localhost:8006 |
| Supabase Auth (via Kong) | 8000 | http://localhost:8000 |
| Executor | 8002 | http://localhost:8002 |
| Copilot Executor | 8008 | http://localhost:8008 |
| WebSocket | 8001 | http://localhost:8001 |
| Database Manager | 8005 | http://localhost:8005 |
| Redis | 6379 | localhost:6379 |
| RabbitMQ | 5672 | localhost:5672 |
Use curl with the auth token for backend API tests. For EVERY API call that changes state, record before/after values:
# Example: List agents
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/graphs | jq . | head -20
# Example: Create an agent
curl -s -X POST http://localhost:8006/api/graphs \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{...}' | jq .
# Example: Run an agent
curl -s -X POST "http://localhost:8006/api/graphs/{graph_id}/execute" \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"data": {...}}'
# Example: Get execution results
curl -s -H "Authorization: Bearer $TOKEN" \
"http://localhost:8006/api/graphs/{graph_id}/executions/{exec_id}" | jq .
State verification pattern (use for EVERY state-changing API call):
# 1. Record BEFORE state
BEFORE_STATE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/{resource} | jq '{relevant_fields}')
echo "BEFORE: $BEFORE_STATE"
# 2. Perform the action
ACTION_RESULT=$(curl -s -X POST ... | jq .)
echo "ACTION RESULT: $ACTION_RESULT"
# 3. Record AFTER state
AFTER_STATE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/{resource} | jq '{relevant_fields}')
echo "AFTER: $AFTER_STATE"
# 4. Log the comparison
echo "=== STATE CHANGE VERIFICATION ==="
echo "Before: $BEFORE_STATE"
echo "After: $AFTER_STATE"
echo "Expected change: {describe what should have changed}"
For timeout/latency/error-handling behavior that would otherwise need a real
LLM call, point the OpenAI SDK at a local mock instead — it honours
OPENAI_BASE_URL, so a small local Responses-API server can stand in for the
provider while everything downstream (executor, credentials, billing) still
runs for real. This proved out 4/4 test items at $0 in this run. This
covers LLM blocks (providers.py) and the Codex block — the copilot's own
LLM calls go through backend/util/clients.py, which passes base_url
explicitly and does not read OPENAI_BASE_URL, so this trick doesn't reach
copilot chat.
# In $BACKEND_DIR/.env, point the OpenAI provider at a local mock server
# that implements the subset of the Responses API you need (e.g. delayed
# responses to test timeout handling, or a 500 to test error surfacing).
# Restart the backend after changing this — it's read at startup.
OPENAI_BASE_URL=http://localhost:{mock_port}/v1
Use a throwaway/dummy provider credential with the mock, never the system credential — only the transport is faked, so the credential lookup still runs for real and whatever key you configure gets sent as a header to your local mock server. A dummy key also means the mock server's logs (which may end up pasted into a PR comment) can't leak a real one. Aside from the credential, this is safe to use for anything that isn't itself testing model output quality.
In docker mode, localhost inside the backend container isn't reachable
from your host-side mock server — point OPENAI_BASE_URL at a
Compose-reachable hostname instead (or host.docker.internal with a
host-gateway entry), and set it before docker compose up or restart the
affected services after changing $BACKEND_DIR/.env.
Keep the mock server's own response timeout short — the OpenAI SDK's default client timeout is 600s, so a hung mock stalls every LLM-backed block for that long instead of failing fast.
Primary tool — use this wherever agent-browser is installed:
# Close any existing session
agent-browser close 2>/dev/null || true
# Use --session-name to persist cookies across navigations
# This means login only needs to happen once per test session
agent-browser --session-name pr-test open 'http://localhost:3000/login' --timeout 15000
# Get interactive elements
agent-browser --session-name pr-test snapshot | grep "textbox\|button"
# Login (read creds from env — set PR_TEST_USER_EMAIL / PR_TEST_USER_PASSWORD or ask the user)
agent-browser --session-name pr-test fill {email_ref} "$PR_TEST_USER_EMAIL"
agent-browser --session-name pr-test fill {password_ref} "$PR_TEST_USER_PASSWORD"
agent-browser --session-name pr-test click {login_button_ref}
sleep 5
# Dismiss cookie banner if present
agent-browser --session-name pr-test click 'text=Accept All' 2>/dev/null || true
# Navigate — cookies are preserved so login persists
agent-browser --session-name pr-test open 'http://localhost:3000/copilot' --timeout 10000
# Take screenshot
agent-browser --session-name pr-test screenshot $RESULTS_DIR/01-page.png
# Interact with elements
agent-browser --session-name pr-test fill {ref} "text"
agent-browser --session-name pr-test press "Enter"
agent-browser --session-name pr-test click {ref}
agent-browser --session-name pr-test click 'text=Button Text'
# Read page content
agent-browser --session-name pr-test snapshot | grep "text:"
Key pages:
/copilot — CoPilot chat (for testing copilot features)/build — Agent builder (for testing block/node features)/build?flowID={id} — Specific agent in builder/library — Agent library (for testing listing/import features)/library/agents/{id} — Agent detail with run history/marketplace — MarketplaceFallback — if agent-browser isn't installed on the host, don't let npx
download it. Use @playwright/test instead — the package is already in the
frontend's node_modules, but the Chromium binary itself is not;
Playwright caches browser binaries separately (~/.cache/ms-playwright/ by
default) and nothing installs them automatically. Run
pnpm exec playwright install chromium once per host if chromium.launch()
fails looking for an executable, then use it for finer control over timing
(waitForSelector, explicit timeouts) than agent-browser's CLI gives you:
cd $FRONTEND_DIR # required — @playwright/test is only declared here, not at the repo root
# Check the installer's own exit status, not grep's — piping through grep to
# drop blank lines would otherwise swallow a real install failure and let
# chromium.launch() below fail later with a much less clear error.
pnpm exec playwright install chromium
[ $? -eq 0 ] || { echo "ERROR: playwright install chromium failed"; exit 1; }
node -e "
const { chromium } = require('@playwright/test');
(async () => {
const browser = await chromium.launch();
try {
const page = await browser.newPage();
await page.goto('http://localhost:3000/login');
await page.screenshot({ path: '$RESULTS_DIR/01-login.png' });
} finally {
await browser.close(); // otherwise a goto timeout leaks a headless Chromium process
}
})();
"
Native mode: when running via poetry run app + pnpm dev, all app logs stream to the .ign.*.logs files written by the tee pipes in 3e-native. rest_server, executor, copilot_executor, websocket, scheduler, notification_server, and database_manager are all subprocesses of the single poetry run app parent, so their output is interleaved in .ign.application.logs.
# Backend (all app subprocesses interleaved)
tail -f $BACKEND_DIR/.ign.application.logs
# Frontend (Next.js dev server)
tail -f $FRONTEND_DIR/.ign.frontend.logs
# Filter for errors across either log
grep -iE "error|exception|traceback" $BACKEND_DIR/.ign.application.logs | tail -20
grep -iE "error|exception|traceback" $FRONTEND_DIR/.ign.frontend.logs | tail -20
Docker mode:
# Backend REST server
docker logs autogpt_platform-rest_server-1 2>&1 | tail -30
# Executor (runs agent graphs)
docker logs autogpt_platform-executor-1 2>&1 | tail -30
# Copilot executor (runs copilot chat sessions)
docker logs autogpt_platform-copilot_executor-1 2>&1 | tail -30
# Frontend
docker logs autogpt_platform-frontend-1 2>&1 | tail -30
# Filter for errors
docker logs autogpt_platform-executor-1 2>&1 | grep -i "error\|exception\|traceback" | tail -20
The copilot uses SSE streaming. To test via API:
# Create a session
SESSION_ID=$(curl -s -X POST 'http://localhost:8006/api/chat/sessions' \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{}' | jq -r '.id // .session_id // ""')
# Stream a message (SSE - will stream chunks)
curl -N -X POST "http://localhost:8006/api/chat/sessions/$SESSION_ID/stream" \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"message": "Hello, what can you help me with?"}' \
--max-time 60 2>/dev/null | head -50
Or test via browser (preferred for UI verification):
agent-browser --session-name pr-test open 'http://localhost:3000/copilot' --timeout 10000
# ... fill chat input and press Enter, wait 20-30s for response
Take a screenshot at EVERY significant test step — before and after interactions, on success, and on failure. This is NON-NEGOTIABLE.
Required screenshot pattern for each test scenario:
# BEFORE the action
agent-browser --session-name pr-test screenshot $RESULTS_DIR/{NN}-{scenario}-before.png
# Perform the action...
# AFTER the action
agent-browser --session-name pr-test screenshot $RESULTS_DIR/{NN}-{scenario}-after.png
Naming convention:
# Examples:
# $RESULTS_DIR/01-login-page-before.png
# $RESULTS_DIR/02-login-page-after.png
# $RESULTS_DIR/03-credits-page-before.png
# $RESULTS_DIR/04-credits-purchase-after.png
# $RESULTS_DIR/05-negative-insufficient-credits.png
# $RESULTS_DIR/06-error-state.png
Minimum requirements:
CRITICAL: After all tests complete, you MUST show every screenshot to the user using the Read tool, with an explanation of what each screenshot shows. This is the most important part of the test report — the user needs to visually verify the results.
For each screenshot:
Read tool to display the PNG file (Claude can read images)Format the output like this:
### Screenshot 1: {descriptive title}
[Read the PNG file here]
**What it shows:** {1-2 sentence explanation of what this screenshot proves}
---
After showing all screenshots, output a detailed summary table:
| # | Scenario | Result | API Evidence | Screenshot Evidence |
|---|---|---|---|---|
| 1 | {name} | PASS/FAIL | Before: X, After: Y | 01-before.png, 02-after.png |
| 2 | ... | ... | ... | ... |
IMPORTANT: As you show each screenshot and record test results, persist them in shell variables for Step 7:
# Build these variables during Step 6 — they are required by Step 7's script
# NOTE: declare -A requires Bash 4.0+. This is standard on modern systems (macOS ships zsh
# but Homebrew bash is 5.x; Linux typically has bash 5.x). If running on Bash <4, use a
# plain variable with a lookup function instead.
declare -A SCREENSHOT_EXPLANATIONS=(
["01-login-page.png"]="Shows the login page loaded successfully with SSO options visible."
["02-builder-with-block.png"]="The builder canvas displays the newly added block connected to the trigger."
# ... one entry per screenshot, using the same explanations you showed the user above
)
TEST_RESULTS_TABLE="| 1 | Login flow | PASS | N/A | 01-login-before.png, 02-login-after.png |
| 2 | Credits purchase | PASS | Before: 100, After: 95 | 03-credits-before.png, 04-credits-after.png |
| 3 | Insufficient credits (negative) | PASS | Credits: 0, rejected | 05-insufficient-credits-error.png |"
# ... one row per test scenario with actual results
Post the report with gh pr comment --attach. Each attached file is uploaded to GitHub's own asset store, and any image the body already references by that path is rewritten to point at the uploaded asset. Screenshots never touch a repo branch.
This step is MANDATORY. Every test run MUST post a PR comment — with the screenshots attached, or an image-free INCOMPLETE report when attachment is unavailable (see the fallback below). Never nothing.
Attachments are permanent. A user-attachments asset stays reachable by anyone holding its URL regardless of repo visibility, and deleting the comment does not reliably revoke it — there is no undo. Look at every screenshot for credentials, tokens, or customer data before this step runs.
Requires gh >= 2.99.0 — --attach landed there. Distro packages lag (Fedora's rpm tops out at 2.97.0), so check the version you are actually running before building the body:
# Too-old gh is not fatal here: the report still gets posted, just without
# images, through the same fallback path a failed upload takes (below).
GH_VERSION=$(gh version 2>/dev/null | head -1 | awk '{print $3}')
# Numeric compare, not `sort -V`: BSD sort on macOS may lack -V, and an empty
# substitution would silently read as "too old" on a perfectly good gh.
GH_MAJOR=${GH_VERSION%%.*}; GH_REST=${GH_VERSION#*.}; GH_MINOR=${GH_REST%%.*}
ATTACH_OK=0
ATTACH_SKIP_REASON=""
if [ "${GH_MAJOR:-0}" -gt 2 ] 2>/dev/null || { [ "${GH_MAJOR:-0}" -eq 2 ] && [ "${GH_MINOR:-0}" -ge 99 ]; } 2>/dev/null; then
ATTACH_OK=1
else
ATTACH_SKIP_REASON="gh ${GH_VERSION:-not found} is below 2.99.0 — upgrade gh"
echo "WARN: ${ATTACH_SKIP_REASON}; posting the report without images."
fi
CRITICAL — NEVER post a bare directory link like https://github.com/.../tree/.... Every screenshot MUST appear as  inline in the PR comment so reviewers can see them without clicking any links. After posting, the verification step below counts the rewritten asset URLs against the number of screenshots and exits 1 on any shortfall — the test run is incomplete until this passes.
CRITICAL — NEVER paste absolute local paths or credentials into the PR comment. Strings like /Users/…, /home/…, C:\… are useless to every reviewer except you, and a bearer token or JWT copied from the live stack's evidence is a real leak in a public comment. The build block below runs an egress check on the exact bytes about to be posted and aborts on either. The ./01-shot.png references the attachments use are relative, not local paths, and are rewritten on upload — they are fine. Keep local paths in $RESULTS_DIR/test-report.md for yourself; only copy the content they reference (excerpts, test names, log lines) into the PR comment, not the path.
Build the body with relative image references, then attach the same paths:
REPO="Significant-Gravitas/AutoGPT"
MAX_ATTACHMENTS=50 # per gh pr comment invocation
# Fail closed: an empty RESULTS_DIR would glob the PR worktree's PNGs into a public comment.
# gh matches --attach to the body by exact path, so the cd is required; undone at the end.
[ -n "$RESULTS_DIR" ] && [ -d "$RESULTS_DIR" ] || { echo "ERROR: RESULTS_DIR is unset or not a directory: '$RESULTS_DIR'"; exit 1; }
STEP7_ORIG_DIR=$PWD
cd "$RESULTS_DIR" || exit 1
NULLGLOB_WAS=$(shopt -p nullglob) # restore the caller's setting, don't force it off
shopt -s nullglob
SCREENSHOT_FILES=(*.png)
$NULLGLOB_WAS
if [ ${#SCREENSHOT_FILES[@]} -eq 0 ]; then
echo "ERROR: No screenshots found in $RESULTS_DIR. Test run is incomplete."
exit 1
fi
# Over the cap, post the report without images rather than nothing at all.
if [ ${#SCREENSHOT_FILES[@]} -gt "$MAX_ATTACHMENTS" ]; then
ATTACH_SKIP_REASON="${#SCREENSHOT_FILES[@]} screenshots exceed the ${MAX_ATTACHMENTS}-attachment limit per comment"
echo "WARN: ${ATTACH_SKIP_REASON}; posting the report without images."
ATTACH_OK=0
fi
IMAGE_MARKDOWN=""
ATTACH_ARGS=()
for BASENAME in "${SCREENSHOT_FILES[@]}"; do
TITLE=$(echo "${BASENAME%.png}" | sed 's/^[0-9]*-//' | sed 's/-/ /g' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) tolower(substr($i,2))}1')
EXPLANATION="${SCREENSHOT_EXPLANATIONS[$BASENAME]}"
if [ -z "$EXPLANATION" ]; then
echo "ERROR: Missing screenshot explanation for $BASENAME. Add it to SCREENSHOT_EXPLANATIONS in Step 6."
exit 1
fi
IMAGE_MARKDOWN="${IMAGE_MARKDOWN}
### ${TITLE}
${EXPLANATION}
"
# Alt text after '#' is the fallback; the body reference above wins when both exist.
ATTACH_ARGS+=(--attach "./${BASENAME}#${TITLE}")
done
# Keep the report and the image section separate: the no-image fallback below
# must not carry "" references it has no attachments for.
RUN_MARKER="<!-- pr-test-report:${PR_NUMBER}:$(date -u +%Y%m%dT%H%M%SZ) -->"
REPORT_BODY="${RUN_MARKER}
## E2E Test Report
| # | Scenario | Result | API Evidence | Screenshot Evidence |
|---|----------|--------|-------------|-------------------|
${TEST_RESULTS_TABLE}
"
COMMENT_FILE=$(mktemp)
printf '%s\n%s\n' "$REPORT_BODY" "$IMAGE_MARKDOWN" > "$COMMENT_FILE"
# Egress check on the exact bytes about to be posted. Fails closed: a missing
# body is an abort, not a pass.
LEAK_RE='(^|[^A-Za-z])(/Users/|/home/|/tmp/|/private/|C:\\|~/)[A-Za-z0-9]|eyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{10,}|(sk|ghp|gho|github_pat)_[A-Za-z0-9_]{16,}|[Bb]earer [A-Za-z0-9._-]{20,}'
[ -s "$COMMENT_FILE" ] || { echo "ABORT: comment body missing or empty."; exit 1; }
if grep -nE "$LEAK_RE" "$COMMENT_FILE"; then
echo "ABORT: local paths or credential-shaped strings in the PR comment body. Rewrite them before posting."
exit 1
fi
# A blind retry duplicates a post GitHub accepted but gh reported failed; the marker
# makes "did it land?" checkable first. Real jq (gh's --jq has no --arg); --paginate
# because the per-issue endpoint ignores sort/direction.
URL_RE='https://[^[:space:]]+#issuecomment-[0-9]+'
POSTED=""
if [ "$ATTACH_OK" = 1 ]; then
for attempt in 1 2 3; do
# The marker was minted seconds ago, so attempt 1 cannot already be posted.
# A lookup that errors is "unknown", not "not found": posting on unknown is
# how the duplicate the marker exists to prevent would get made.
if [ "$attempt" -gt 1 ]; then
if ! FOUND=$(set -o pipefail; gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" --paginate \
| jq -r --arg m "$RUN_MARKER" 'first(.[] | select(.body | contains($m)) | .html_url) // empty'); then
echo "Attempt $attempt: marker lookup failed; not posting until it can be confirmed"
sleep $((attempt * attempt * 2)); continue
fi
POSTED=${FOUND%%$'\n'*}
[ -n "$POSTED" ] && break
fi
# Take the URL by shape, not position: gh prints its update notice on stderr at
# exit, and a trailing one would otherwise ride into COMMENT_ID below.
OUT=$(gh pr comment "$PR_NUMBER" --repo "$REPO" --body-file "$COMMENT_FILE" "${ATTACH_ARGS[@]}" 2>&1)
POSTED=$(printf '%s' "$OUT" | grep -oE "$URL_RE" | tail -1)
[ -n "$POSTED" ] && break
echo "Attempt $attempt failed: $OUT"
[ "$attempt" -lt 3 ] && sleep $((attempt * attempt * 2)) # 2s, 8s — secondary rate limits need room
done
fi
If gh is too old, the run is over the cap, or all three attempts failed, post the report without images and say so, so the run is visibly incomplete rather than silently missing its evidence. The run continues into Step 8 — an image-free report is not approvable, and that is Step 8's job to say:
LOOKUP_UNKNOWN=0
if [ -z "$POSTED" ] && [ "$ATTACH_OK" = 1 ]; then
# One last marker check — a final attempt may have landed despite reporting failure.
if FOUND=$(set -o pipefail; gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" --paginate \
| jq -r --arg m "$RUN_MARKER" 'first(.[] | select(.body | contains($m)) | .html_url) // empty'); then
POSTED=${FOUND%%$'\n'*}
else
LOOKUP_UNKNOWN=1
fi
fi
RUN_INCOMPLETE=0
if [ -z "$POSTED" ] && [ "$LOOKUP_UNKNOWN" = 1 ]; then
# Can't tell whether the report landed; a fallback post here risks a duplicate.
RUN_INCOMPLETE=1
echo "ERROR: could not confirm whether the report was posted (GitHub API errors). Check the PR for marker ${RUN_MARKER} before re-running Step 7."
elif [ -z "$POSTED" ]; then
RUN_INCOMPLETE=1
[ "$ATTACH_OK" = 1 ] && ATTACH_SKIP_REASON="the upload failed 3 times"
FALLBACK_FILE=$(mktemp)
{
printf '%s\n' "$REPORT_BODY"
printf '## ⚠️ Screenshots not attached\n\n'
printf '%s. Filenames, for manual drag-and-drop into a reply:\n\n' "$ATTACH_SKIP_REASON"
printf -- '- `%s`\n' "${SCREENSHOT_FILES[@]}"
printf '\n**Run status:** INCOMPLETE until the files are attached and visible inline in the PR.\n'
} > "$FALLBACK_FILE"
# The degraded path gets the same egress check as the full body.
if grep -nE "$LEAK_RE" "$FALLBACK_FILE"; then
echo "ABORT: local paths or credential-shaped strings in the fallback body."
exit 1
fi
OUT=$(gh pr comment "$PR_NUMBER" --repo "$REPO" --body-file "$FALLBACK_FILE" 2>&1)
POSTED=$(printf '%s' "$OUT" | grep -oE "$URL_RE" | tail -1)
rm -f "$FALLBACK_FILE"
echo "Posted without images (RUN_INCOMPLETE=1): ${POSTED:-FAILED — $OUT}"
fi
rm -f "$COMMENT_FILE"
cd "$STEP7_ORIG_DIR"
Verify the rendered comment actually carries every screenshot as an uploaded asset (skipped for an image-free fallback, which Step 8 already treats as not approvable):
if [ "$RUN_INCOMPLETE" = 0 ]; then
COMMENT_ID="${POSTED##*issuecomment-}"
BODY=$(gh api "repos/${REPO}/issues/comments/${COMMENT_ID}" --jq '.body')
EXPECTED=${#SCREENSHOT_FILES[@]}
# Count, don't just detect: a partial rewrite leaves broken "](./x.png)"
# links behind while the body still contains *some* asset URLs.
UPLOADED=$(printf '%s' "$BODY" | grep -o 'github.com/user-attachments/' | wc -l | tr -d ' ')
LEFTOVER=$(printf '%s' "$BODY" | grep -o '](\./' | wc -l | tr -d ' ')
RAW_IMGS=$(printf '%s' "$BODY" | grep -oE '!\[[^]]*\]\(https://raw\.githubusercontent\.com' | wc -l | tr -d ' ')
if [ "$RAW_IMGS" -gt 0 ]; then
echo "ERROR: $RAW_IMGS image(s) still point at raw repo URLs. Screenshots must be attachments, not files in the repo." >&2
exit 1
fi
if [ "$LEFTOVER" -gt 0 ] || [ "$UPLOADED" -ne "$EXPECTED" ]; then
echo "ERROR: expected $EXPECTED uploaded attachments, found $UPLOADED, with $LEFTOVER unrewritten ./ reference(s) — partial upload." >&2
exit 1
fi
echo "✓ $EXPECTED screenshots verified as GitHub attachments"
fi
The PR comment MUST include:
After the test comment is posted, evaluate whether the run was thorough enough to make a merge decision, then post a formal GitHub review (approve or request changes). This step is mandatory — every test run MUST end with a formal review decision.
Re-read the PR description:
gh pr view "$PR_NUMBER" --json body --jq '.body' --repo "$REPO"
Score the run against each criterion:
| Criterion | Pass condition |
|---|---|
| Coverage | Every feature/change described in the PR has at least one test scenario |
| All scenarios pass | No FAIL rows in the results table |
| Negative tests | At least one failure-path test per feature (invalid input, unauthorized, edge case) |
| Before/after evidence | Every state-changing API call has before/after values logged |
| Screenshots are meaningful | Screenshots show the actual state change, not just a loading spinner or blank page |
| No regressions | Existing core flows (login, agent create/run) still work |
ALL criteria pass → APPROVE
Any scenario FAIL or missing PR feature → REQUEST_CHANGES (list gaps)
Evidence weak (no before/after, vague shots) → REQUEST_CHANGES (list what's missing)
Screenshots not attached (RUN_INCOMPLETE=1) → REQUEST_CHANGES (the evidence is not on the PR)
REVIEW_FILE=$(mktemp)
# Count results
PASS_COUNT=$(echo "$TEST_RESULTS_TABLE" | grep -c "PASS" || true)
FAIL_COUNT=$(echo "$TEST_RESULTS_TABLE" | grep -c "FAIL" || true)
TOTAL=$(( PASS_COUNT + FAIL_COUNT ))
# List any coverage gaps found during evaluation (populate this array as you assess)
# e.g. COVERAGE_GAPS=("PR claims to add X but no test covers it")
COVERAGE_GAPS=()
# Step 7 posts an image-free report instead of aborting; that run is not approvable.
[ "${RUN_INCOMPLETE:-0}" = 1 ] && COVERAGE_GAPS+=("Screenshots were not attached to the test report — attach them and re-run verification")
If APPROVING — all criteria met, zero failures, full coverage:
cat > "$REVIEW_FILE" <<REVIEWEOF
## E2E Test Evaluation — APPROVED
**Results:** ${PASS_COUNT}/${TOTAL} scenarios passed.
**Coverage:** All features described in the PR were exercised.
**Evidence:** Before/after API values logged for all state-changing operations; screenshots show meaningful state transitions.
**Negative tests:** Failure paths tested for each feature.
No regressions observed on core flows.
REVIEWEOF
gh pr review "$PR_NUMBER" --repo "$REPO" --approve --body "$(cat "$REVIEW_FILE")"
echo "✅ PR approved"
If REQUESTING CHANGES — any failure, coverage gap, or missing evidence:
FAIL_LIST=$(echo "$TEST_RESULTS_TABLE" | grep "FAIL" | awk -F'|' '{print "- Scenario" $2 "failed"}' || true)
cat > "$REVIEW_FILE" <<REVIEWEOF
## E2E Test Evaluation — Changes Requested
**Results:** ${PASS_COUNT}/${TOTAL} scenarios passed, ${FAIL_COUNT} failed.
### Required before merge
${FAIL_LIST}
$(for gap in "${COVERAGE_GAPS[@]}"; do echo "- $gap"; done)
Please fix the above and re-run the E2E tests.
REVIEWEOF
gh pr review "$PR_NUMBER" --repo "$REPO" --request-changes --body "$(cat "$REVIEW_FILE")"
echo "❌ Changes requested"
rm -f "$REVIEW_FILE"
Rules:
--fix mode, fix all failures before posting the review — the review reflects the final state after fixesRUN_INCOMPLETE=1 — the evidence is not on the PRWhen --fix is present, the standard is HIGHER. Do not just note issues — FIX them immediately.
pytest.mark.xfail(reason="..."). For frontend/Playwright bugs, write a test with .fixme annotation. Run it to confirm it fails as expected.agent-browser screenshot $RESULTS_DIR/{NN}-broken-{description}.pngcd $PLATFORM_DIR && docker compose up --build -d {service_name}
# e.g., docker compose up --build -d rest_server
# e.g., docker compose up --build -d frontend
agent-browser screenshot $RESULTS_DIR/{NN}-fixed-{description}.pngcd $WORKTREE_PATH
git add -A
git commit -m "fix: {description of fix}"
git push
test scenario → find issue (bug OR UX problem) → screenshot broken state
→ fix code → rebuild affected service only → re-test → screenshot fixed state
→ verify no regressions → commit + push
→ repeat for next scenario
→ after ALL scenarios pass, run full re-test to verify everything together
Key differences from non-fix mode:
Cause: Better Auth's default minimum password length is 12 characters, or
AUTH_ALLOW_NEW_ACCOUNTS=false / AUTH_SIGNUP_ALLOWLIST is blocking new
accounts, or you're rate-limited after repeated signup attempts (429 Too many requests) — all three degrade into an empty $TOKEN further down if you
don't check the raw response (see 3h).
Fix: Log the raw signup/sign-in response body before minting the token,
not just the token itself.
Cause: CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true but CLAUDE_CODE_OAUTH_TOKEN is not set or expired.
Fix: Re-extract the OAuth token from macOS keychain (see step 3b, Option 1) and recreate the container (docker compose up -d copilot_executor). The backend auto-provisions ~/.claude/.credentials.json from the env var on startup. No npm install or claude login needed — the SDK bundles its own CLI binary.
Cause: The Dockerfile auto-provisions system chromium on all architectures (including ARM64). If your branch is behind dev, this may not be present yet.
Fix: Check if chromium exists: which chromium || which chromium-browser. If missing, install it: apt-get install -y chromium and set AGENT_BROWSER_EXECUTABLE_PATH=/usr/bin/chromium in the container environment.
Cause: text=X matches all elements containing that text.
Fix: Use agent-browser snapshot to get specific ref=eNN references, then use those: agent-browser click eNN.
Fix: agent-browser click 'text=Accept All' before other interactions.
Symptom: Copilot logs say claude: command not found or similar when starting an SDK turn.
Cause: Image was built without poetry install (stale base layer, or Dockerfile bypass). The SDK CLI ships inside the claude_agent_sdk Poetry dep — it is NOT an npm package.
Fix: Rebuild the image cleanly: docker compose build --no-cache copilot_executor && docker compose up -d copilot_executor. Do NOT docker exec ... npm install -g @anthropic-ai/claude-code — that is outdated guidance and will pollute the container with a second CLI that the SDK won't use.
Symptom: agent-browser screenshot exits with code 124 even on about:blank.
Cause: Stuck CDP connection or Chromium process tree. Seen on macOS when a prior /pr-test left a zombie Chrome for Testing.
Fix: pkill -9 -f "agent-browser|chromium|Chrome for Testing" && sleep 2, then reopen the browser with a fresh --session-name. If still failing, verify via agent-browser eval + agent-browser snapshot (DOM state) instead of relying on PNGs — the feature under test is the same.
docker compose upFix: Wait and check health: docker compose ps. Common cause: migration hasn't finished. Check: docker logs autogpt_platform-migrate-1 2>&1 | tail -5. If the db container isn't healthy: docker restart autogpt_platform-db-1 && sleep 10.
Cause: docker compose up --build reuses cached COPY layers from previous builds. If the PR branch changes Python files but the previous build already cached that layer from dev, the container runs dev code.
Fix: Always use docker compose build --no-cache for the first build of a PR branch. Subsequent rebuilds within the same branch can use --build.
agent-browser open loses login sessionCause: Without session persistence, agent-browser open starts fresh.
Fix: Use --session-name pr-test on ALL agent-browser commands. This auto-saves/restores cookies and localStorage across navigations. Alternatively, use agent-browser eval "window.location.href = '...'" to navigate within the same context.