================================================================================
Arena Watcher v5 重建指引（Reboot 後由零開始）
================================================================================
版本：v1.0（2026-10-05）
適用：Hatch VM（~/workspace/arena-system/）A2/A3/A4 三席
原則：每一步都要驗證成功先做下一步；唔好跳步

================================================================================
第 0 步：確認基本環境
================================================================================

[0.1] 確認 SSH 去 server 通
~/workspace/ssh-newserver.sh 'echo OK'
預期：OK

[0.2] 確認 Python 3
python3 --version
預期：Python 3.10+

[0.3] 確認磁碟空間
df -h ~/workspace | tail -1
預期：可用 >2GB

================================================================================
第 1 步：系統依賴（apt）
================================================================================

[1.1] 安裝 Xvfb 同必要庫
sudo apt update
sudo apt install -y xvfb x11vnc fluxbox \
libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 \
libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
libgbm1 libpango-1.0-0 libcairo2 libasound2

[1.2] 驗證
which Xvfb
預期：/usr/bin/Xvfb

================================================================================
第 2 步：Playwright venv 重建
================================================================================

[2.1] 刪舊 venv（如果爛咗）
rm -rf ~/workspace/arena-system/venv

[2.2] 建新 venv
python3 -m venv ~/workspace/arena-system/venv

[2.3] 裝 Playwright
~/workspace/arena-system/venv/bin/pip install --upgrade pip
~/workspace/arena-system/venv/bin/pip install playwright==1.63.0

[2.4] 裝 Playwright Chromium（bundle 版，唔用 /opt/meta-chromium）
~/workspace/arena-system/venv/bin/playwright install chromium --with-deps
注意：--with-deps 可能要 sudo；如果唔得，返去第 1 步手動裝齊庫

[2.5] 驗證
~/workspace/arena-system/venv/bin/python -c "from playwright.sync_api import sync_playwright; print('OK')"
預期：OK

[2.6] 驗證 Chromium binary 存在
ls ~/.cache/ms-playwright/ | grep chromium
預期：見到 chromium-12xx 之類目錄

================================================================================
第 3 步：VPN SSH Tunnel
================================================================================

[3.1] 確認配置檔存在
cat ~/workspace/vpn-tunnel/vpn-proxies.conf
預期內容（7 行）：
JP1 11080 161.33.178.218 日本1
JP2 11081 151.145.67.176 日本2
JP3 11082 158.101.145.210 日本3
JP4 11083 140.245.82.16 日本4
ip2 11084 217.142.189.57 自建2
ip3 11085 217.142.189.119 自建3
ip4 11086 140.245.60.36 自建4

[3.2] 啟動 tunnel
~/workspace/vpn-tunnel/vpn-tunnel.sh start

[3.3] 驗證 7 個端口有 listen
ss -tlnp 2>/dev/null | grep -E '1108[0-6]'
預期：7 行，每個端口一行

[3.4] 驗證 SOCKS5 通（抽查一個）
curl -s --socks5 127.0.0.1:11080 --max-time 10 https://api.ipify.org
預期：返一個 IP（應該係 161.33.178.218）

[3.5] 如果唔通，睇 log
tail -20 ~/workspace/vpn-tunnel/watchdog.log

================================================================================
第 4 步：Xvfb 虛擬顯示
================================================================================

[4.1] 啟動 Xvfb:99
Xvfb:99 -screen 0 1400x900x24 &
echo $! > /tmp/xvfb-99.pid

[4.2] 驗證
DISPLAY=:99 xdpyinfo 2>/dev/null | head -3
預期：見到 screen #0 dimensions: 1400x900

[4.3] 如果 xdpyinfo 冇裝，用另一個方法驗
ps aux | grep "[X]vfb:99"
預期：見到 Xvfb:99 行緊

注意：viewport 必須係 1400x900（CA bm.js 原規格，固定座標依賴呢個尺寸）

================================================================================
第 5 步：Browser Profile 準備
================================================================================

[5.1] 確認本地 profile 目錄
ls ~/workspace/arena-system/local-profiles/
預期：ai-0 ai-1 ai-12（至少呢三個）

[5.2] 確認 seat-profile-map.conf
cat ~/workspace/arena-system/seat-profile-map.conf
預期：
A2 xai-3
A3 ai-0
A4 ai-1

[5.3] 確認 seats.conf
cat ~/workspace/arena-system/seats.conf
預期：A2/A3/A4 三行，狀態 active，VPN 同 URL 啱

[5.4] 如果本地 profile 唔見咗，由 server 拉返
# A2 例子（其他席位換對應名）
~/workspace/ssh-newserver.sh 'ls /root/profiles/xai-3/'
# 確認 server 有，先用 ram-profile.sh prepare
~/workspace/arena-system/scripts/ram-profile.sh prepare a2

[5.5] 驗證 RAM profile
ls /dev/shm/arena-profiles/
預期：a2 a3 a4 三個目錄

================================================================================
第 6 步：開 Browser（每席一個）
================================================================================

重要：一定要用 Playwright launch_persistent_context，唔准用
/opt/meta-chromium/chrome raw CLI（指引鐵律，2026-10-03 定）

[6.1] 用 seat-browser.py 開（標準做法）
# 每席獨立跑，唔好並行開多個
DISPLAY=:99 ~/workspace/arena-system/venv/bin/python \
~/workspace/arena-system/scripts/seat-browser.py A2 &

[6.2] 等 15 秒，驗證 browser process
ps aux | grep "[c]hrome.*ai-12"
預期：見到 chrome process，user-data-dir 指住 ai-12

[6.3] 驗證 CDP port 有 listen
# seat-browser.py 會開對應嘅 debugging port
ss -tlnp 2>/dev/null | grep -E '92(2[0-9])'
預期：見到對應端口

[6.4] 重複 [6.1]-[6.3] 開 A3、A4
A3 profile ai-0，A4 profile ai-1

[6.5] 驗證三個 browser 都在生
ps aux | grep "[c]hrome" | wc -l
預期：>=3（每個 browser 有多個 process，實際會多過 3）

================================================================================
第 7 步：登入驗證（每個 browser）
================================================================================

[7.1] 經 CDP 檢查 Google 登入狀態
# 用 Playwright connect 返個 browser，開 myaccount.google.com 睇下係咪登入緊
# 如果要人手登入，睇 watcher-login-guide.txt標準流程

[7.2] 經 CDP 檢查 Arena 登入狀態
# 開對應嘅 arena.ai/agent/<uuid>，睇下係咪入到對話頁

[7.3] 三席都要驗，唔可以假設

注意：如果要重登，睇 ~/workspace/your_files/watcher-login-guide.txt
標準流程：解鎖三個 policy → 重開 browser → 登 Google → 登 Arena → 鎖返

================================================================================
第 8 步：啟動 Watcher
================================================================================

[8.1] 啟動三席 watcher
~/workspace/arena-system/scripts/start-watcher.sh start A2
~/workspace/arena-system/scripts/start-watcher.sh start A3
~/workspace/arena-system/scripts/start-watcher.sh start A4

[8.2] 驗證
~/workspace/arena-system/scripts/start-watcher.sh status
預期：三個都「運行中」

[8.3] 驗證 log 有寫入
sleep 35
tail -5 ~/workspace/arena-system/logs/watcher-a2.log
預期：見到 30 秒檢查嘅 log

================================================================================
第 9 步：Cron Jobs（如果 reboot 清咗）
================================================================================

[9.1] 睇下而家有咩 cron
crontab -l

[9.2] 如果冇咗，由备份恢復
# cron 定義喺 ~/workspace/cron.d/（如果有的話）
ls ~/workspace/cron.d/

[9.3] 最少要有呢幾條（按實際調整）：
# VPN tunnel watchdog（每分鐘）
# watcher watchdog
# tg-relay-minute-watch（每分鐘）
# council-transcript-backup（每 2 分鐘）

[9.4] 驗證 cron 行緊
sleep 70
tail -5 ~/workspace/vpn-tunnel/watchdog.log
預期：有新嘅檢查記錄

================================================================================
第 10 步：全面驗收
================================================================================

[10.1] Browser 三個在生
ps aux | grep "[X]vfb:99" && echo "Xvfb OK"
~/workspace/arena-system/scripts/start-watcher.sh status

[10.2] VPN 7 個端口通
~/workspace/vpn-tunnel/vpn-tunnel.sh status

[10.3] Watcher log 有 30 秒心跳
for s in a2 a3 a4; do echo "=== $s ==="; tail -2 ~/workspace/arena-system/logs/watcher-$s.log; done

[10.4] 登入狀態（三席都要入到 Arena）
# 經 VNC 或 CDP 目視確認

全部 PASS 先算重建完成。

================================================================================
故障排除
================================================================================

Q: Playwright 裝唔到 chromium？
A: 試 ~/workspace/arena-system/venv/bin/playwright install --with-deps chromium
唔得就手動 sudo apt 裝第 1 步啲庫

Q: VPN tunnel 起唔到？
A: 先確認 ~/workspace/ssh-newserver.sh 'echo OK' 通
再睇 ~/workspace/vpn-tunnel/watchdog.log

Q: Browser 開咗但入唔到 Arena？
A: 睇係咪要重登（睇 watcher-login-guide.txt）
檢查 SOCKS5 proxy 啱唔啱（seats.conf 嘅 VPN 要對 vpn-proxies.conf）

Q: Profile 唔見咗？
A: 由 server 拉：ram-profile.sh prepare <seat>
server 路徑：/root/profiles/<名>（睇 seat-profile-map.conf）

Q: 唔小心用咗 /opt/meta-chromium/chrome raw CLI？
A: 即刻停咗佢，返去第 6 步用 Playwright 重開
指引鐵律：一定要 Playwright launch_persistent_context

================================================================================
檔案清單（重建涉及嘅全部檔案）
================================================================================

配置檔：
~/workspace/arena-system/seats.conf
~/workspace/arena-system/seat-profile-map.conf
~/workspace/vpn-tunnel/vpn-proxies.conf

腳本：
~/workspace/arena-system/scripts/start-watcher.sh
~/workspace/arena-system/scripts/arena-watcher.py
~/workspace/arena-system/scripts/ram-profile.sh
~/workspace/arena-system/scripts/seat-browser.py
~/workspace/vpn-tunnel/vpn-tunnel.sh
~/workspace/vpn-tunnel/tunnel-watchdog.sh
~/workspace/arena-system/watcher-watchdog.sh
~/workspace/arena-system/watcher-plus1-check.sh

Profile：
~/workspace/arena-system/local-profiles/ai-0/
~/workspace/arena-system/local-profiles/ai-1/
~/workspace/arena-system/local-profiles/ai-12/

指引：
~/workspace/your_files/watcher-login-guide.txt（登入流程）
~/workspace/your_files/dj-host-guide.txt（主持指引）
呢份檔案（重建指引）

================================================================================