init

.env.example

@@ -0,0 +1,59 @@
+# =============================================================================
+# 环境变量配置模板 (Environment Variables Template)
+# =============================================================================
+# 使用说明：
+# 1. 复制此文件并重命名为 .env
+# 2. 将所有占位符（如 your_xxx_here）替换为实际的配置值
+# 3. .env 文件包含敏感信息，已在 .gitignore 中排除，请勿提交到版本控制
+#
+# 配置优先级（从高到低）：
+# 1. 环境变量 (.env 文件)
+# 2. config.yaml
+#
+# 建议：
+# - 敏感信息（API密钥、Token、密码）使用 .env 文件
+# - 非敏感的本地路径/参数使用 config.yaml
+# =============================================================================
+
+# =============================================================================
+# GuiAgent 配置
+# =============================================================================
+
+# GuiAgent 日志输出目录（默认: ./output）
+GUIAGENT_LOG_DIR=./output
+
+# =============================================================================
+# KeyPool 中转服务配置
+# =============================================================================
+# 如果使用自建 KeyPool 中转服务且要求认证，在此填入访问密钥
+# 留空则不发送认证头
+KEY_POOL_API_KEY=
+
+
+# =============================================================================
+# 场景配置 - 测试账户凭证（用于 scene_configs.json 中的登录/注册指令）
+# =============================================================================
+# 用途：GuiAgent 在自动化处理登录/注册场景时使用的测试账户信息
+# 这些变量在 scene_configs.json 中通过 ${VAR_NAME} 格式引用
+
+SCENE_EMAIL=your_test_email@gmail.com
+SCENE_PASSWORD=your_test_password_here
+SCENE_USERNAME=your_test_username_here
+SCENE_PHONE=+1234567890
+
+# =============================================================================
+# 微信/企业微信通知 Token 配置
+# =============================================================================
+# 用途：项目中的通知功能（如测试结果、异常告警）
+# 获取方式：
+# - 微信：通过企业微信机器人或个人通知服务
+# - 企业微信：企业微信 > 群机器人 > Webhook 地址中的 key 参数
+
+# 微信通知 Token（个人）
+# 格式：WECHAT_TOKEN_<NAME>
+
+
+# 企业微信通知 Token（团队）
+# 格式：WECOM_TOKEN_<NAME>
+# WECOM_TOKEN_TEAM1=your_wecom_webhook_token_here
+

.gitattributes

@@ -0,0 +1,3 @@
+**/go_ios/bin/* filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,27 @@
+output/*
+.env
+DroidBot/saveapk/
+**/__pycache__/
+*.pyc
+venv/*
+GuiAgent/logs/agent.log
+pcap_droid_script/*
+.vscode/
+.idea/
+GuiAgent/data/
+.DS_Store
+ios-link
+logs/*
+runtime/*
+*.plist
+data_process/
+test_agent_output/
+tmp/
+output_backup/
+
+# Configuration files (keep only example files)
+config.yaml
+config.local.yaml
+
+# Google OAuth credentials (GuiAgent)
+DroidBot/guiagent_core/token.json

CONFIG_README.md

@@ -0,0 +1,356 @@
+# 配置管理系统说明
+
+## 概述
+
+本项目采用分层配置管理系统，支持环境变量和配置文件两种方式，实现敏感信息与应用配置的分离。
+
+## 配置文件结构
+
+```
+autool/
+├── .env                    # 环境变量配置（敏感信息，不提交）
+├── .env.example            # 环境变量配置模板（提交到版本库）
+├── config/
+│   ├── current_env.txt     # 当前环境标识（prod/test）
+│   ├── prod.yaml           # 生产环境配置
+│   ├── test.yaml           # 测试环境配置
+│   ├── local.yaml          # 本地覆盖配置（不提交）
+│   └── local.yaml.example  # 本地配置示例
+├── config_loader.py        # 配置文件加载模块
+├── env_loader.py           # 环境变量加载模块
+└── config_usage_examples.py # 使用示例
+```
+
+## 配置优先级
+
+配置加载的优先级从高到低为：
+
+1. **环境变量**（`.env` 文件或系统环境变量）
+2. **本地配置**（`config/local.yaml` 或 `config/local.json`）
+3. **环境配置**（`config/prod.yaml` 或 `config/test.yaml`）
+
+## 快速开始
+
+### 1. 配置环境变量（敏感信息）
+
+```bash
+# 复制模板文件
+cp .env.example .env
+
+# 编辑 .env 文件，填入实际的 API 密钥
+vim .env
+```
+
+`.env` 文件示例：
+```bash
+# AI 服务 API 密钥
+GEMINI_API_KEY=your_actual_gemini_api_key_here
+AZURE_API_KEY=your_actual_azure_api_key_here
+QWEN_API_KEY=sk-your_actual_qwen_api_key_here
+GLM_API_KEY=your_actual_glm_api_key_here
+
+# 通知服务 Token
+WECHAT_TOKEN_YFZ=your_actual_wechat_token_here
+WECOM_TOKEN_TEAM1=your_actual_wecom_token_here
+
+# 系统认证
+IOS_PASSWORD=your_ios_device_password_here
+```
+
+### 2. 配置本地开发环境（可选）
+
+如果需要覆盖环境配置（如修改路径、调试参数等），可创建本地配置文件：
+
+```bash
+# 复制本地配置模板
+cp config/local.yaml.example config/local.yaml
+
+# 编辑本地配置
+vim config/local.yaml
+```
+
+`config/local.yaml` 示例：
+```yaml
+# 本地开发配置覆盖
+logging:
+  level: DEBUG  # 覆盖生产环境的 INFO
+
+output:
+  base_dir: D:/local_output  # 使用本地路径
+
+redis:
+  host: 127.0.0.1  # 使用本地 Redis
+  db: 9  # 独立的数据库编号
+```
+
+### 3. 在代码中使用配置
+
+#### 读取环境变量（推荐用于敏感信息）
+
+```python
+from env_loader import get_env, get_env_bool, get_env_int
+
+# 读取必需的环境变量
+gemini_api_key = get_env(
+    "GEMINI_API_KEY",
+    required=True,
+    hint="See .env.example for setup instructions"
+)
+
+# 读取可选的环境变量（带默认值）
+azure_api_key = get_env("AZURE_API_KEY", default="")
+debug_mode = get_env_bool("DEBUG", default=False)
+timeout = get_env_int("TIMEOUT", default=30)
+```
+
+#### 读取配置文件（推荐用于应用配置）
+
+```python
+from config_loader import load_config
+
+# 加载配置（自动根据 current_env.txt 选择环境）
+config = load_config()
+
+# 读取配置项
+output_dir = config.get("output", {}).get("base_dir", "./output")
+log_level = config.get("logging", {}).get("level", "INFO")
+redis_host = config.get("redis", {}).get("host", "localhost")
+```
+
+#### 在配置文件中引用环境变量
+
+配置文件支持 `${ENV_VAR_NAME}` 格式引用环境变量：
+
+`config/prod.yaml`:
+```yaml
+notifications:
+  wechat:
+    yfz: "${WECHAT_TOKEN_YFZ}"  # 从环境变量读取
+  wecom:
+    team1: "${WECOM_TOKEN_TEAM1}"
+```
+
+加载后会自动展开：
+```python
+config = load_config()
+token = config["notifications"]["wechat"]["yfz"]  # 自动展开为实际的 token 值
+```
+
+## 最佳实践
+
+### 1. 敏感信息 vs 应用配置
+
+**使用 `.env` 文件的场景**：
+- API 密钥、Token
+- 数据库密码、Redis 密码
+- 系统账户密码
+- 其他不应出现在版本库中的敏感信息
+
+**使用配置文件（`config/*.yaml`）的场景**：
+- 应用功能开关
+- 超时设置、重试次数等参数
+- 文件路径、目录配置
+- 日志级别、输出格式等
+- 非敏感的业务配置
+
+### 2. 环境隔离
+
+```bash
+# 切换到测试环境
+echo "test" > config/current_env.txt
+
+# 切换回生产环境
+echo "prod" > config/current_env.txt
+```
+
+### 3. 本地开发配置
+
+本地配置文件（`config/local.yaml`）用于覆盖环境配置，常见用途：
+
+- 使用本地路径替代网络共享路径（提升速度）
+- 启用详细日志（`DEBUG`）
+- 连接本地数据库/Redis（避免影响生产环境）
+- 临时关闭某些功能特性
+
+### 4. 团队协作
+
+**提交到版本库的文件**：
+- `.env.example` - 环境变量模板，不含真实密钥
+- `config/prod.yaml`, `config/test.yaml` - 环境配置
+- `config/local.yaml.example` - 本地配置示例
+
+**不提交到版本库的文件**（已在 `.gitignore` 中）：
+- `.env` - 包含真实密钥
+- `config/local.yaml`, `config/local.json` - 个人本地配置
+
+## 迁移指南
+
+### 从硬编码配置迁移
+
+**旧代码**：
+```python
+# 硬编码在代码中
+GEMINI_API_KEY = "AQ.Ab8RN6INSIgaGdgj5hmmYmi35QLw6K3likBz2bP37_I_F6V5aQ"
+OUTPUT_DIR = "./output"
+LOG_LEVEL = "INFO"
+```
+
+**新代码**：
+```python
+from env_loader import get_env
+from config_loader import load_config
+
+# 敏感信息从环境变量读取
+GEMINI_API_KEY = get_env("GEMINI_API_KEY", required=True)
+
+# 应用配置从配置文件读取
+config = load_config()
+OUTPUT_DIR = config.get("output", {}).get("base_dir", "./output")
+LOG_LEVEL = config.get("logging", {}).get("level", "INFO")
+```
+
+**配置文件**：
+
+`.env`:
+```bash
+GEMINI_API_KEY=AQ.Ab8RN6INSIgaGdgj5hmmYmi35QLw6K3likBz2bP37_I_F6V5aQ
+```
+
+`config/prod.yaml`:
+```yaml
+output:
+  base_dir: ./output
+logging:
+  level: INFO
+```
+
+### 向后兼容处理
+
+如果配置文件不存在，给出友好提示：
+
+```python
+from config_loader import load_config
+
+try:
+    config = load_config()
+except FileNotFoundError as e:
+    print(f"Configuration file not found: {e}")
+    print("Please set up the configuration files in the config/ directory")
+    # 使用默认配置或退出
+    config = {}  # 使用空配置或默认值
+```
+
+环境变量不存在时的友好提示：
+
+```python
+from env_loader import get_env, print_env_setup_guide, check_required_env_vars
+
+# 检查必需的环境变量
+missing_vars = check_required_env_vars(["GEMINI_API_KEY", "AZURE_API_KEY"])
+
+if missing_vars:
+    print(f"Missing required environment variables: {', '.join(missing_vars)}")
+    print_env_setup_guide()
+    raise ValueError("Please configure environment variables")
+```
+
+## 常见问题
+
+### Q1: `.env` 文件和 `config/local.yaml` 有什么区别？
+
+- `.env` 用于**敏感信息**（API 密钥、密码等），格式固定为 `KEY=VALUE`
+- `config/local.yaml` 用于**本地配置覆盖**（路径、参数等），支持嵌套结构
+
+### Q2: 如何验证配置是否生效？
+
+```python
+from env_loader import get_env
+from config_loader import load_config
+
+# 检查环境变量
+print(f"GEMINI_API_KEY: {get_env('GEMINI_API_KEY', default='<not set>')}")
+
+# 检查配置文件
+config = load_config()
+print(f"Environment: {config.get('environment', {}).get('name', 'unknown')}")
+print(f"Log level: {config.get('logging', {}).get('level', 'INFO')}")
+```
+
+### Q3: 配置文件支持哪些格式？
+
+- **环境变量**：`.env` 文件（`KEY=VALUE` 格式）
+- **配置文件**：`.yaml`、`.yml`、`.json`
+
+推荐使用 YAML 格式，因为：
+- 支持注释
+- 支持嵌套结构
+- 可读性更好
+
+### Q4: 如何在配置文件中引用环境变量？
+
+使用 `${ENV_VAR_NAME}` 格式：
+
+```yaml
+notifications:
+  wechat:
+    yfz: "${WECHAT_TOKEN_YFZ}"  # 引用环境变量
+```
+
+加载时会自动展开为环境变量的值。
+
+### Q5: 环境变量和配置文件冲突时如何处理？
+
+环境变量的优先级最高。例如：
+
+`.env`:
+```
+LOG_LEVEL=DEBUG
+```
+
+`config/prod.yaml`:
+```yaml
+logging:
+  level: INFO
+```
+
+如果在配置文件中引用环境变量：
+```yaml
+logging:
+  level: "${LOG_LEVEL}"  # 实际值为 DEBUG（环境变量）
+```
+
+## 完整示例
+
+参考 `config_usage_examples.py` 文件，包含：
+- 环境变量读取示例
+- 配置文件读取示例
+- 完整的初始化函数
+- 配置验证逻辑
+- 迁移指南
+
+## 安全建议
+
+1. **永远不要提交包含真实密钥的文件**
+   - `.env` 已在 `.gitignore` 中排除
+   - 提交前检查：`git status` 确保 `.env` 不在待提交列表中
+
+2. **定期轮换 API 密钥**
+   - 定期更新 `.env` 中的密钥
+   - 旧密钥在服务提供商处失效
+
+3. **最小权限原则**
+   - API 密钥只授予必需的权限
+   - 生产环境和开发环境使用不同的密钥
+
+4. **密钥泄露应急处理**
+   - 立即在服务提供商处撤销泄露的密钥
+   - 生成新密钥并更新 `.env` 文件
+   - 检查是否有未授权使用记录
+
+## 参考资料
+
+- 配置加载模块：`config_loader.py`
+- 环境变量加载模块：`env_loader.py`
+- 使用示例：`config_usage_examples.py`
+- 环境变量模板：`.env.example`
+- 本地配置示例：`config/local.yaml.example`

DroidBot/PLATFORM_GUIDE.md

@@ -0,0 +1,226 @@
+# DroidBot 多平台适配指南
+
+本文档介绍 DroidBot 的多平台架构设计和接口标准，帮助开发者扩展新平台支持。
+
+## 架构概述
+
+DroidBot 采用抽象工厂模式，将平台无关的核心逻辑与平台特定实现分离：
+
+```
+┌──────────────────────────────────────────────────────────┐
+│                    DroidBot Core                         │
+│  (input_policy.py, input_manager.py, utg.py)            │
+└─────────────────────────┬────────────────────────────────┘
+                          │ 调用抽象接口
+┌─────────────────────────▼────────────────────────────────┐
+│                  core/ 抽象层                            │
+│  AbstractDevice, AbstractDeviceState, AbstractInputEvent │
+│  PlatformFactory                                         │
+└─────────────────────────┬────────────────────────────────┘
+                          │ 平台实现
+┌─────────────────────────▼────────────────────────────────┐
+│               platforms/                                 │
+│  ├── android/     ← AndroidDevice, AndroidDeviceState    │
+│  ├── ios/         ← (预留)                               │
+│  └── windows/     ← (预留)                               │
+└──────────────────────────────────────────────────────────┘
+```
+
+## 核心接口
+
+### 1. AbstractDevice
+
+设备抽象基类，定义平台无关的设备操作接口。
+
+```python
+class AbstractDevice(ABC):
+    # === 必须实现的方法 ===
+    @abstractmethod
+    def set_up(self) -> None: ...
+    @abstractmethod
+    def connect(self) -> bool: ...
+    @abstractmethod
+    def disconnect(self) -> None: ...
+    @abstractmethod
+    def tear_down(self) -> None: ...
+    @abstractmethod
+    def check_connectivity(self) -> bool: ...
+    
+    @abstractmethod
+    def get_current_state(self) -> 'AbstractDeviceState': ...
+    @abstractmethod
+    def get_display_info(self, refresh: bool = False) -> Dict[str, Any]: ...
+    
+    @abstractmethod
+    def send_event(self, event: 'AbstractInputEvent') -> bool: ...
+    @abstractmethod
+    def take_screenshot(self, path: str) -> bool: ...
+    @abstractmethod
+    def unlock(self) -> None: ...
+    
+    @abstractmethod
+    def is_foreground(self) -> bool: ...
+    @abstractmethod
+    def start_app(self) -> bool: ...
+    @abstractmethod
+    def pull_back_to_app(self) -> bool: ...
+    
+    @property
+    @abstractmethod
+    def app_identifier(self) -> str: ...
+    @abstractmethod
+    def get_platform_name(self) -> str: ...
+    @abstractmethod
+    def get_device_info(self) -> Dict[str, Any]: ...
+    @abstractmethod
+    def create_event_from_dict(self, event_dict: Dict) -> 'AbstractInputEvent': ...
+```
+
+### 2. AbstractDeviceState
+
+设备状态抽象基类，定义 UI 状态信息接口。
+
+```python
+class AbstractDeviceState(ABC):
+    # === 必须实现的属性 ===
+    @property
+    @abstractmethod
+    def views(self) -> List[Dict[str, Any]]: ...
+    
+    @property
+    @abstractmethod
+    def state_str(self) -> str: ...
+    
+    @property
+    @abstractmethod
+    def foreground_page(self) -> Optional[str]: ...
+    
+    # === 必须实现的方法 ===
+    @abstractmethod
+    def get_possible_input(self) -> List['AbstractInputEvent']: ...
+    @abstractmethod
+    def to_dict(self) -> Dict[str, Any]: ...
+    @abstractmethod
+    def get_text_representation(self, merge_buttons: bool = False) -> tuple: ...
+```
+
+### 3. AbstractInputEvent
+
+输入事件抽象基类。
+
+```python
+class AbstractInputEvent(ABC):
+    @abstractmethod
+    def send(self, device: 'AbstractDevice') -> bool: ...
+    @abstractmethod
+    def to_dict(self) -> Dict[str, Any]: ...
+    @abstractmethod
+    def get_event_str(self, state: Optional['AbstractDeviceState'] = None) -> str: ...
+```
+
+## 统一视图数据结构
+
+各平台需将 UI 元素转换为统一格式：
+
+```python
+{
+    # 必需字段
+    "temp_id": int,                          # 临时 ID
+    "bounds": [[int, int], [int, int]],      # 边界框 [[left,top], [right,bottom]]
+    "class": str,                            # 元素类型
+    
+    # 文本相关
+    "text": Optional[str],                   # 显示文本
+    "content_description": Optional[str],   # 无障碍描述
+    
+    # 层级关系
+    "parent": Optional[int],                # 父节点 ID
+    "children": List[int],                  # 子节点 ID 列表
+    
+    # 交互属性
+    "enabled": bool,
+    "visible": bool,
+    "clickable": bool,
+    "scrollable": bool,
+    "editable": bool,
+    "checkable": bool,
+    "long_clickable": bool,
+    
+    # 平台特有
+    "resource_id": Optional[str]            # 平台元素标识
+}
+```
+
+## 平台映射表
+
+| 抽象接口 | Android | iOS | Windows |
+|---------|---------|-----|---------|
+| `foreground_page` | `package/activity` | `bundleId/viewController` | `exe_path/window_title` |
+| `app_identifier` | package_name | bundleId | exe_path/window_class |
+| `resource_id` | resource_id | accessibilityIdentifier | AutomationId |
+
+## 添加新平台步骤
+
+### 1. 创建平台目录
+
+```bash
+mkdir -p droidbot/platforms/ios
+```
+
+### 2. 实现核心类
+
+```python
+# platforms/ios/__init__.py
+from .ios_device import IOSDevice
+from .ios_device_state import IOSDeviceState
+from .ios_input_event import IOSTouchEvent, IOSKeyEvent, ...
+
+def register_ios_platform():
+    from ...core.platform_factory import PlatformFactory, Platform
+    
+    event_classes = {
+        'touch': IOSTouchEvent,
+        'key': IOSKeyEvent,
+        # ...
+    }
+    
+    PlatformFactory.register_platform(
+        Platform.IOS,
+        IOSDevice,
+        IOSDeviceState,
+        event_classes
+    )
+
+# 模块导入时自动注册
+register_ios_platform()
+```
+
+### 3. 添加平台枚举
+
+```python
+# core/platform_factory.py
+class Platform(Enum):
+    ANDROID = "android"
+    IOS = "ios"          # 新增
+    WINDOWS = "windows"  # 新增
+```
+
+### 4. 在 platforms/__init__.py 导入
+
+```python
+from .ios import IOSDevice, IOSDeviceState
+```
+
+## 事件类型
+
+所有平台必须支持的事件类型：
+
+| 事件类型 | 说明 | 工厂键名 |
+|---------|------|---------|
+| Touch | 点击 | `touch` |
+| LongTouch | 长按 | `long_touch` |
+| Swipe | 滑动 | `swipe` |
+| Scroll | 滚动 | `scroll` |
+| SetText | 输入文本 | `set_text` |
+| Key | 按键 | `key` |
+| KillApp | 终止应用 | `kill_app` |

DroidBot/README.md

@@ -0,0 +1,62 @@
+# DroidBot - 多平台自动化 UI 测试框架
+
+DroidBot 是一个轻量级的自动化 UI 测试框架，支持多平台（目前支持 Android）。通过智能的 UI 探索策略和可扩展的架构设计，帮助开发者进行应用测试和 UI 自动化。
+
+## 特性
+
+- **多平台支持**: 采用抽象工厂模式，易于扩展新平台
+- **智能探索**: 基于记忆引导的 UI 探索策略 (MemoryGuidedPolicy)
+- **CV 模式**: 支持计算机视觉识别 UI 元素
+- **GuiAgent 集成**: 可集成 LLM 驱动的 GUI 代理处理复杂场景
+- **脚本支持**: 支持自定义脚本控制测试流程
+
+## 快速开始
+
+### 安装
+
+```bash
+cd DroidBot
+pip install -e .
+```
+
+### 运行
+
+```bash
+python start.py -d <device_serial> -a <package_name> -o <output_dir>
+```
+
+### 主要参数
+
+| 参数 | 说明 |
+|------|------|
+| `-d` | 设备序列号 |
+| `-a` | 被测应用包名 |
+| `-o` | 输出目录 |
+| `-policy` | 探索策略 (memory_guided, manual, none) |
+| `-cv_mode` | 启用 CV 模式 |
+| `--enable_guiagent` | 启用 GuiAgent |
+
+## 架构
+
+```
+droidbot/
+├── core/                       # 核心抽象层
+│   ├── abstract_device.py      # 设备抽象基类
+│   ├── abstract_device_state.py # 状态抽象基类
+│   ├── abstract_input_event.py # 事件抽象基类
+│   └── platform_factory.py     # 平台工厂
+├── platforms/                  # 平台实现
+│   ├── android/               # Android 实现
+│   └── ios/                   # iOS 预留
+├── input_policy.py            # 输入策略
+├── input_manager.py           # 输入管理器
+└── utg.py                     # UI 转换图
+```
+
+## 扩展新平台
+
+参见 [PLATFORM_GUIDE.md](./PLATFORM_GUIDE.md) 了解如何添加新平台支持。
+
+## 许可证
+
+MIT License

DroidBot/__init__.py

@@ -0,0 +1,4 @@
+from .droidbot import DroidBot
+from .core import PlatformFactory, Platform, AbstractDevice
+
+__all__ = ['DroidBot', 'PlatformFactory', 'Platform', 'AbstractDevice']

DroidBot/core/__init__.py

@@ -0,0 +1,18 @@
+# Core module for platform-agnostic abstractions
+from .abstract_device import AbstractDevice
+from .abstract_device_state import AbstractDeviceState
+from .abstract_input_event import AbstractInputEvent, EventType
+from .abstract_app import AbstractApp
+from .platform_factory import PlatformFactory, Platform
+from .event_log import EventLog
+
+__all__ = [
+    'AbstractDevice',
+    'AbstractDeviceState', 
+    'AbstractInputEvent',
+    'AbstractApp',
+    'EventType',
+    'PlatformFactory',
+    'Platform',
+    'EventLog',
+]

DroidBot/core/abstract_app.py

@@ -0,0 +1,92 @@
+"""
+Abstract App Base Class
+Platform-agnostic application interface that all platform implementations must inherit.
+"""
+from abc import ABC, abstractmethod
+from typing import Optional, List, Any
+import logging
+
+
+class AbstractApp(ABC):
+    """
+    所有平台应用的抽象基类
+    
+    定义了应用操作的标准接口，包括：
+    - 应用标识
+    - 入口点管理
+    - 启动/停止命令
+    """
+    
+    def __init__(self, output_dir: Optional[str] = None):
+        """
+        初始化应用基类
+        
+        :param output_dir: 输出目录路径
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.output_dir = output_dir
+    
+    # ==================== 应用标识 ====================
+    
+    @property
+    @abstractmethod
+    def identifier(self) -> str:
+        """
+        获取应用唯一标识符
+        
+        Android: package_name
+        iOS: bundle_id
+        Windows: exe_path 或 window_class
+        
+        :return: 应用标识符
+        """
+        pass
+    
+    def get_package_name(self) -> str:
+        """
+        获取应用包名（兼容旧接口）
+        
+        :return: 应用标识符
+        """
+        return self.identifier
+    
+    # ==================== 入口点管理 ====================
+    
+    @property
+    def main_activity(self) -> Optional[str]:
+        """
+        获取应用主入口点（可选）
+        
+        Android: main_activity
+        iOS: main_scene
+        Windows: main_window_class
+        
+        :return: 主入口点，如果不适用则返回 None
+        """
+        return None
+    
+    @property
+    def activities(self) -> List[str]:
+        """
+        获取应用入口点列表（可选）
+        
+        Android: activities 列表
+        iOS: scenes 列表
+        
+        :return: 入口点列表
+        """
+        return []
+    
+    # ==================== 可选接口 ====================
+    
+    def get_start_with_profiling_intent(self, trace_file: str, sampling: Optional[int] = None) -> Any:
+        """
+        获取带性能分析的启动命令（可选）
+        
+        :param trace_file: 跟踪文件路径
+        :param sampling: 采样间隔
+        :return: 带性能分析的启动命令
+        :raises NotImplementedError: 如果平台不支持
+        """
+        raise NotImplementedError("This platform does not support profiling intent")
+

DroidBot/core/abstract_device.py

@@ -0,0 +1,282 @@
+"""
+Abstract Device Base Class
+Platform-agnostic device interface that all platform implementations must inherit.
+"""
+from abc import ABC, abstractmethod
+from typing import Optional, Dict, Any, List, Tuple
+import logging
+
+
+class AbstractDevice(ABC):
+    """
+    所有设备平台的抽象基类
+    
+    定义了设备操作的标准接口，包括：
+    - 设备连接/断开
+    - 状态获取
+    - 事件发送
+    - 屏幕操作
+    - 应用管理
+    """
+    
+    def __init__(self, output_dir: Optional[str] = None):
+        """
+        初始化设备基类
+        
+        :param output_dir: 输出目录路径
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.output_dir = output_dir
+        self.connected = False
+        self.display_info = None
+    
+    # ==================== 连接管理 ====================
+    
+    @abstractmethod
+    def set_up(self) -> None:
+        """设置设备连接前的准备工作"""
+        pass
+    
+    @abstractmethod
+    def connect(self) -> bool:
+        """
+        连接到设备
+        
+        :return: 连接是否成功
+        """
+        pass
+    
+    @abstractmethod
+    def disconnect(self) -> None:
+        """断开设备连接"""
+        pass
+    
+    @abstractmethod
+    def tear_down(self) -> None:
+        """清理设备资源"""
+        pass
+    
+    @abstractmethod
+    def check_connectivity(self) -> bool:
+        """
+        检查设备连接状态
+        
+        :return: 是否已连接
+        """
+        pass
+    
+    # ==================== 状态获取 ====================
+
+    @abstractmethod
+    def check_network(self, host: str = "8.8.8.8") -> bool:
+        """
+        检查设备内部网络是否连通
+        
+        :param host: 测试目标主机
+        :return: True 如果网络可用, False 否则
+        """
+        pass    
+    
+    @abstractmethod
+    def get_current_state(self) -> 'AbstractDeviceState':
+        """
+        获取当前设备状态
+        
+        :return: 设备状态对象
+        """
+        pass
+    
+    @abstractmethod
+    def get_display_info(self, refresh: bool = False) -> Dict[str, Any]:
+        """
+        获取显示信息
+        
+        :param refresh: 是否刷新缓存
+        :return: 包含 width, height, density 等信息的字典
+        """
+        pass
+    
+    def get_width(self, refresh: bool = False) -> int:
+        """获取屏幕宽度"""
+        display_info = self.get_display_info(refresh=refresh)
+        return display_info.get("width", 0)
+    
+    def get_height(self, refresh: bool = False) -> int:
+        """获取屏幕高度"""
+        display_info = self.get_display_info(refresh=refresh)
+        return display_info.get("height", 0)
+    
+    # ==================== 事件发送 ====================
+    
+    @abstractmethod
+    def send_event(self, event: 'AbstractInputEvent') -> bool:
+        """
+        发送输入事件到设备
+        
+        :param event: 输入事件对象
+        :return: 是否发送成功
+        """
+        pass
+    
+    # ==================== 屏幕操作 ====================
+    
+    @abstractmethod
+    def take_screenshot(self, path: str) -> bool:
+        """
+        截取屏幕
+        
+        :param path: 截图保存路径
+        :return: 是否成功
+        """
+        pass
+    
+    @abstractmethod
+    def unlock(self) -> None:
+        """解锁屏幕"""
+        pass
+    
+    # ==================== 应用管理 ====================
+    
+    @abstractmethod
+    def is_foreground(self) -> bool:
+        """
+        检查被测应用是否在前台
+        
+        :return: 是否在前台
+        """
+        pass
+    
+    @abstractmethod
+    def start_app(self) -> bool:
+        """
+        启动被测应用
+        
+        :return: 是否成功
+        """
+        pass
+    
+    def install_app(self) -> bool:
+        """
+        安装应用（可选接口，移动平台适用）
+        
+        :return: 是否成功
+        :raises NotImplementedError: 如果平台不支持
+        """
+        raise NotImplementedError("This platform does not support app installation")
+    
+    def uninstall_app(self) -> bool:
+        """
+        卸载应用（可选接口，移动平台适用）
+        
+        :return: 是否成功
+        :raises NotImplementedError: 如果平台不支持
+        """
+        raise NotImplementedError("This platform does not support app uninstallation")
+    
+    @abstractmethod
+    def pull_back_to_app(self) -> bool:
+        """
+        将被测应用拉回前台
+        
+        :return: 是否成功
+        """
+        pass
+
+    def get_redirect_target_info(self) -> Optional[Dict[str, Optional[str]]]:
+        """
+        获取应用离开前台后的目标信息（平台特定，可选接口）
+
+        返回格式:
+            {
+                "target": 跳转目标标识符,
+                "type": "app_store" | "launcher" | "other" | "unknown"
+            }
+
+        Android 可返回包名及分类；其他平台默认不实现，返回 None。
+        """
+        pass
+    
+    @property
+    @abstractmethod
+    def app_identifier(self) -> str:
+        """
+        获取被测应用的唯一标识符
+        
+        :return: 应用标识符（Android: package_name, Windows: window_class/exe_path）
+        """
+        pass
+    
+    # ==================== 设备信息 ====================
+    
+    @abstractmethod
+    def get_platform_name(self) -> str:
+        """
+        获取平台名称
+        
+        :return: 平台名称 (如 'android', 'windows')
+        """
+        pass
+
+    
+    # ==================== 可选功能: 性能分析 ====================
+    def run_initial_setup(self) -> bool:
+        """
+        使用 GuiAgent 处理应用的初始化任务（填写初始信息、进入游戏等）
+        此方法在 InputPolicy.start() 调用，只执行一次，且目前只用在cv模式
+        """
+        pass
+
+    def start_profiling(self, trace_file: str, sampling: Optional[int] = None) -> bool:
+        """
+        启动性能分析(可选接口,平台特定实现)
+        
+        :param trace_file: 跟踪文件路径
+        :param sampling: 采样间隔(可选)
+        :return: 是否成功启动
+        """
+        return False  # 默认不支持
+    
+    def stop_profiling(self, trace_file: str, output_path: str) -> bool:
+        """
+        停止性能分析(可选接口,平台特定实现)
+        
+        :param trace_file: 跟踪文件路径
+        :param output_path: 输出路径
+        :return: 是否成功停止
+        """
+        return False  # 默认不支持
+    
+    def get_traffic_domains(self, remote_dir: str) -> Optional[str]:
+        """
+        获取最新的流量域名日志文件内容(可选接口)
+        
+        :param remote_dir: 远程日志目录
+        :return: 日志内容或 None
+        """
+        return None
+    
+    @property
+    def captured_traffic_dir(self) -> Optional[str]:
+        """
+        获取捕获的流量日志存储目录
+        
+        :return: 目录路径或 None
+        """
+        return None
+    
+    def is_traffic_capture_running(self) -> bool:
+        """
+        检查流量抓包工具是否正在运行(可选接口)
+        
+        :return: True 如果正在运行
+        """
+        raise NotImplementedError("This platform does not support traffic capture")
+    
+    def restart_traffic_capture(self, package_name: str) -> None:
+        """
+        重启流量抓包工具(可选接口)
+        
+        :param package_name: 需要抓包的应用标识符
+        """
+        raise NotImplementedError("This platform does not support traffic capture")
+    

DroidBot/core/abstract_device_state.py

@@ -0,0 +1,256 @@
+"""
+Abstract Device State Base Class
+Platform-agnostic device state interface.
+"""
+from abc import ABC, abstractmethod
+from typing import Optional, Dict, Any, List, Set
+try:
+    from typing import TypedDict
+except ImportError:
+    from typing_extensions import TypedDict
+import os
+
+
+class ViewDict(TypedDict, total=False):
+    """
+    统一的视图/控件字典结构
+    
+    所有平台（Android/iOS/Web）和来源（Accessibility Tree/CV）
+    都必须输出此格式，确保事件处理代码可以统一处理。
+    """
+    # === 必需字段 ===
+    bounds: List[List[int]]  # [[x1, y1], [x2, y2]] 坐标
+    text: str                # 文本内容
+    visible: bool            # 是否可见
+    enabled: bool            # 是否启用
+    clickable: bool          # 是否可点击
+    editable: bool           # 是否可编辑
+    scrollable: bool         # 是否可滚动
+    children: List[int]      # 子节点索引列表
+    view_str: str            # 唯一标识符（自动生成）
+    
+    # === 可选字段 ===
+    content_description: str  # 无障碍描述
+    resource_id: str          # 资源ID
+    class_name: str           # 控件类名（避免 'class' 关键字）
+    temp_id: int              # 临时索引
+    parent: int               # 父节点索引
+    source: str               # 来源: 'accessibility' 或 'cv'
+    
+    # === 扩展字段（特定平台可能需要）===
+    signature: str            # 内容签名
+    long_clickable: bool      # 是否可长按
+    checkable: bool           # 是否可勾选
+    checked: bool             # 是否已勾选
+    selected: bool            # 是否已选中
+
+
+class AbstractDeviceState(ABC):
+    """
+    设备状态抽象基类
+    
+    定义了设备状态的标准接口，包括：
+    - 视图/控件信息
+    - 状态标识
+    - 可能的输入事件
+    - 前台活动信息
+    """
+    
+    def __init__(self, device: 'AbstractDevice', tag: Optional[str] = None, 
+                 screenshot_path: Optional[str] = None):
+        """
+        初始化设备状态
+        
+        :param device: 设备对象
+        :param tag: 状态标签
+        :param screenshot_path: 截图路径
+        """
+        self.device = device
+        self._screenshot_path = screenshot_path
+        
+        if tag is None:
+            from datetime import datetime
+            tag = datetime.now().strftime("%Y-%m-%d_%H%M%S")
+        self.tag = tag
+        
+        # 缓存的状态信息
+        self._state_str = None
+        self._possible_events = None
+        self._views = None
+    
+    # ==================== 视图信息 ====================
+    
+    @property
+    @abstractmethod
+    def views(self) -> List[Dict[str, Any]]:
+        """
+        获取当前界面的所有视图/控件元素
+        
+        :return: 视图字典列表，每个字典包含控件的属性信息
+        """
+        pass
+    
+    @property
+    def view_tree(self) -> Dict[str, Any]:
+        """
+        获取视图树结构（可选实现）
+        
+        :return: 视图树字典
+        """
+        return {}
+    
+    @property
+    def cv_views(self) -> List['ViewDict']:
+        """
+        获取 CV 检测到的视图列表
+        
+        :return: CV 视图列表，符合 ViewDict 格式
+        """
+        return []
+    
+    # ==================== 状态标识 ====================
+    
+    @property
+    @abstractmethod
+    def state_str(self) -> str:
+        """
+        获取状态的唯一标识字符串
+        
+        :return: 状态标识字符串（通常是 MD5 哈希）
+        """
+        pass
+    
+    @property
+    def structure_str(self) -> str:
+        """
+        获取状态的结构标识（忽略内容）
+        
+        :return: 结构标识字符串
+        """
+        return self.state_str
+
+    @property
+    def search_content(self) -> str:
+        """
+        获取用于搜索的文本内容（可选，用于 UTG 可视化）
+        
+        :return: 搜索内容字符串，默认返回空字符串
+        """
+        return ""
+    
+    # ==================== 输入事件 ====================
+    
+    @abstractmethod
+    def get_possible_input(self) -> List['AbstractInputEvent']:
+        """
+        获取当前状态可能的输入事件列表
+        
+        :return: 可能的输入事件列表
+        """
+        pass
+    
+    # ==================== 活动/页面信息 ====================
+    
+    @property
+    @abstractmethod
+    def foreground_page(self) -> Optional[str]:
+        """
+        获取前台页面/窗口标识
+        
+        :return: 前台页面名称，如果无法获取则返回 None
+        """
+        pass
+    
+    # ==================== 截图 ====================
+    
+    @property
+    def screenshot_path(self) -> Optional[str]:
+        """获取截图路径"""
+        return self._screenshot_path
+    
+    @screenshot_path.setter
+    def screenshot_path(self, value: str):
+        """设置截图路径"""
+        self._screenshot_path = value
+    
+    # ==================== 屏幕尺寸 ====================
+    
+    @property
+    def width(self) -> int:
+        """获取屏幕宽度"""
+        return self.device.get_width()
+    
+    @property
+    def height(self) -> int:
+        """获取屏幕高度"""
+        return self.device.get_height()
+    
+    # ==================== 序列化 ====================
+    
+    @abstractmethod
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        序列化为字典
+        
+        :return: 状态信息字典
+        """
+        pass
+    
+    def to_json(self) -> str:
+        """
+        序列化为 JSON 字符串
+        
+        :return: JSON 字符串
+        """
+        import json
+        return json.dumps(self.to_dict(), indent=2)
+    
+    # ==================== 状态保存 ====================
+    
+    def save2dir(self, output_dir: Optional[str] = None) -> None:
+        """
+        保存状态到目录，使用 flush + fsync 确保数据落盘
+
+        :param output_dir: 输出目录，默认使用设备的输出目录
+        """
+        try:
+            if output_dir is None:
+                if self.device.output_dir is None:
+                    return
+                output_dir = os.path.join(self.device.output_dir, "states")
+
+            if not os.path.exists(output_dir):
+                os.makedirs(output_dir)
+
+            # 保存状态 JSON
+            dest_state_json_path = os.path.join(output_dir, f"state_{self.tag}.json")
+            with open(dest_state_json_path, "w") as f:
+                f.write(self.to_json())
+                f.flush()
+                os.fsync(f.fileno())
+
+            # 复制截图
+            if self.screenshot_path and os.path.exists(self.screenshot_path):
+                import shutil
+                ext = os.path.splitext(self.screenshot_path)[1]
+                dest_screenshot_path = os.path.join(output_dir, f"screen_{self.tag}{ext}")
+                if os.path.abspath(self.screenshot_path) != os.path.abspath(dest_screenshot_path):
+                    shutil.copyfile(self.screenshot_path, dest_screenshot_path)
+                    self._screenshot_path = dest_screenshot_path
+
+        except Exception as e:
+            self.device.logger.error(f"Error saving state: {e}")
+    
+    # ==================== 辅助方法 ====================
+    
+    @staticmethod
+    def get_view_center(view_dict: Dict[str, Any]) -> tuple:
+        """
+        获取视图中心点坐标
+        
+        :param view_dict: 视图字典
+        :return: (x, y) 坐标元组
+        """
+        bounds = view_dict.get('bounds', [[0, 0], [0, 0]])
+        return (bounds[0][0] + bounds[1][0]) / 2, (bounds[0][1] + bounds[1][1]) / 2
+    

DroidBot/core/abstract_input_event.py

@@ -0,0 +1,381 @@
+"""
+Abstract Input Event Base Class
+Platform-agnostic input event interface.
+"""
+from abc import ABC, abstractmethod
+from typing import Optional, Dict, Any, List
+from enum import Enum
+import json
+
+
+class EventType(Enum):
+    """事件类型枚举 - 仅包含跨平台通用事件"""
+    # 基础交互事件
+    TOUCH = "touch"
+    LONG_TOUCH = "long_touch"
+    SWIPE = "swipe"
+    SCROLL = "scroll"
+    
+    # 文本输入事件
+    SET_TEXT = "set_text"
+    
+    # 按键事件
+    KEY = "key"
+    
+    # 应用控制事件（平台无关）
+    LAUNCH_APP = "launch_app"    # 启动应用
+    KILL_APP = "kill_app"        # 强制终止应用
+    INTENT = "intent"            # Android Intent事件
+    
+    # 特殊事件
+    MANUAL = "manual"
+    EXIT = "exit"
+    
+    # 选择事件
+    SELECT = "select"
+    UNSELECT = "unselect"
+
+
+class AbstractInputEvent(ABC):
+    """
+    输入事件抽象基类
+    
+    定义了所有输入事件的标准接口，包括：
+    - 事件发送
+    - 事件序列化
+    - 事件字符串表示
+    """
+    
+    def __init__(self, event_type: EventType):
+        """
+        初始化输入事件
+        
+        :param event_type: 事件类型
+        """
+        self.event_type = event_type
+        self.log_lines = None
+    
+    # ==================== 事件发送 ====================
+    
+    @abstractmethod
+    def send(self, device: 'AbstractDevice') -> bool:
+        """
+        发送事件到设备
+        
+        :param device: 设备对象
+        :return: 是否发送成功
+        """
+        pass
+    
+    # ==================== 序列化 ====================
+    
+    @abstractmethod
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        序列化为字典
+        
+        :return: 事件信息字典
+        """
+        pass
+    
+    def to_json(self) -> str:
+        """
+        序列化为 JSON 字符串
+        
+        :return: JSON 字符串
+        """
+        return json.dumps(self.to_dict())
+    
+    def __str__(self) -> str:
+        return self.to_json()
+    
+    # ==================== 事件描述 ====================
+    
+    @abstractmethod
+    def get_event_str(self, state: Optional['AbstractDeviceState'] = None) -> str:
+        """
+        获取事件的字符串描述
+        
+        :param state: 可选的设备状态对象
+        :return: 事件描述字符串
+        """
+        pass
+    
+    # ==================== 视图信息 ====================
+    
+    def get_views(self) -> List[Dict[str, Any]]:
+        """
+        获取事件关联的视图列表
+        
+        :return: 视图字典列表
+        """
+        return []
+
+
+# ==================== 基础事件类型 ====================
+
+class BaseTouchEvent(AbstractInputEvent):
+    """触摸事件基类"""
+    
+    def __init__(self, x: Optional[int] = None, y: Optional[int] = None, 
+                 view: Optional[Dict[str, Any]] = None):
+        """
+        初始化触摸事件
+        
+        :param x: X 坐标
+        :param y: Y 坐标
+        :param view: 目标视图（如果提供，将使用视图中心点）
+        """
+        super().__init__(EventType.TOUCH)
+        self.x = x
+        self.y = y
+        self.view = view
+        
+        # 如果提供了视图，使用视图中心点
+        if view is not None and (x is None or y is None):
+            from .abstract_device_state import AbstractDeviceState
+            center = AbstractDeviceState.get_view_center(view)
+            self.x = int(center[0])
+            self.y = int(center[1])
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """发送事件 - 需要由平台特定子类实现"""
+        raise NotImplementedError("BaseTouchEvent.send() must be implemented by platform-specific subclass")
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "event_type": self.event_type.value,
+            "x": self.x,
+            "y": self.y,
+            "view": self.view
+        }
+    
+    def get_event_str(self, state: Optional['AbstractDeviceState'] = None) -> str:
+        if self.view is not None:
+            view_str = self.view.get('view_str', str(self.view))
+            return f"Touch({view_str})"
+        return f"Touch({self.x}, {self.y})"
+    
+
+
+class BaseLongTouchEvent(AbstractInputEvent):
+    """长按事件基类"""
+    
+    def __init__(self, x: Optional[int] = None, y: Optional[int] = None,
+                 view: Optional[Dict[str, Any]] = None, duration: int = 2000):
+        """
+        初始化长按事件
+        
+        :param x: X 坐标
+        :param y: Y 坐标
+        :param view: 目标视图
+        :param duration: 长按持续时间（毫秒）
+        """
+        super().__init__(EventType.LONG_TOUCH)
+        self.x = x
+        self.y = y
+        self.view = view
+        self.duration = duration
+        
+        if view is not None and (x is None or y is None):
+            from .abstract_device_state import AbstractDeviceState
+            center = AbstractDeviceState.get_view_center(view)
+            self.x = int(center[0])
+            self.y = int(center[1])
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """发送事件 - 需要由平台特定子类实现"""
+        raise NotImplementedError("BaseLongTouchEvent.send() must be implemented by platform-specific subclass")
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "event_type": self.event_type.value,
+            "x": self.x,
+            "y": self.y,
+            "duration": self.duration,
+            "view": self.view
+        }
+    
+    def get_event_str(self, state: Optional['AbstractDeviceState'] = None) -> str:
+        if self.view is not None:
+            view_str = self.view.get('view_str', str(self.view))
+            return f"LongTouch({view_str})"
+        return f"LongTouch({self.x}, {self.y}, {self.duration}ms)"
+
+
+
+class BaseSwipeEvent(AbstractInputEvent):
+    """滑动事件基类"""
+    
+    def __init__(self, start_x: int, start_y: int, end_x: int, end_y: int,
+                 duration: int = 500, view: Optional[Dict[str, Any]] = None):
+        """
+        初始化滑动事件
+        
+        :param start_x: 起始 X 坐标
+        :param start_y: 起始 Y 坐标
+        :param end_x: 结束 X 坐标
+        :param end_y: 结束 Y 坐标
+        :param duration: 滑动持续时间（毫秒）
+        :param view: 关联的视图字典（可选）
+        """
+        super().__init__(EventType.SWIPE)
+        self.start_x = start_x
+        self.start_y = start_y
+        self.end_x = end_x
+        self.end_y = end_y
+        self.duration = duration
+        self.view = view
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """发送事件 - 需要由平台特定子类实现"""
+        raise NotImplementedError("BaseSwipeEvent.send() must be implemented by platform-specific subclass")
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "event_type": self.event_type.value,
+            "start_x": self.start_x,
+            "start_y": self.start_y,
+            "end_x": self.end_x,
+            "end_y": self.end_y,
+            "duration": self.duration,
+            "view": self.view
+        }
+    
+    def get_event_str(self, state: Optional['AbstractDeviceState'] = None) -> str:
+        if self.view is not None:
+            view_str = self.view.get('view_str', str(self.view))
+            return f"Swipe({view_str}, {self.start_x},{self.start_y} -> {self.end_x},{self.end_y})"
+        return f"Swipe({self.start_x},{self.start_y} -> {self.end_x},{self.end_y})"
+
+
+
+class BaseScrollEvent(AbstractInputEvent):
+    """滚动事件基类"""
+    
+    DIRECTION_UP = "up"
+    DIRECTION_DOWN = "down"
+    DIRECTION_LEFT = "left"
+    DIRECTION_RIGHT = "right"
+    
+    def __init__(self, direction: str = "down", view: Optional[Dict[str, Any]] = None):
+        """
+        初始化滚动事件
+        
+        :param direction: 滚动方向 ('up', 'down', 'left', 'right')
+        :param view: 目标视图（可选）
+        """
+        super().__init__(EventType.SCROLL)
+        self.direction = direction
+        self.view = view
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """发送事件 - 需要由平台特定子类实现"""
+        raise NotImplementedError("BaseScrollEvent.send() must be implemented by platform-specific subclass")
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "event_type": self.event_type.value,
+            "direction": self.direction,
+            "view": self.view
+        }
+    
+    def get_event_str(self, state: Optional['AbstractDeviceState'] = None) -> str:
+        if self.view is not None:
+            view_str = self.view.get('view_str', str(self.view))
+            return f"Scroll({view_str}, {self.direction})"
+        return f"Scroll({self.direction})"
+
+
+
+class BaseSetTextEvent(AbstractInputEvent):
+    """文本输入事件基类"""
+    
+    def __init__(self, text: str, view: Optional[Dict[str, Any]] = None):
+        """
+        初始化文本输入事件
+        
+        :param text: 要输入的文本
+        :param view: 目标视图
+        """
+        super().__init__(EventType.SET_TEXT)
+        self.text = text
+        self.view = view
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """发送事件 - 需要由平台特定子类实现"""
+        raise NotImplementedError("BaseSetTextEvent.send() must be implemented by platform-specific subclass")
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "event_type": self.event_type.value,
+            "text": self.text,
+            "view": self.view
+        }
+    
+    def get_event_str(self, state: Optional['AbstractDeviceState'] = None) -> str:
+        if self.view is not None:
+            view_str = self.view.get('view_str', str(self.view))
+            return f"SetText({view_str}, '{self.text}')"
+        return f"SetText('{self.text}')"
+
+
+class BaseKeyEvent(AbstractInputEvent):
+    """按键事件基类"""
+    
+    # 通用按键名称
+    KEY_BACK = "BACK"
+    KEY_HOME = "HOME"
+    KEY_MENU = "MENU"
+    KEY_ENTER = "ENTER"
+    KEY_ESCAPE = "ESCAPE"
+    
+    def __init__(self, key_name: str):
+        """
+        初始化按键事件
+        
+        :param key_name: 按键名称
+        """
+        super().__init__(EventType.KEY)
+        self.key_name = key_name
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """发送事件 - 需要由平台特定子类实现"""
+        raise NotImplementedError("BaseKeyEvent.send() must be implemented by platform-specific subclass")
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "event_type": self.event_type.value,
+            "key_name": self.key_name
+        }
+    
+    def get_event_str(self, state: Optional['AbstractDeviceState'] = None) -> str:
+        return f"Key({self.key_name})"
+    
+
+
+class BaseKillAppEvent(AbstractInputEvent):
+    """终止应用事件基类"""
+    
+    def __init__(self, app=None):
+        """
+        初始化终止应用事件
+        
+        :param app: 要终止的应用
+        """
+        super().__init__(EventType.KILL_APP)
+        self.app = app
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """发送事件 - 需要由平台特定子类实现"""
+        raise NotImplementedError("BaseKillAppEvent.send() must be implemented by platform-specific subclass")
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "event_type": self.event_type.value,
+            "app": str(self.app) if self.app else None
+        }
+    
+    def get_event_str(self, state: Optional['AbstractDeviceState'] = None) -> str:
+        return f"KillApp({self.app})"
+

DroidBot/core/event_log.py

@@ -0,0 +1,161 @@
+"""
+Event Log
+Platform-agnostic event logging for recording device interactions.
+"""
+import json
+import os
+import time
+
+
+class EventLog:
+    """
+    事件日志类
+    
+    用于记录设备交互事件，包括开始状态、结束状态、性能分析等。
+    """
+
+    def __init__(self, device, app, event, profiling_method=None, tag=None):
+        """
+        初始化事件日志
+        
+        :param device: 设备对象
+        :param app: 应用对象
+        :param event: 事件对象
+        :param profiling_method: 性能分析方法
+        :param tag: 日志标签
+        """
+        self.device = device
+        self.app = app
+        self.event = event
+        if tag is None:
+            from datetime import datetime
+            tag = datetime.now().strftime("%Y-%m-%d_%H%M%S")
+        self.tag = tag
+
+        self.from_state = None
+        self.to_state = None
+        self.event_str = None
+
+        self.profiling_method = profiling_method
+        self.trace_remote_file = "/data/local/tmp/event.trace"
+        self.is_profiling = False
+        self.sampling = None
+        
+        # sampling feature was added in Android 5.0 (API level 21)
+        self.sampling = None
+        if profiling_method is not None and str(profiling_method) != "full":
+            # Check device capability safely
+            if hasattr(device, 'get_sdk_version') and device.get_sdk_version() >= 21:
+                try:
+                    self.sampling = int(profiling_method)
+                except:
+                    pass
+
+    def to_dict(self):
+        """序列化为字典"""
+        return {
+            "tag": self.tag,
+            "event": self.event.to_dict(),
+            "start_state": self.from_state.state_str if self.from_state else None,
+            "stop_state": self.to_state.state_str if self.to_state else None,
+            "event_str": self.event_str
+        }
+
+    def save2dir(self, output_dir=None):
+        """保存事件到目录"""
+        if output_dir is None:
+            if self.device.output_dir is None:
+                return
+            else:
+                output_dir = os.path.join(self.device.output_dir, "events")
+        try:
+            if not os.path.exists(output_dir):
+                os.makedirs(output_dir)
+            event_json_file_path = "%s/event_%s.json" % (output_dir, self.tag)
+            with open(event_json_file_path, "w") as f:
+                json.dump(self.to_dict(), f, indent=2)
+        except Exception as e:
+            self.device.logger.error("Saving event to dir failed, %s" % e)
+
+    def save_views(self, output_dir=None):
+        """保存视图"""
+        views = self.event.get_views()
+        if views and self.from_state and hasattr(self.from_state, 'save_view_img'):
+            for view_dict in views:
+                self.from_state.save_view_img(view_dict=view_dict, output_dir=output_dir)
+
+    def is_start_event(self):
+        """检查是否是启动事件"""
+        from .abstract_input_event import EventType
+        if hasattr(self.event, 'event_type') and self.event.event_type == EventType.INTENT:
+            intent_cmd = getattr(self.event, 'intent', '')
+            if intent_cmd and hasattr(self.app, 'get_package_name'):
+                if "start" in str(intent_cmd) and self.app.get_package_name() in str(intent_cmd):
+                    return True
+        return False
+
+    def start(self):
+        """开始发送事件"""
+        if hasattr(self.device, '_last_state') and self.device._last_state is not None:
+            self.from_state = self.device._last_state
+        else:
+            print("Warning: No last state available, using current state as start state.")
+            self.from_state = self.device.get_current_state()
+        self.start_profiling()
+        self.event_str = self.event.get_event_str(self.from_state)
+        print(f"[DroidBot] Action: {self.event_str}")  # 添加打印以确保可见
+        self.device.logger.info("Action: %s" % self.event_str)
+        self.device.send_event(self.event)
+
+    def start_profiling(self):
+        """开始性能分析"""
+        if self.profiling_method is None:
+            return
+        if self.is_profiling:
+            return
+        
+        # 尝试使用设备提供的统一接口
+        if hasattr(self.device, 'start_profiling'):
+            success = self.device.start_profiling(self.trace_remote_file, self.sampling)
+            if success:
+                self.is_profiling = True
+                return
+                
+        # 处理启动事件的特殊情况 (Legacy Android support)
+        # 如果是启动事件，并且应用尚未运行，可能需要修改 Intent
+        if self.is_start_event():
+            if hasattr(self.app, 'get_start_with_profiling_intent'):
+                start_intent = self.app.get_start_with_profiling_intent(self.trace_remote_file, self.sampling)
+                if hasattr(self.event, 'intent') and hasattr(start_intent, 'get_cmd'):
+                    self.event.intent = start_intent.get_cmd()
+            self.is_profiling = True
+
+    def stop(self):
+        """结束发送事件"""
+        self.stop_profiling()
+        self.to_state = self.device.get_current_state()
+        if hasattr(self.device, '_last_state'):
+            self.device._last_state = self.to_state
+        self.save2dir()
+        self.save_views()
+
+    def stop_profiling(self, output_dir=None):
+        """停止性能分析"""
+        if self.profiling_method is None:
+            return
+        if not self.is_profiling:
+            return
+            
+        if output_dir is None:
+            if self.device.output_dir is None:
+                return
+            else:
+                output_dir = os.path.join(self.device.output_dir, "events")
+        if not os.path.exists(output_dir):
+            os.makedirs(output_dir)
+        event_trace_local_path = "%s/event_trace_%s.trace" % (output_dir, self.tag)
+            
+        # 尝试使用设备提供的统一接口
+        if hasattr(self.device, 'stop_profiling'):
+            self.device.stop_profiling(self.trace_remote_file, event_trace_local_path)
+

DroidBot/core/platform_factory.py

@@ -0,0 +1,131 @@
+"""
+Platform Factory
+Factory class for creating platform-specific implementations.
+"""
+from enum import Enum
+from typing import Type, Dict, Any, Optional
+import logging
+
+
+class Platform(Enum):
+    """支持的平台枚举"""
+    ANDROID = "android"
+    IOS = "ios"
+    WINDOWS = "windows"
+    WEB = "web"
+    # 未来可扩展
+    # MACOS = "macos"
+    # LINUX = "linux"
+
+
+class PlatformFactory:
+    """
+    平台工厂类
+    
+    用于注册和创建平台特定的实现。
+    """
+    
+    _logger = logging.getLogger("PlatformFactory")
+    
+    # 注册的平台实现
+    _device_classes: Dict[Platform, Type] = {}
+    _state_classes: Dict[Platform, Type] = {}
+    _event_classes: Dict[Platform, Dict[str, Type]] = {}
+    
+    @classmethod
+    def register_device(cls, platform: Platform, device_class: Type) -> None:
+        """
+        注册设备类
+        
+        :param platform: 平台类型
+        :param device_class: 设备类
+        """
+        cls._device_classes[platform] = device_class
+        cls._logger.info(f"Registered device class for {platform.value}: {device_class.__name__}")
+    
+    @classmethod
+    def register_state(cls, platform: Platform, state_class: Type) -> None:
+        """
+        注册状态类
+        
+        :param platform: 平台类型
+        :param state_class: 状态类
+        """
+        cls._state_classes[platform] = state_class
+        cls._logger.info(f"Registered state class for {platform.value}: {state_class.__name__}")
+    
+    @classmethod
+    def register_events(cls, platform: Platform, event_classes: Dict[str, Type]) -> None:
+        """
+        注册事件类
+        
+        :param platform: 平台类型
+        :param event_classes: 事件类字典 {事件类型名: 事件类}
+        """
+        cls._event_classes[platform] = event_classes
+        cls._logger.info(f"Registered {len(event_classes)} event classes for {platform.value}")
+    
+    @classmethod
+    def register_platform(cls, platform: Platform,
+                         device_class: Type,
+                         state_class: Type,
+                         event_classes: Dict[str, Type]) -> None:
+        """
+        一次性注册平台所有类
+        
+        :param platform: 平台类型
+        :param device_class: 设备类
+        :param state_class: 状态类
+        :param event_classes: 事件类字典
+        """
+        cls.register_device(platform, device_class)
+        cls.register_state(platform, state_class)
+        cls.register_events(platform, event_classes)
+    
+    # ==================== 工厂方法 ====================
+    
+    @classmethod
+    def _to_platform(cls, platform_input: Any) -> Platform:
+        """Helper to convert string or Enum to Platform Enum"""
+        if isinstance(platform_input, Platform):
+            return platform_input
+        try:
+            return Platform(platform_input)
+        except ValueError:
+            raise ValueError(f"Invalid platform: {platform_input}. Available: {[p.value for p in cls._device_classes.keys()]}")
+
+    @classmethod
+    def create_device(cls, platform: Any, **kwargs) -> 'AbstractDevice':
+        """
+        创建设备实例
+        
+        :param platform: 平台类型 (Enum 或 str)
+        :param kwargs: 设备初始化参数
+        :return: 设备实例
+        :raises ValueError: 如果平台未注册
+        """
+        platform_enum = cls._to_platform(platform)
+        if platform_enum not in cls._device_classes:
+            available = [p.value for p in cls._device_classes.keys()]
+            raise ValueError(
+                f"Platform '{platform_enum.value}' is not registered. "
+                f"Available platforms: {available}"
+            )
+        return cls._device_classes[platform_enum](**kwargs)
+    
+    @classmethod
+    def get_event_class(cls, platform: Any, event_type: str) -> Type:
+        """
+        获取事件类
+        
+        :param platform: 平台类型 (Enum 或 str)
+        :param event_type: 事件类型名
+        :return: 事件类
+        """
+        platform_enum = cls._to_platform(platform)
+        if platform_enum not in cls._event_classes:
+            raise ValueError(f"No event classes registered for platform: {platform_enum.value}")
+        if event_type not in cls._event_classes[platform_enum]:
+            raise ValueError(f"Event type '{event_type}' not found for platform: {platform_enum.value}")
+        return cls._event_classes[platform_enum][event_type]
+    

DroidBot/cv/cv.py

@@ -0,0 +1,320 @@
+# This file is created by Minyi Liu (GitHub ID: MiniMinyi)
+# The hash algorithm is copied from:
+# https://github.com/hjaurum/DHash/blob/master/dHash.py
+
+def load_image_from_path(img_path):
+    """
+    Load an image from path
+    :param img_path: The path to the image
+    :return:
+    """
+    import cv2
+    import numpy as np
+    # Fix for reading images with Chinese paths on Windows
+    # Use IMREAD_COLOR to ensure 3 channels (BGR) as cv2.imread does by default
+    return cv2.imdecode(np.fromfile(img_path, dtype=np.uint8), cv2.IMREAD_COLOR)
+
+
+def load_image_from_buf(img_bytes):
+    """
+    Load an image from a byte array
+    :param img_bytes: The byte array of an image
+    :return:
+    """
+    import cv2
+    import numpy
+    img_bytes = numpy.frombuffer(img_bytes, dtype=numpy.uint8)
+    return cv2.imdecode(img_bytes, cv2.IMREAD_UNCHANGED)
+
+
+# -------------------------------------------------------------
+# 辅助函数：将 OpenCV 图像对象编码为 Base64
+# -------------------------------------------------------------
+import cv2
+import numpy as np
+import base64
+import os
+from typing import Optional, List, Tuple, Dict, Any
+
+# 全局变量用于缓存模型实例
+_model_handler = None
+_model_config = None
+
+class OmniParserModelManager:
+    """
+    OmniParser模型管理器，使用单例模式确保模型只初始化一次
+    """
+    _instance = None
+    _handler = None
+    
+    def __new__(cls):
+        if cls._instance is None:
+            cls._instance = super(OmniParserModelManager, cls).__new__(cls)
+        return cls._instance
+    
+    def __init__(self):
+        if self._handler is None:
+            self._initialize_model()
+    
+    def _initialize_model(self):
+        """初始化OmniParser模型"""
+        from .handler import EndpointHandler
+        
+        # 默认配置
+        config = {
+            "model_dir": os.path.join(os.path.dirname(__file__), "."),
+            "mode": "custom",
+            "ocr_languages": ["ch_sim", "en"],
+            "image_size": 1280,
+            "bbox_threshold": 0.2,
+            "iou_threshold": 0.5,
+        }
+        
+        print("初始化OmniParser模型...")
+        self._handler = EndpointHandler(
+            model_dir=config["model_dir"],
+            enable_yolo=True, 
+            enable_ocr=True, 
+            enable_caption=False,
+            ocr_languages=config["ocr_languages"],
+        )
+        self._config = config
+        print("OmniParser模型初始化完成")
+    
+    def get_handler(self):
+        """获取模型处理器实例"""
+        return self._handler
+    
+    def get_config(self):
+        """获取模型配置"""
+        return self._config
+
+
+def get_model_manager() -> OmniParserModelManager:
+    """获取模型管理器单例实例"""
+    return OmniParserModelManager()
+
+
+def load_image_from_buf(img_bytes):
+    """
+    Load image from bytes
+    :param img_bytes: bytes of image
+    :return: numpy.ndarray, representing an image in opencv
+    """
+    import cv2
+    import numpy
+    img_bytes = numpy.frombuffer(img_bytes, dtype=numpy.uint8)
+    return cv2.imdecode(img_bytes, cv2.IMREAD_UNCHANGED)
+
+
+def _encode_cv_image(img):
+    """
+    将 OpenCV 图像 (numpy.ndarray) 编码为 Base64 字符串。
+    """
+    import cv2
+    import base64
+    
+    # 将图像编码为 PNG 格式的字节流
+    success, encoded_image = cv2.imencode(".png", img)
+    if not success:
+        raise ValueError("无法将图像编码为 Base64")
+        
+    # 转换为 Base64 字符串 (不带 MIME 头)
+    raw_b64 = base64.b64encode(encoded_image).decode("ascii")
+    
+    # 加上 MIME 头
+    return f"data:image/png;base64,{raw_b64}"
+
+
+def enhance_contrast_gamma(image, gamma=1.0):
+    """
+    Apply Power Law Transformation to the image.
+    Output = (Input/255) ^ gamma * 255
+    
+    Gamma > 1.0 will darken the image (crush shadows).
+    Gamma < 1.0 will brighten the image.
+    """
+    if gamma == 1.0:
+        return image
+        
+    import numpy as np
+    import cv2
+    
+    # Direct power law transformation
+    table = np.array([((i / 255.0) ** gamma) * 255
+        for i in np.arange(0, 256)]).astype("uint8")
+    return cv2.LUT(image, table)
+
+
+def find_views(img) -> List[Dict[str, Any]]:
+    """
+    使用 OmniParser 模型查找给定 UI 截图中的矩形视图。
+    默认应用 Gamma=4.0 的对比度增强以压暗背景（处理弹窗场景）。
+    
+    :param img: numpy.ndarray, representing an image in opencv
+    :return: List[ViewDict] - 符合统一 ViewDict 格式的视图列表
+    """
+    import cv2
+    
+    # 默认应用 Gamma 校正 (Gamma=4.0) 以压暗背景
+    # 用户需求：默认执行此预处理，无需外部传参
+    print(f"[CV] Applying Default Gamma Correction (gamma=4.0)...")
+    img = enhance_contrast_gamma(img, gamma=4.0)
+    # 保存处理后的图像到当前目录，方便查看
+    import os
+    import cv2
+    # output_path = os.path.join(".", "gamma_enhanced.png")
+    # cv2.imwrite(output_path, img)
+    # print(f"[CV] 已保存 Gamma 校正后的图像到: {os.path.abspath(output_path)}")
+    # 获取图像的原始尺寸
+    height, width = img.shape[:2]
+    
+    # 获取模型管理器实例（单例模式，确保只初始化一次）
+    model_manager = get_model_manager()
+    handler = model_manager.get_handler()
+    config = model_manager.get_config()
+    
+    # 构造推理请求
+    image_b64 = _encode_cv_image(img) 
+    
+    payload = {
+        "inputs": {
+            "image": image_b64,
+            # 模型推理尺寸可以设为一个标准值，例如 1280x1280
+            "image_size": {"w": config["image_size"], "h": config["image_size"]},
+            "bbox_threshold": config["bbox_threshold"],
+            "iou_threshold": config["iou_threshold"],
+        }
+    }
+    
+    # 执行推理
+    result = handler(payload)
+    
+    # 解析和转换结果为统一的 ViewDict 格式
+    views = []
+    
+    if "bboxes" in result:
+        # 打印模型识别出的元素总数（过滤前）
+        total_detected = len(result["bboxes"])
+        print(f"[CV] 模型识别出的元素总数（过滤前）: {total_detected}")
+        
+        # 打印各类型元素数量
+        icon_count = sum(1 for box in result["bboxes"] if box.get("type") == "icon")
+        text_count = sum(1 for box in result["bboxes"] if box.get("type") == "text")
+        
+        print(f"[CV]  - 图标元素数量: {icon_count}")
+        print(f"[CV]  - 文本元素数量: {text_count}")
+        
+        for idx, box_data in enumerate(result["bboxes"]):
+                
+            # 模型返回的归一化坐标 [x1, y1, x2, y2]
+            bbox_norm = box_data.get("bbox")
+            
+            if not bbox_norm or len(bbox_norm) < 4:
+                continue
+            
+            # 归一化坐标
+            x1_norm, y1_norm, x2_norm, y2_norm = bbox_norm
+            
+            # 转换为绝对像素坐标
+            x1 = int(x1_norm * width)
+            y1 = int(y1_norm * height)
+            x2 = int(x2_norm * width)
+            y2 = int(y2_norm * height)
+            
+            # 过滤掉不合理的尺寸
+            if x2 <= x1 or y2 <= y1:
+                continue
+                
+            # 获取文本信息（如果有）
+            text = box_data.get("content", "")
+            element_type = box_data.get("type", "cv_element")
+            is_interactive = box_data.get("interactivity", True)
+            
+            # 按照 _get_view_signature 格式生成 signature，供 input_policy 使用
+            view_text_sig = text if text and len(text) <= 50 else "None"
+            signature = "[class]%s[resource_id]%s[text]%s[%s,,]" % (
+                element_type or "None",
+                "",  # CV 视图无 resource_id
+                view_text_sig or "None",
+                "enabled",
+            )
+            
+            # 构建统一的 ViewDict 格式
+            view = {
+                # === 必需字段 ===
+                "bounds": [[x1, y1], [x2, y2]],
+                "text": text,
+                "visible": True,
+                "enabled": True,
+                "clickable": is_interactive,
+                "editable": False,
+                "scrollable": False,
+                "children": [],
+                "view_str": f"cv_{idx}_{x1}_{y1}_{x2}_{y2}",
+                "signature": signature,
+                
+                # === 可选字段 ===
+                "content_description": "",
+                "resource_id": "",
+                "class_name": element_type,
+                "temp_id": idx,
+                "parent": -1,
+                "source": "cv",
+            }
+            views.append(view)
+
+    return views
+
+
+
+def calculate_dhash(img):
+    """
+    Calculate the dhash value of an image.
+    :param img: numpy.ndarray, representing an image in opencv
+    :return:
+    """
+    difference = _calculate_pixel_difference(img)
+    # convert to hex
+    decimal_value = 0
+    hash_string = ""
+    for index, value in enumerate(difference):
+        if value:
+            decimal_value += value * (2 ** (index % 8))
+        if index % 8 == 7:  # every eight binary bit to one hex number
+            hash_string += str(hex(decimal_value)[2:-1].rjust(2, "0"))  # 0xf=>0x0f
+            decimal_value = 0
+    return hash_string
+
+
+def _calculate_pixel_difference(img):
+    """
+    Calculate difference between pixels
+    :param img: numpy.ndarray, representing an image in opencv
+    """
+    import cv2
+    resize_width = 18
+    resize_height = 16
+    # 1. resize to 18*16
+    smaller_image = cv2.resize(img, (resize_width, resize_height))
+
+    # 2. calculate grayscale
+    grayscale_image = cv2.cvtColor(smaller_image, cv2.COLOR_BGR2GRAY)
+
+    # 3. calculate difference between pixels
+    difference = []
+    for row in range(resize_height):
+        for col in range(resize_width - 1):
+            difference.append(grayscale_image[row][col] > grayscale_image[row][col + 1])
+    return difference
+
+
+def dhash_hamming_distance(dhash1, dhash2):
+    """
+    Calculate the hamming distance between two dhash values
+    :param dhash1: str, the dhash of an image returned by `calculate_dhash`
+    :param dhash2: str, the dhash of an image returned by `calculate_dhash`
+    :return: int, the hamming distance between two dhash values
+    """
+    difference = (int(dhash1, 16)) ^ (int(dhash2, 16))
+    return bin(difference).count("1")

DroidBot/cv/handler.py

@@ -0,0 +1,733 @@
+import base64
+import io
+import os
+from typing import Any, Dict, List, Literal, Optional, Tuple, Union
+
+# 在任何 matplotlib 导入之前强制设置非交互式后端
+# 避免在 macOS 自动化版本中弹出 Quartz 窗口导致阴塞
+import matplotlib
+matplotlib.use('Agg')
+
+import cv2
+import easyocr
+import numpy as np
+import torch
+from PIL import Image
+from PIL.Image import Image as ImageType
+from supervision.detection.core import Detections
+from supervision.draw.color import Color, ColorPalette
+from torchvision.ops import box_convert
+from torchvision.transforms import ToPILImage
+from transformers import AutoModelForCausalLM, AutoProcessor
+from transformers.image_utils import load_image
+from ultralytics import YOLO
+
+class EndpointHandler:
+    """
+    OmniParser 推理处理器
+    
+    Args:
+        model_dir: 模型目录路径
+        enable_ocr: 是否启用 OCR 文字识别（默认 True）
+        enable_caption: 是否启用图标描述生成（默认 True）
+        enable_yolo: 是否启用 YOLO 图标检测（默认 True）
+        ocr_languages: OCR 语言列表（默认 ["ch_sim", "en"] 中英文）
+    """
+    
+    def __init__(
+        self,
+        model_dir: str = os.path.dirname(os.path.abspath(__file__)),
+        enable_ocr: bool = True,
+        enable_caption: bool = True,
+        enable_yolo: bool = True,
+        ocr_languages: Optional[List[str]] = None,
+    ) -> None:
+        # 保存开关状态
+        self.enable_ocr = enable_ocr
+        self.enable_caption = enable_caption
+        self.enable_yolo = enable_yolo
+        
+        # 选择计算设备
+        if torch.cuda.is_available():
+            self.device = torch.device("cuda")
+            device_name = torch.cuda.get_device_name(0)
+            print(f"🚀 使用设备: GPU ({device_name})")
+        elif torch.backends.mps.is_available():
+            self.device = torch.device("mps")
+            print("🚀 使用设备: GPU (Apple MPS)")
+        else:
+            self.device = torch.device("cpu")
+            print("⚠️ 使用设备: CPU (推理较慢，建议安装 CUDA 版 PyTorch)")
+
+        # 显示启用的模块
+        modules = []
+        if enable_yolo:
+            modules.append("YOLO检测")
+        if enable_ocr:
+            modules.append("OCR文字")
+        if enable_caption:
+            modules.append("图标描述")
+        print(f"📦 启用模块: {', '.join(modules) if modules else '无'}")
+
+        # YOLO 图标检测模型
+        self.yolo = None
+        if enable_yolo:
+            self.yolo = YOLO(f"{model_dir}/icon_detect/model.pt")
+            self.yolo.to(self.device)
+
+        # Florence-2 图标描述模型
+        self.processor = None
+        self.model = None
+        if enable_caption:
+            self.processor = AutoProcessor.from_pretrained(
+                "microsoft/Florence-2-base", trust_remote_code=True
+            )
+            self.model = AutoModelForCausalLM.from_pretrained(
+                f"{model_dir}/icon_caption",
+                torch_dtype=torch.float16 if self.device.type != "cpu" else torch.float32,
+                trust_remote_code=True,
+            ).to(self.device)
+
+        # EasyOCR 文字识别模型
+        self.ocr = None
+        if enable_ocr:
+            langs = ocr_languages or ["ch_sim", "en"]
+            model_storage_directory = os.path.join(model_dir, "easyocr_models")
+            print(f"📄 EasyOCR 模型目录: {model_storage_directory}")
+            
+            # EasyOCR recognition.py 内部 DataLoader 将 pin_memory 写死为 True
+            # 但 MPS 设备不支持 pin_memory，通过 monkey-patch 消除警告
+            if self.device.type == "mps":
+                try:
+                    import easyocr.recognition as _easyocr_recognition
+                    import torch.utils.data as _torch_data
+                    _original_DataLoader = _torch_data.DataLoader
+                    def _patched_DataLoader(*args, **kwargs):
+                        kwargs['pin_memory'] = False
+                        return _original_DataLoader(*args, **kwargs)
+                    _easyocr_recognition.DataLoader = _patched_DataLoader
+                except Exception:
+                    pass  # 补丁失败不影响功能
+            
+            self.ocr = easyocr.Reader(
+                langs, 
+                gpu=self.device.type == "cuda",
+                model_storage_directory=model_storage_directory,
+                download_enabled=True
+            )
+
+        # box annotator
+        self.annotator = BoxAnnotator()
+
+    def __call__(self, data: Dict[str, Any]) -> Any:
+        data = data.pop("inputs")
+
+        # read image
+        image = load_image(data["image"])
+
+        # 1. OCR Step（如果启用）
+        ocr_texts, ocr_bboxes = [], []
+        if self.enable_ocr and self.ocr is not None:
+            ocr_texts, ocr_bboxes = self.check_ocr_bboxes(
+                image,
+                out_format="xyxy",
+                ocr_kwargs={"text_threshold": 0.4},
+            )
+        
+        # 2. YOLO + Caption Step
+        annotated_image, filtered_bboxes_out = self.get_som_labeled_img(
+            image,
+            image_size=data.get("image_size", None),
+            ocr_texts=ocr_texts,
+            ocr_bboxes=ocr_bboxes,
+            bbox_threshold=data.get("bbox_threshold", 0.05), 
+            iou_threshold=data.get("iou_threshold", 0.5),
+        )
+        return {
+            "image": annotated_image,
+            "bboxes": filtered_bboxes_out,
+        }
+
+    def check_ocr_bboxes(
+        self,
+        image: ImageType,
+        out_format: Literal["xywh", "xyxy"] = "xywh",
+        ocr_kwargs: Optional[Dict[str, Any]] = {},
+    ) -> Tuple[List[str], List[List[int]]]:
+        # 🔧 修复点 1：RBGA -> RGBA
+        if image.mode == "RGBA": 
+            image = image.convert("RGB")
+
+        result = self.ocr.readtext(np.array(image), **ocr_kwargs)  # type: ignore
+        texts = [str(item[1]) for item in result]
+        bboxes = [
+            self.coordinates_to_bbox(item[0], format=out_format) for item in result
+        ]
+        return (texts, bboxes)
+
+    @staticmethod
+    def coordinates_to_bbox(
+        coordinates: np.ndarray, format: Literal["xywh", "xyxy"] = "xywh"
+    ) -> List[int]:
+        if format == "xywh":
+            return [
+                int(coordinates[0][0]),
+                int(coordinates[0][1]),
+                int(coordinates[2][0] - coordinates[0][0]),
+                int(coordinates[2][1] - coordinates[0][1]),
+            ]
+        elif format == "xyxy":
+            return [
+                int(coordinates[0][0]),
+                int(coordinates[0][1]),
+                int(coordinates[2][0]),
+                int(coordinates[2][1]),
+            ]
+        else:
+            raise ValueError(f"Unsupported format: {format}")
+
+    @staticmethod
+    def bbox_area(bbox: List[int], w: int, h: int) -> int:
+        bbox = [bbox[0] * w, bbox[1] * h, bbox[2] * w, bbox[3] * h]
+        return (bbox[2] - bbox[0]) * (bbox[3] - bbox[1])
+
+    @staticmethod
+    def remove_bbox_overlap(
+        xyxy_bboxes: List[Dict[str, Any]],
+        ocr_bboxes: Optional[List[Dict[str, Any]]] = None,
+        iou_threshold: Optional[float] = 0.7,
+    ) -> List[Dict[str, Any]]:
+        filtered_bboxes = []
+        if ocr_bboxes is not None:
+            filtered_bboxes.extend(ocr_bboxes)
+
+        for i, bbox_outter in enumerate(xyxy_bboxes):
+            bbox_left = bbox_outter["bbox"]
+            valid_bbox = True
+
+            for j, bbox_inner in enumerate(xyxy_bboxes):
+                if i == j:
+                    continue
+
+                bbox_right = bbox_inner["bbox"]
+                if (
+                    intersection_over_union(
+                        bbox_left,
+                        bbox_right,
+                    )
+                    > iou_threshold  # type: ignore
+                ) and (area(bbox_left) > area(bbox_right)):
+                    valid_bbox = False
+                    break
+
+            if valid_bbox is False:
+                continue
+
+            if ocr_bboxes is None:
+                filtered_bboxes.append(bbox_outter)
+                continue
+
+            box_added = False
+            ocr_labels = []
+            for ocr_bbox in ocr_bboxes:
+                if not box_added:
+                    bbox_right = ocr_bbox["bbox"]
+                    # 检查是否重叠（合并 Icon 和 OCR 文字）
+                    if overlap(bbox_right, bbox_left):
+                        try:
+                            ocr_labels.append(ocr_bbox["content"])
+                            if ocr_bbox in filtered_bboxes:
+                                filtered_bboxes.remove(ocr_bbox)
+                        except Exception as e:
+                            self.logger.error(f"Error removing bbox overlap: {e}")
+                            continue
+                    elif overlap(bbox_left, bbox_right):
+                        box_added = True
+                        break
+
+            if not box_added:
+                filtered_bboxes.append(
+                    {
+                        "type": "icon",
+                        "bbox": bbox_outter["bbox"],
+                        "interactivity": True,
+                        "content": " ".join(ocr_labels) if ocr_labels else None,
+                    }
+                )
+
+        return filtered_bboxes
+
+    def get_som_labeled_img(
+        self,
+        image: ImageType,
+        image_size: Optional[Dict[Literal["w", "h"], int]] = None,
+        ocr_texts: Optional[List[str]] = None,
+        ocr_bboxes: Optional[List[List[int]]] = None,
+        bbox_threshold: float = 0.05,
+        iou_threshold: Optional[float] = None,
+        caption_prompt: Optional[str] = None,
+        caption_batch_size: int = 64,
+    ) -> Tuple[str, List[Dict[str, Any]]]:
+        if image.mode == "RGBA":
+            image = image.convert("RGB")
+
+        w, h = image.size
+        image_np = np.asarray(image)
+        
+        # YOLO 检测（如果启用）
+        xyxy_bboxes_raw = torch.tensor([])  # 默认空
+        if self.enable_yolo and self.yolo is not None:
+            if image_size is None:
+                imgsz = [h, w] 
+            else:
+                imgsz = [image_size.get("h", h), image_size.get("w", w)]
+
+            out = self.yolo.predict(
+                image,
+                imgsz=imgsz,
+                conf=bbox_threshold,
+                iou=iou_threshold or 0.7,
+                verbose=False,
+            )[0]
+            
+            if out.boxes is not None:
+                xyxy_bboxes_raw = out.boxes.xyxy
+                xyxy_bboxes_raw = xyxy_bboxes_raw / torch.Tensor([w, h, w, h]).to(xyxy_bboxes_raw.device)
+
+        # 处理 OCR 检测框
+        ocr_bboxes_normalized = []
+        if ocr_bboxes:
+            ocr_bboxes_tensor = torch.tensor(ocr_bboxes) / torch.Tensor([w, h, w, h])
+            ocr_bboxes_normalized = ocr_bboxes_tensor.tolist()
+
+        ocr_bbox_dicts = [
+            {
+                "type": "text",
+                "bbox": bbox,
+                "interactivity": False,
+                "content": text,
+                "source": "box_ocr_content_ocr",
+            }
+            for bbox, text in zip(ocr_bboxes_normalized, ocr_texts or [])
+            if self.bbox_area(bbox, w, h) > 0
+        ]
+        
+        # 处理 YOLO 检测框
+        yolo_bbox_dicts = [
+            {
+                "type": "icon",
+                "bbox": bbox,
+                "interactivity": True,
+                "content": None,
+                "source": "box_yolo_content_yolo",
+            }
+            for bbox in xyxy_bboxes_raw.tolist()
+            if self.bbox_area(bbox, w, h) > 0
+        ]
+
+        filtered_bboxes = self.remove_bbox_overlap(
+            xyxy_bboxes=yolo_bbox_dicts,
+            ocr_bboxes=ocr_bbox_dicts if ocr_bbox_dicts else None,
+            iou_threshold=iou_threshold or 0.7,
+        )
+
+        filtered_bboxes_out = sorted(
+            filtered_bboxes, key=lambda x: x["content"] is None
+        )
+        starting_idx = next(
+            (
+                idx
+                for idx, bbox in enumerate(filtered_bboxes_out)
+                if bbox["content"] is None
+            ),
+            -1,
+        )
+
+        filtered_bboxes = torch.tensor([box["bbox"] for box in filtered_bboxes_out])
+        
+        # 如果启用了 Caption 且有需要描述的图标
+        if starting_idx != -1 and self.enable_caption and self.model is not None:
+            non_ocr_bboxes = filtered_bboxes[starting_idx:]
+            bbox_images = []
+            for _, coordinates in enumerate(non_ocr_bboxes):
+                try:
+                    xmin, xmax = (
+                        int(coordinates[0] * image_np.shape[1]),
+                        int(coordinates[2] * image_np.shape[1]),
+                    )
+                    ymin, ymax = (
+                        int(coordinates[1] * image_np.shape[0]),
+                        int(coordinates[3] * image_np.shape[0]),
+                    )
+                    cropped_image = image_np[ymin:ymax, xmin:xmax, :]
+                    cropped_image = cv2.resize(cropped_image, (64, 64))
+                    bbox_images.append(ToPILImage()(cropped_image))
+                except Exception as e:
+                    self.logger.error(f"Error cropping bbox: {e}")
+                    continue
+
+            if caption_prompt is None:
+                caption_prompt = "<CAPTION>"
+
+            captions = []
+            for idx in range(0, len(bbox_images), caption_batch_size):  # type: ignore
+                batch = bbox_images[idx : idx + caption_batch_size]  # type: ignore
+                if not batch: break 
+                inputs = self.processor(
+                    images=batch,
+                    text=[caption_prompt] * len(batch),
+                    return_tensors="pt",
+                    do_resize=False,
+                )
+                if self.device.type in {"cuda", "mps"}:
+                    inputs = inputs.to(device=self.device, dtype=torch.float16)
+
+                with torch.inference_mode():
+                    generated_ids = self.model.generate(
+                        input_ids=inputs["input_ids"],
+                        pixel_values=inputs["pixel_values"],
+                        max_new_tokens=20,
+                        num_beams=1,
+                        do_sample=False,
+                        early_stopping=False,
+                    )
+
+                generated_texts = self.processor.batch_decode(
+                    generated_ids, skip_special_tokens=True
+                )
+                captions.extend([text.strip() for text in generated_texts])
+
+            ocr_texts = [f"Text Box ID {idx}: {text}" for idx, text in enumerate(ocr_texts)]  # type: ignore
+            for _, bbox in enumerate(filtered_bboxes_out):
+                if bbox["content"] is None and captions:
+                    bbox["content"] = captions.pop(0)
+
+        filtered_bboxes = box_convert(
+            boxes=filtered_bboxes, in_fmt="xyxy", out_fmt="cxcywh"
+        )
+
+        annotated_image = image_np.copy()
+        bboxes_annotate = filtered_bboxes * torch.Tensor([w, h, w, h])
+        xyxy_annotate = box_convert(
+            bboxes_annotate, in_fmt="cxcywh", out_fmt="xyxy"
+        ).numpy()
+        detections = Detections(xyxy=xyxy_annotate)
+        labels = [str(idx) for idx in range(bboxes_annotate.shape[0])]
+
+        annotated_image = self.annotator.annotate(
+            scene=annotated_image,
+            detections=detections,
+            labels=labels,
+            image_size=(w, h),
+        )
+        assert w == annotated_image.shape[1] and h == annotated_image.shape[0]
+
+        out_image = Image.fromarray(annotated_image)
+        out_buffer = io.BytesIO()
+        out_image.save(out_buffer, format="PNG")
+        encoded_image = base64.b64encode(out_buffer.getvalue()).decode("ascii")
+
+        return encoded_image, filtered_bboxes_out
+
+
+def area(bbox: List[int]) -> int:
+    return (bbox[2] - bbox[0]) * (bbox[3] - bbox[1])
+
+
+def intersection_area(bbox_left: List[int], bbox_right: List[int]) -> int:
+    # 计算两个 bbox 的交集面积
+    # 交集的左边界取两者的最大值，右边界取两者的最小值
+    x_overlap = max(0, min(bbox_left[2], bbox_right[2]) - max(bbox_left[0], bbox_right[0]))
+    y_overlap = max(0, min(bbox_left[3], bbox_right[3]) - max(bbox_left[1], bbox_right[1]))
+    return x_overlap * y_overlap
+
+
+def intersection_over_union(bbox_left: List[int], bbox_right: List[int]) -> float:
+    intersection = intersection_area(bbox_left, bbox_right)
+    bbox_left_area = area(bbox_left)
+    bbox_right_area = area(bbox_right)
+    union = bbox_left_area + bbox_right_area - intersection + 1e-6
+
+    ratio_left, ratio_right = 0, 0
+    if bbox_left_area > 0 and bbox_right_area > 0:
+        ratio_left = intersection / bbox_left_area
+        ratio_right = intersection / bbox_right_area
+    return max(intersection / union, ratio_left, ratio_right)
+
+
+def overlap(bbox_left: List[int], bbox_right: List[int]) -> bool:
+    intersection = intersection_area(bbox_left, bbox_right)
+    ratio_left = intersection / area(bbox_left)
+    # 🔧 修复点 3：从 0.80 降低到 0.50
+    # 作用：只要 OCR 文字框有一半在 YOLO 图标框内，就认为它们是一体的，
+    # 避免因为框对齐不准导致 Icon 框被丢弃或产生两个框
+    return ratio_left > 0.50
+
+
+class BoxAnnotator:
+    def __init__(
+        self,
+        color: Union[Color, ColorPalette] = ColorPalette.DEFAULT,  # type: ignore
+        thickness: int = 3,
+        text_color: Color = Color.BLACK,  # type: ignore
+        text_scale: float = 0.5,
+        text_thickness: int = 2,
+        text_padding: int = 10,
+        avoid_overlap: bool = True,
+    ):
+        self.color: Union[Color, ColorPalette] = color
+        self.thickness: int = thickness
+        self.text_color: Color = text_color
+        self.text_scale: float = text_scale
+        self.text_thickness: int = text_thickness
+        self.text_padding: int = text_padding
+        self.avoid_overlap: bool = avoid_overlap
+
+    def annotate(
+        self,
+        scene: np.ndarray,
+        detections: Detections,
+        labels: Optional[List[str]] = None,
+        skip_label: bool = False,
+        image_size: Optional[Tuple[int, int]] = None,
+    ) -> np.ndarray:
+        font = cv2.FONT_HERSHEY_SIMPLEX
+        for i in range(len(detections)):
+            x1, y1, x2, y2 = detections.xyxy[i].astype(int)
+            class_id = (
+                detections.class_id[i] if detections.class_id is not None else None
+            )
+            idx = class_id if class_id is not None else i
+            color = (
+                self.color.by_idx(idx)
+                if isinstance(self.color, ColorPalette)
+                else self.color
+            )
+            cv2.rectangle(
+                img=scene,
+                pt1=(x1, y1),
+                pt2=(x2, y2),
+                color=color.as_bgr(),
+                thickness=self.thickness,
+            )
+            if skip_label:
+                continue
+
+            text = (
+                f"{class_id}"
+                if (labels is None or len(detections) != len(labels))
+                else labels[i]
+            )
+
+            text_width, text_height = cv2.getTextSize(
+                text=text,
+                fontFace=font,
+                fontScale=self.text_scale,
+                thickness=self.text_thickness,
+            )[0]
+
+            if not self.avoid_overlap:
+                text_x = x1 + self.text_padding
+                text_y = y1 - self.text_padding
+
+                text_background_x1 = x1
+                text_background_y1 = y1 - 2 * self.text_padding - text_height
+
+                text_background_x2 = x1 + 2 * self.text_padding + text_width
+                text_background_y2 = y1
+            else:
+                (
+                    text_x,
+                    text_y,
+                    text_background_x1,
+                    text_background_y1,
+                    text_background_x2,
+                    text_background_y2,
+                ) = self.get_optimal_label_pos(
+                    self.text_padding,
+                    text_width,
+                    text_height,
+                    x1,
+                    y1,
+                    x2,
+                    y2,
+                    detections,
+                    image_size,
+                )
+
+            cv2.rectangle(
+                img=scene,
+                pt1=(text_background_x1, text_background_y1),
+                pt2=(text_background_x2, text_background_y2),
+                color=color.as_bgr(),
+                thickness=cv2.FILLED,
+            )
+            box_color = color.as_rgb()
+            luminance = (
+                0.299 * box_color[0] + 0.587 * box_color[1] + 0.114 * box_color[2]
+            )
+            text_color = (0, 0, 0) if luminance > 160 else (255, 255, 255)
+            cv2.putText(
+                img=scene,
+                text=text,
+                org=(text_x, text_y),
+                fontFace=font,
+                fontScale=self.text_scale,
+                color=text_color,
+                thickness=self.text_thickness,
+                lineType=cv2.LINE_AA,
+            )
+        return scene
+
+    @staticmethod
+    def get_optimal_label_pos(
+        text_padding, text_width, text_height, x1, y1, x2, y2, detections, image_size
+    ):
+        def get_is_overlap(
+            detections,
+            text_background_x1,
+            text_background_y1,
+            text_background_x2,
+            text_background_y2,
+            image_size,
+        ):
+            is_overlap = False
+            for i in range(len(detections)):
+                detection = detections.xyxy[i].astype(int)
+                if (
+                    intersection_over_union(
+                        [
+                            text_background_x1,
+                            text_background_y1,
+                            text_background_x2,
+                            text_background_y2,
+                        ],
+                        detection,
+                    )
+                    > 0.3
+                ):
+                    is_overlap = True
+                    break
+            if (
+                text_background_x1 < 0
+                or text_background_x2 > image_size[0]
+                or text_background_y1 < 0
+                or text_background_y2 > image_size[1]
+            ):
+                is_overlap = True
+            return is_overlap
+
+        text_x = x1 + text_padding
+        text_y = y1 - text_padding
+
+        text_background_x1 = x1
+        text_background_y1 = y1 - 2 * text_padding - text_height
+
+        text_background_x2 = x1 + 2 * text_padding + text_width
+        text_background_y2 = y1
+        is_overlap = get_is_overlap(
+            detections,
+            text_background_x1,
+            text_background_y1,
+            text_background_x2,
+            text_background_y2,
+            image_size,
+        )
+        if not is_overlap:
+            return (
+                text_x,
+                text_y,
+                text_background_x1,
+                text_background_y1,
+                text_background_x2,
+                text_background_y2,
+            )
+
+        text_x = x1 - text_padding - text_width
+        text_y = y1 + text_padding + text_height
+
+        text_background_x1 = x1 - 2 * text_padding - text_width
+        text_background_y1 = y1
+
+        text_background_x2 = x1
+        text_background_y2 = y1 + 2 * text_padding + text_height
+        is_overlap = get_is_overlap(
+            detections,
+            text_background_x1,
+            text_background_y1,
+            text_background_x2,
+            text_background_y2,
+            image_size,
+        )
+        if not is_overlap:
+            return (
+                text_x,
+                text_y,
+                text_background_x1,
+                text_background_y1,
+                text_background_x2,
+                text_background_y2,
+            )
+
+        text_x = x2 + text_padding
+        text_y = y1 + text_padding + text_height
+
+        text_background_x1 = x2
+        text_background_y1 = y1
+
+        text_background_x2 = x2 + 2 * text_padding + text_width
+        text_background_y2 = y1 + 2 * text_padding + text_height
+
+        is_overlap = get_is_overlap(
+            detections,
+            text_background_x1,
+            text_background_y1,
+            text_background_x2,
+            text_background_y2,
+            image_size,
+        )
+        if not is_overlap:
+            return (
+                text_x,
+                text_y,
+                text_background_x1,
+                text_background_y1,
+                text_background_x2,
+                text_background_y2,
+            )
+
+        text_x = x2 - text_padding - text_width
+        text_y = y1 - text_padding
+
+        text_background_x1 = x2 - 2 * text_padding - text_width
+        text_background_y1 = y1 - 2 * text_padding - text_height
+
+        text_background_x2 = x2
+        text_background_y2 = y1
+
+        is_overlap = get_is_overlap(
+            detections,
+            text_background_x1,
+            text_background_y1,
+            text_background_x2,
+            text_background_y2,
+            image_size,
+        )
+        if not is_overlap:
+            return (
+                text_x,
+                text_y,
+                text_background_x1,
+                text_background_y1,
+                text_background_x2,
+                text_background_y2,
+            )
+
+        return (
+            text_x,
+            text_y,
+            text_background_x1,
+            text_background_y1,
+            text_background_x2,
+            text_background_y2,
+        )

DroidBot/cv/huggingface_models/config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "directionality": "bidi",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "pooler_fc_size": 768,
+  "pooler_num_attention_heads": 12,
+  "pooler_num_fc_layers": 3,
+  "pooler_size_per_head": 128,
+  "pooler_type": "first_token_transform",
+  "type_vocab_size": 2,
+  "vocab_size": 119547
+}

DroidBot/cv/huggingface_models/tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": false, "model_max_length": 512}

DroidBot/cv/icon_detect/LICENSE

@@ -0,0 +1,661 @@
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU Affero General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<https://www.gnu.org/licenses/>.

DroidBot/cv/icon_detect/model.yaml

@@ -0,0 +1,129 @@
+backbone:
+- - -1
+  - 1
+  - Conv
+  - - 64
+    - 3
+    - 2
+- - -1
+  - 1
+  - Conv
+  - - 128
+    - 3
+    - 2
+- - -1
+  - 3
+  - C2f
+  - - 128
+    - true
+- - -1
+  - 1
+  - Conv
+  - - 256
+    - 3
+    - 2
+- - -1
+  - 6
+  - C2f
+  - - 256
+    - true
+- - -1
+  - 1
+  - Conv
+  - - 512
+    - 3
+    - 2
+- - -1
+  - 6
+  - C2f
+  - - 512
+    - true
+- - -1
+  - 1
+  - Conv
+  - - 1024
+    - 3
+    - 2
+- - -1
+  - 3
+  - C2f
+  - - 1024
+    - true
+- - -1
+  - 1
+  - SPPF
+  - - 1024
+    - 5
+ch: 3
+depth_multiple: 0.33
+head:
+- - -1
+  - 1
+  - nn.Upsample
+  - - None
+    - 2
+    - nearest
+- - - -1
+    - 6
+  - 1
+  - Concat
+  - - 1
+- - -1
+  - 3
+  - C2f
+  - - 512
+- - -1
+  - 1
+  - nn.Upsample
+  - - None
+    - 2
+    - nearest
+- - - -1
+    - 4
+  - 1
+  - Concat
+  - - 1
+- - -1
+  - 3
+  - C2f
+  - - 256
+- - -1
+  - 1
+  - Conv
+  - - 256
+    - 3
+    - 2
+- - - -1
+    - 12
+  - 1
+  - Concat
+  - - 1
+- - -1
+  - 3
+  - C2f
+  - - 512
+- - -1
+  - 1
+  - Conv
+  - - 512
+    - 3
+    - 2
+- - - -1
+    - 9
+  - 1
+  - Concat
+  - - 1
+- - -1
+  - 3
+  - C2f
+  - - 1024
+- - - 15
+    - 18
+    - 21
+  - 1
+  - Detect
+  - - nc
+nc: 1
+scale: ''
+width_multiple: 0.25
+yaml_file: weights/icon_detect_v1_5/model.yaml

DroidBot/cv/icon_detect/train_args.yaml

@@ -0,0 +1,107 @@
+train_args:
+  agnostic_nms: false
+  amp: true
+  augment: false
+  auto_augment: randaugment
+  batch: 64
+  box: 7.5
+  cache: false
+  cfg: null
+  classes: null
+  close_mosaic: 10
+  cls: 0.5
+  conf: null
+  copy_paste: 0.0
+  cos_lr: false
+  crop_fraction: 1.0
+  degrees: 0.0
+  deterministic: true
+  device:
+  - 0
+  - 1
+  - 2
+  - 3
+  dfl: 1.5
+  dnn: false
+  dropout: 0.0
+  dynamic: false
+  embed: null
+  epochs: 20
+  erasing: 0.4
+  exist_ok: false
+  fliplr: 0.5
+  flipud: 0.0
+  format: torchscript
+  fraction: 1.0
+  freeze: null
+  half: false
+  hsv_h: 0.015
+  hsv_s: 0.7
+  hsv_v: 0.4
+  imgsz: 1280
+  int8: false
+  iou: 0.7
+  keras: false
+  kobj: 1.0
+  label_smoothing: 0.0
+  line_width: null
+  lr0: 0.01
+  lrf: 0.01
+  mask_ratio: 4
+  max_det: 300
+  mixup: 0.0
+  mode: train
+  model: yolov8n.pt
+  momentum: 0.937
+  mosaic: 0.0
+  multi_scale: false
+  nbs: 64
+  nms: false
+  opset: null
+  optimize: false
+  optimizer: auto
+  overlap_mask: true
+  patience: 100
+  perspective: 0.0
+  plots: true
+  pose: 12.0
+  pretrained: true
+  profile: false
+  project: null
+  rect: false
+  resume: false
+  retina_masks: false
+  save: true
+  save_conf: false
+  save_crop: false
+  save_frames: false
+  save_hybrid: false
+  save_json: false
+  save_period: -1
+  save_txt: false
+  scale: 0.5
+  seed: 0
+  shear: 0.0
+  show: false
+  show_boxes: true
+  show_conf: true
+  show_labels: true
+  simplify: false
+  single_cls: false
+  source: null
+  split: val
+  stream_buffer: false
+  task: detect
+  time: null
+  tracker: botsort.yaml
+  translate: 0.1
+  val: true
+  verbose: true
+  vid_stride: 1
+  visualize: false
+  warmup_bias_lr: 0.0
+  warmup_epochs: 3.0
+  warmup_momentum: 0.8
+  weight_decay: 0.0005
+  workers: 8
+  workspace: 4

DroidBot/droidbot.py

@@ -0,0 +1,221 @@
+# This file contains the main class of droidbot
+# It can be used after AVD was started, app was installed, and adb had been set up properly
+# By configuring and creating a droidbot instance,
+# droidbot will start interacting with Android in AVD like a human
+import logging
+import os
+import sys
+import pkg_resources
+import shutil
+from threading import Timer
+
+from .core import PlatformFactory, Platform
+from .input_manager import InputManager
+
+# Import platforms to trigger auto-registration with PlatformFactory
+# This ensures platforms are registered before create_device is called
+from . import platforms  # noqa: F401
+
+# 从 exceptions 模块导入致命异常（避免循环导入）
+from .exceptions import FATAL_EXCEPTIONS  # noqa: F401
+
+
+class DroidBot(object):
+    """
+    The main class of droidbot
+    """
+    # this is a single instance class
+    instance = None
+
+    def __init__(self,
+                 package_name=None,
+                 device_serial=None,
+                 is_emulator=False,
+                 output_dir=None,
+                 policy_name=None,
+                 random_input=False,
+                 event_count=None,
+                 event_interval=None,
+                 timeout=None,
+                 keep_app=None,
+                 keep_env=False,
+                 cv_mode=False,
+                 debug_mode=False,
+                 profiling_method=None,
+                 grant_perm=False,
+                 enable_accessibility_hard=False,
+
+                 humanoid=None,
+                 ignore_ad=False,
+                 replay_output=None,
+                  enable_guiagent=False,
+                  app_name=None,
+                  platform="android",
+                  pcap_callback=None,
+                  enable_app_block=False,
+                  **kwargs):
+        """
+        initiate droidbot with configurations
+        :return:
+        """
+        # 注意：日志配置现在由统一的 logging_config 模块在入口文件中管理
+        # 不在这里调用 basicConfig，避免覆盖已有配置
+
+        self.logger = logging.getLogger('DroidBot')
+        DroidBot.instance = self
+
+        self.output_dir = output_dir
+        if output_dir is not None:
+            if not os.path.isdir(output_dir):
+                os.makedirs(output_dir)
+            html_index_path = pkg_resources.resource_filename("DroidBot", "resources/index.html")
+            stylesheets_path = pkg_resources.resource_filename("DroidBot", "resources/stylesheets")
+            target_stylesheets_dir = os.path.join(output_dir, "stylesheets")
+            if os.path.exists(target_stylesheets_dir):
+                shutil.rmtree(target_stylesheets_dir)
+            shutil.copy(html_index_path, output_dir)
+            shutil.copytree(stylesheets_path, target_stylesheets_dir)
+
+        self.timeout = timeout
+        self.timer = None
+        self.keep_env = keep_env
+        self.keep_app = keep_app
+
+        self.device = None
+        self.input_manager = None
+        self.enable_accessibility_hard = enable_accessibility_hard
+        self.humanoid = humanoid
+        self.ignore_ad = ignore_ad
+        self.replay_output = replay_output
+        self.enable_guiagent = enable_guiagent
+        self.app_name = app_name
+        self.pcap_callback = pcap_callback
+        self.enable_app_block = enable_app_block
+
+        self.enabled = True
+        self._timeout_triggered = False
+        self._stopped = False
+
+        try:
+            # Use PlatformFactory to create device
+            platform_enum = Platform(platform)
+            
+            # Build platform-specific device arguments
+            if platform_enum == Platform.IOS:
+                # iOS device arguments
+                device_kwargs = {
+                    'wda_url': kwargs.get('wda_url', 'http://localhost:8100'),
+                    'bundle_id': package_name,  # package_name is bundle_id for iOS
+                    'output_dir': self.output_dir,
+                    'cv_mode': cv_mode,
+                    'udid': device_serial,  # device_serial maps to udid for iOS
+                    'debug_mode': debug_mode,
+                }
+            elif platform_enum == Platform.WINDOWS:
+                # Windows device arguments
+                device_kwargs = {
+                    'window_title': self.app_name,  # app_name maps to window_title
+                    'exe_path': device_serial,      # device_serial maps to exe_path
+                    'steam_game_id': package_name,  # package_name maps to steam_game_id
+                    'output_dir': self.output_dir,
+                    'cv_mode': cv_mode,
+                    'debug_mode': debug_mode,
+                }
+            elif platform_enum == Platform.WEB:
+                # Web device arguments
+                device_kwargs = {
+                    'app_path': package_name,    # URL作为app_path
+                    'output_dir': self.output_dir,
+                    'browser': kwargs.get('browser', 'chrome'),
+                    'engine': kwargs.get('engine', 'playwright'),
+                    'headless': kwargs.get('headless', False),
+                }
+            else:
+                # Android device arguments
+                device_kwargs = {
+                    'device_serial': device_serial,
+                    'is_emulator': is_emulator,
+                    'output_dir': self.output_dir,
+                    'app_path': package_name,
+                    'cv_mode': cv_mode,
+                    'grant_perm': grant_perm,
+                    'enable_accessibility_hard': self.enable_accessibility_hard,
+                    'humanoid': self.humanoid,
+                    'ignore_ad': ignore_ad,
+                    **kwargs,
+                }
+            
+            self.device = PlatformFactory.create_device(platform_enum, **device_kwargs)
+
+            self.input_manager = InputManager(
+                device=self.device,
+                policy_name=policy_name,
+                random_input=random_input,
+                event_count=event_count,
+                event_interval=event_interval,
+                profiling_method=profiling_method,
+                replay_output=replay_output,
+                enable_guiagent=self.enable_guiagent,
+                app_name=self.app_name,
+                pcap_callback=self.pcap_callback,
+                enable_app_block=self.enable_app_block)
+        except Exception:
+            self.stop()
+            raise
+
+    def start(self):
+        """
+        start interacting
+        :return:
+        """
+        if not self.enabled:
+            return
+        self.logger.info("Starting DroidBot")
+        try:
+            if self.timeout > 0:
+                self.timer = Timer(self.timeout, self._on_timeout)
+                self.timer.start()
+
+            self.device.set_up()
+
+            if not self.enabled:
+                return
+            self.device.connect()
+
+            if not self.enabled:
+                return
+            # self.device.install_app() # app is already installed
+
+
+
+            if not self.enabled:
+                return 
+            self.input_manager.start()
+
+        except Exception:
+            raise
+        finally:
+            self.stop()
+            self.logger.info("DroidBot Stopped")
+
+    def _on_timeout(self):
+        """Timer 线程只发出停止信号，避免跨线程关闭 Playwright。"""
+        self._timeout_triggered = True
+        self.enabled = False
+        self.logger.warning(f"DroidBot timeout reached ({self.timeout}s), requesting graceful stop")
+        if self.input_manager:
+            self.input_manager.stop()
+
+    def stop(self):
+        if self._stopped:
+            return
+        self._stopped = True
+        self.enabled = False
+        if self.timer and self.timer.is_alive():
+            self.timer.cancel()
+
+        if self.input_manager:
+            self.logger.info("Total steps: %d" % self.input_manager.total_exploring_steps)
+            self.input_manager.stop()
+        if self.device:
+            self.device.disconnect()

DroidBot/exceptions.py

@@ -0,0 +1,80 @@
+# -*- coding: utf-8 -*-
+"""
+DroidBot Exceptions Module
+
+定义致命异常类型，供其他模块导入使用。
+独立模块避免循环导入问题。
+"""
+
+import logging
+logger = logging.getLogger(__name__)
+
+class InputInterruptedException(Exception):
+    """
+    Exception in InputManager
+    """
+    pass
+
+class ADBException(Exception):
+    """
+    Exception in ADB connection
+    """
+    pass
+
+class AppCrashException(Exception):
+    """
+    Exception raised when app crashes repeatedly (闪退异常)
+    Triggered when the app fails to restart multiple times consecutively.
+    """
+    pass
+
+class AppNeedUpdateException(Exception):
+    """
+    Exception raised when app redirects to Google Play Store (需更新异常)
+    Triggered when the foreground package changes to com.android.vending.
+    """
+    pass
+
+class AppLaunchErrorException(Exception):
+    """
+    Exception raised when app redirects to another app (启动异常-跳转)
+    Triggered when the foreground package changes to a third-party app.
+    """
+    pass
+
+class ExplorationStuckException(Exception):
+    """
+    Exception raised when exploration is stuck (探索停滞异常)
+    Triggered when no new states are discovered for a prolonged period.
+    """
+    pass
+
+# 致命异常集合 - 遇到这些异常应立即中止并抛出
+# 先定义基本异常，避免循环导入问题
+# 其他模块可从此处导入使用: from .exceptions import FATAL_EXCEPTIONS
+FATAL_EXCEPTIONS = (
+    ADBException,             # ADB 断联
+    AppCrashException,       # 应用闪退
+    AppNeedUpdateException,  # 需更新（跳转Google Play）
+    AppLaunchErrorException, # 启动异常（跳转其他应用）
+    ExplorationStuckException, # 探索停滞
+    KeyboardInterrupt,        # 用户手动中断
+    SystemExit,             # 系统退出
+)
+
+# 导入 WDA 异常类 (iOS 相关) - 延迟导入避免循环依赖
+try:
+    from .platforms.ios.wda.exceptions import (
+        MuxError, MuxConnectError, WDAError, WDAStuckError
+    )
+    # 仅 WDAStuckError（多次恢复失败）是致命异常
+    # 普通 WDA 异常由 IOSDevice._on_wda_failure() 处理，触发异步恢复
+    FATAL_EXCEPTIONS = FATAL_EXCEPTIONS + (WDAStuckError,)
+    
+    # 导出普通 WDA 异常供 ios_start.py 等模块捕获使用
+    WDA_RECOVERABLE_EXCEPTIONS = (WDAError, MuxError, MuxConnectError)
+except ImportError:
+    # wda 模块不可用，继续使用基本异常集合
+    logger.warning("WDA 异常导入失败，仅使用基本异常集合")
+    WDA_RECOVERABLE_EXCEPTIONS = ()
+

DroidBot/guiagent_bridge.py

@@ -0,0 +1,838 @@
+"""
+GuiAgent集成模块 (使用guiagent_core重构版本)
+用于在DroidBot中调用GuiAgent处理特定场景
+
+重构说明：
+- 使用guiagent_core的GuiAgentDecisionMaker替代完整GuiAgent
+- 消除重复device连接，复用DroidBot的device
+- 保持所有接口不变，确保向后兼容
+- 场景配置从统一配置文件加载，支持平台特定覆盖
+"""
+import os
+import sys
+import logging
+import time
+from typing import Dict, List, Optional, Tuple, Any
+
+# 添加autool根路径到系统路径（用于导入setting）
+from pathlib import Path
+autool_root = str(Path(__file__).resolve().parent.parent)
+if autool_root not in sys.path:
+    sys.path.insert(0, autool_root)
+
+# 尝试导入guiagent_core
+try:
+    from .guiagent_core import GuiAgentDecisionMaker, ContextManager
+    GUIAGENT_AVAILABLE = True
+except ImportError as e:
+    logging.error(f"无法导入guiagent_core: {e}")
+    GUIAGENT_AVAILABLE = False
+
+
+class GuiAgentBridge:
+    """
+    DroidBot与GuiAgent之间的桥梁（使用guiagent_core重构版本）
+    
+    主要改进：
+    1. 使用GuiAgentDecisionMaker替代完整GuiAgent
+    2. 复用DroidBot的device进行截图和执行
+    3. 添加详细debug日志
+    4. 保持所有现有接口不变
+    """
+    
+    def __init__(self, device, app=None, app_name=None, utg=None, input_manager=None):
+        """
+        初始化GuiAgent桥接器
+        :param device: DroidBot的设备实例
+        :param app: DroidBot的App实例（可选）
+        :param app_name: App名称（可选，直接指定）
+        :param utg: UTG实例（用于记录每步state transition）
+        :param input_manager: InputManager实例（用于设置执行flag）
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.device = device
+        self.app = app
+        self.app_name = app_name
+        self.utg = utg  # UTG引用，用于记录每步state transition
+        self.input_manager = input_manager  # input_manager引用，用于设置执行flag
+        self.decision_maker = None
+        self.context = None
+        self.is_enabled = GUIAGENT_AVAILABLE
+        
+        
+        # 加载场景配置 (从统一配置文件加载，支持平台覆盖)
+        platform_name = device.get_platform_name() if device else "android"
+        try:
+            from .guiagent_core.scene_config_loader import load_scene_config
+            scene_config = load_scene_config(platform_name)
+            self.keywords = scene_config.get("keywords", {})
+            self.instructions = scene_config.get("instructions", {})
+            self.step_limits = scene_config.get("step_limits", {})
+            self.logger.info(f"[GuiAgent] 加载场景配置成功: platform={platform_name}, scenes={list(self.keywords.keys())}")
+        except Exception as e:
+            self.logger.error(f"[GuiAgent] 加载场景配置失败: {e}, 使用空配置")
+            import traceback
+            self.logger.debug(traceback.format_exc())
+            # 降级处理: 使用空配置
+            self.keywords = {}
+            self.instructions = {}
+            self.step_limits = {}
+
+
+        # 上次动作坐标（用于在截图上绘制绿圈标记）
+        self._last_action_coords = None  # (x, y) 绝对像素坐标
+        
+        # 状态监控相关
+        self.last_state_time = time.time()
+        self.last_state_hash = None
+        
+        # 界面状态跟踪 - 记录每个界面已经处理的操作类型
+        self.processed_states = {}  # {state_hash: set(categories)}
+        
+        if self.is_enabled:
+            self._init_agent()
+    
+    def _init_agent(self):
+        """初始化GuiAgent决策引擎（使用guiagent_core）"""
+        try:
+            # 获取设备分辨率
+            display_info = self.device.get_display_info()
+            width = display_info.get('width', 1080)
+            height = display_info.get('height', 1920)
+            resolution = (width, height)
+            
+            self.logger.debug(f"初始化GuiAgent决策引擎: resolution={resolution}")
+            
+            # 根据设备平台动态配置
+            platform_name = self.device.get_platform_name() if self.device else "android"
+            # 强制使用归一化坐标
+            absolute_mode = False
+            
+            # 创建决策引擎（不创建设备连接）
+            self.decision_maker = GuiAgentDecisionMaker(
+                platform=platform_name,
+                resolution=resolution,
+                absolute_mode=absolute_mode
+            )
+            
+            self.logger.info(f"GuiAgent决策引擎初始化成功 (guiagent_core版本)")
+            self.logger.debug(f"决策引擎配置: platform={platform_name}, resolution={resolution}, absolute_mode={absolute_mode}")
+        except Exception as e:
+            self.logger.error(f"GuiAgent决策引擎初始化失败: {e}")
+            import traceback
+            self.logger.debug(traceback.format_exc())
+            self.is_enabled = False
+    
+    def is_state_processed(self, current_state, category: str) -> bool:
+        """
+        检查当前界面是否已经处理过特定类别的操作
+        :param current_state: 当前设备状态
+        :param category: 操作类别
+        :return: 如果已处理过返回True，否则返回False
+        """
+        if not current_state or not hasattr(current_state, 'state_str'):
+            return False
+            
+        state_hash = current_state.state_str
+        if state_hash in self.processed_states:
+            return category in self.processed_states[state_hash]
+        return False
+    
+    def mark_state_processed(self, current_state, category: str):
+        """
+        标记当前界面已处理过特定类别的操作
+        :param current_state: 当前设备状态
+        :param category: 操作类别
+        """
+        if not current_state or not hasattr(current_state, 'state_str'):
+            return
+            
+        state_hash = current_state.state_str
+        if state_hash not in self.processed_states:
+            self.processed_states[state_hash] = set()
+        self.processed_states[state_hash].add(category)
+        self.logger.info(f"[GuiAgent] 标记界面 {state_hash[:16]}... 已处理 '{category}' 操作")
+    
+    def check_keywords(self, text: str) -> Optional[str]:
+        """
+        检查文本中是否包含需要特殊处理的关键词
+        :param text: 要检查的文本
+        :return: 如果找到关键词，返回对应的类别，否则返回None
+        """
+        if not text:
+            return None
+            
+        text_lower = text.lower()
+        for category, keywords in self.keywords.items():
+            for keyword in keywords:
+                if keyword.lower() in text_lower:
+                    self.logger.debug(f"[GuiAgent] 检测到关键词: '{keyword}' (类别: {category})")
+                    return category
+        return None
+    
+    def get_text_within_bounds(self, view: dict, all_views: list) -> str:
+        """
+        获取控件边界框内所有视图的文本（用于关键词检测）
+        :param view: 目标视图字典
+        :param all_views: 当前状态的所有视图列表
+        :return: 边界框内所有文本的拼接字符串
+        """
+        bounds = view.get('bounds', [[0, 0], [0, 0]])
+        left, top = bounds[0]
+        right, bottom = bounds[1]
+        
+        if right <= left or bottom <= top:
+            return view.get('text', '') or view.get('content_description', '') or ''
+        
+        texts = []
+        for v in all_views:
+            v_bounds = v.get('bounds', [[0, 0], [0, 0]])
+            v_left, v_top = v_bounds[0]
+            v_right, v_bottom = v_bounds[1]
+            
+            if v_left >= left and v_top >= top and v_right <= right and v_bottom <= bottom:
+                if v.get('text'):
+                    texts.append(v['text'])
+                elif v.get('content_description'):
+                    texts.append(v['content_description'])
+        
+        return ' '.join(texts)
+    
+    def _get_scene_max_steps(self, category: str, explicit_max_steps: Optional[int] = None) -> int:
+        """
+        获取指定场景的最大步数
+        优先级: 显式传入参数 > scene_configs.json 场景配置 > default_steps > 硬编码兜底20
+        :param category: 场景类别
+        :param explicit_max_steps: 显式传入的步数（None 表示未指定）
+        :return: 最大步数
+        """
+        if explicit_max_steps is not None:
+            self.logger.debug(f"[GuiAgent] 场景 '{category}' 使用显式步数: {explicit_max_steps}")
+            return explicit_max_steps
+        
+        # 过滤掉注释键，只查找场景配置
+        limits = {k: v for k, v in self.step_limits.items() if not k.startswith('_')}
+        default_steps = limits.get("default_steps", 20)
+        scene_steps = limits.get(category, default_steps)
+        source = "场景配置" if category in limits else "default_steps"
+        self.logger.debug(f"[GuiAgent] 场景 '{category}' 使用步数: {scene_steps} (来源: {source})")
+        return scene_steps
+
+    def handle_with_guiagent(self, category: str, context: Dict[str, Any] = None,
+                              max_steps: Optional[int] = None, max_error_steps: int = 20) -> Tuple[bool, Optional[str], Optional[int]]:
+        """
+        使用GuiAgent处理特定场景
+        
+        注意：此方法会执行完整个任务（多步操作），然后返回成功/失败
+        这与input_policy的预期一致：调用后等待agent处理完，记录为一次事件
+        
+        重要改进：
+        1. 设置执行flag，执行过程中不进行前台应用拉回检测，不进行总步数累计
+        2. 记录每次agent决策event前后的state到UTG
+        3. 记录token开销并在执行完成后打印
+        
+        :param category: 场景类别（login, register, payment, game等）
+        :param context: 上下文信息
+        :param max_steps: 最大执行步数（默认 20）
+        :param max_error_steps: 最大连续错误步数（默认 20）
+        :return: (处理是否成功, 失败原因, stuck_reason_code)
+                 stuck_reason_code: 仅当 report_stuck_reason 时返回具体code，否则为None
+        """
+        if not self.is_enabled or not self.decision_maker:
+            self.logger.warning("[GuiAgent] GuiAgent不可用，无法处理")
+            return False, None, None
+            
+        current_state = context.get("state") if context else None
+        
+        
+        # 检查当前界面是否已经处理过此类操作
+        if current_state and self.is_state_processed(current_state, category):
+            self.logger.info(f"[GuiAgent] 当前界面已处理过 '{category}' 操作，跳过")
+            return False, None, None
+            
+        # 设置执行flag - 执行过程中跳过前台检查和步数累计
+        if self.input_manager:
+            self.input_manager.is_guiagent_executing = True
+            self.logger.debug("[GuiAgent] 设置执行flag，跳过前台检查和步数累计")
+        
+        try:
+            # 生成任务指令
+            # 生成任务指令
+            instruction = self._generate_instruction(category, context)
+            
+            self.logger.info(f"[GuiAgent] 开始处理任务 (类别: {category}): {instruction}")
+
+            # [二级检测] 验证当前界面是否确实属于目标场景（跳过卡住状态的验证）
+            if category not in ("game_initial", "explore_stuck", "stuck_escape"):
+                # 先截图用于验证
+                screenshot_path = self.device.take_screenshot()
+                if screenshot_path and not self.decision_maker.verify_screen(category, screenshot_path=screenshot_path):
+                    self.logger.warning(f"界面验证失败，当前可能并非 {category} 场景，跳过 GuiAgent 处理")
+                    return False, None, None
+            
+            # 创建新的上下文（重置上次动作坐标，从fastbot重新进入时不记忆）
+            self._last_action_coords = None
+            self.context = self.decision_maker.create_context(instruction)
+            
+            MAX_STEPS = self._get_scene_max_steps(category, max_steps)
+            MAX_ERROR_STEPS = max_error_steps
+            self.logger.info(f"[GuiAgent] 场景 '{category}' 步数限制: {MAX_STEPS} 步")
+            executed_steps = 0
+            error_steps = 0
+            success_count = 0
+            task_finished = False  # 标记agent是否明确报告任务完成
+            
+            # 记录初始state
+            before_state = self.device.get_current_state()
+            
+            guiagent_message = None  # 统一的消息记录（成功或失败）
+            stuck_reason_code = None  # 记录卡住原因代码（仅 report_stuck_reason 时设置）
+            
+            while executed_steps < MAX_STEPS:
+                if error_steps >= MAX_ERROR_STEPS:
+                    self.logger.error(f"[GuiAgent] 总计失败{MAX_ERROR_STEPS}次，终止处理")
+                    if category in ('login', 'register'):
+                        category_name = "登录" if category == "login" else "注册"
+                        guiagent_message = f"{category_name}失败: 连续错误次数达到上限({MAX_ERROR_STEPS}次)"
+                        self.logger.warning(f"[GuiAgent] {guiagent_message}")
+                    error_steps = 0
+                    break
+                
+                self.logger.debug(f"[GuiAgent] 执行第 {executed_steps + 1}/{MAX_STEPS} 步")
+                
+                # 1. 截图（使用DroidBot的设备）
+                screenshot_path = self.device.take_screenshot()
+                if not screenshot_path:
+                    self.logger.warning("[GuiAgent] 截图失败，终止处理")
+                    error_steps += 1
+                    # 尝试等待 WDA 就绪
+                    if hasattr(self.device, '_wait_wda_ready'):
+                        wda_ready = self.device._wait_wda_ready(timeout=30)
+                        if not wda_ready and hasattr(self.device, '_on_wda_failure'):
+                            self.device._on_wda_failure("GuiAgent 截图失败")
+                            # 等待 WDA 恢复完成
+                            time.sleep(5)
+                    else:
+                        time.sleep(2)
+                    continue
+                
+                self.logger.debug(f"[GuiAgent] 截图完成: {screenshot_path}")
+                
+                # 决定是否需要添加网格和动作标记（例如卡住检测和某些特定场景不需要）
+                draw_grid_and_marker = category not in ("explore_stuck",)
+                
+                # 2. 调用决策引擎（传入step用于日志记录）
+                decision = self.decision_maker.decide_next_action(
+                    screenshot_path=screenshot_path,
+                    context=self.context,
+                    step=executed_steps + 1,
+                    last_action_coords=self._last_action_coords,
+                    draw_grid_and_marker=draw_grid_and_marker
+                )
+                if not decision.get('success'):
+                    error_msg = decision.get('error', 'Unknown error')
+                    self.logger.error(f"[GuiAgent] 决策失败: {error_msg}")
+                    error_steps += 1
+                    time.sleep(2)
+                    continue
+                
+                executed_steps += 1
+                action_type = decision['action_type']
+                action = decision.get('action_data', {})
+                thought = decision.get('thought', '')
+                self.logger.info(f"[GuiAgent] LLM决策: {action} | Thought: {thought}")
+                
+                # 3. 检查是否完成
+                if decision.get('is_finished'):
+                    self.logger.info(f"[GuiAgent] 任务完成 (共执行 {executed_steps} 步)")
+                    success_count += 1
+                    task_finished = True  # 明确标记任务完成
+                    
+                    # 处理 report_stuck_reason 动作
+                    if action_type == 'report_stuck_reason':
+                        reason_code = str(action.get('reason_code', '0'))
+                        raw_message = action.get('message', '')
+                        stuck_reason_code = int(reason_code)  # 记录 reason_code 用于上报
+                        
+                        STUCK_REASON_MAP = {
+                            1: "[失败] 登录注册",
+                            2: "[失败] 启动异常",
+                            3: "[成功] 测试正常",
+                            4: "[失败] 地区限制",
+                            5: "[失败] 需付费",
+                            6: "[失败] 虚拟手机号无效",
+                            7: "[失败] 虚拟身份无效",
+                            8: "[失败] 注册校验失败",
+                            9: "[失败] 非开放注册",
+                            10: "[失败] 应用停服",
+                            11: "[失败] 需实体证件",
+                            12: "[失败] Root模式下无法使用",
+                            0: "[失败] 其他原因",
+                        }
+                        
+                        if reason_code == '3':
+                            self.logger.info("[GuiAgent] 卡住诊断: 测试正常，尝试脱离卡住状态")
+                            if current_state and self.is_state_processed(current_state, "stuck_escape"):
+                                self.logger.info("[GuiAgent] stuck_escape 已处理过，不再重复调用")
+                                task_finished = False
+                            else:
+                                escape_success, _, _ = self.handle_with_guiagent(
+                                    category="stuck_escape",
+                                    context=context
+                                )
+                                if escape_success:
+                                    # 成功脱困不属于错误上报，保持消息为空。
+                                    self.logger.info("[GuiAgent] 已脱离卡住状态，不生成汇报消息")
+                                else:
+                                    stuck_reason_code = 0
+                                    if raw_message:
+                                        guiagent_message = f"探索卡住: [失败] {raw_message}"
+                                    else:
+                                        guiagent_message = "探索卡住: [失败] 脱离卡住状态失败"
+                                    self.logger.warning(f"[GuiAgent] 脱离卡住状态失败，保留原始卡住原因: {guiagent_message}")
+                        else:
+                            reason_desc = STUCK_REASON_MAP.get(int(reason_code), f"[失败] 未知原因({reason_code})")
+                            guiagent_message = f"探索卡住: {reason_desc}"
+                            if raw_message:
+                                guiagent_message += f" ({raw_message})"
+                            self.logger.info(f"[GuiAgent] 卡住诊断结果: {guiagent_message}")
+                    elif category in ('login', 'register'):
+                        category_name = "登录" if category == "login" else "注册"
+                        guiagent_message = f"{category_name}成功"
+                        self.logger.info(f"[GuiAgent] {guiagent_message}")
+                    break
+                
+                # 4. 处理receive_email特殊动作：获取邮件内容并反馈给上下文
+                if action_type == 'receive_email':
+                    from .guiagent_core.utils import receive_email
+                    email_results = receive_email()
+                    if email_results:
+                        feedbacks = []
+                        for i, email in enumerate(email_results):
+                            feedbacks.append(f"邮件{i+1}主题: '{email.get('subject', '')}', 内容: {email.get('content', '')}\n")
+                        feedback = "\n".join(feedbacks)
+                        
+                        self.logger.info(f"[GuiAgent] 收到 {len(email_results)} 封邮件: {feedback[:100]}...")
+                        self.context.add_user_message(f"近期收到以下邮件，请你根据信息判断选取其中的哪一封:\n{feedback}")
+                        self.logger.info("[GuiAgent] 已将邮件内容发送给智能体")
+                    else:
+                        self.context.add_user_message("未收到新邮件")
+                        self.logger.warning("[GuiAgent] 未收到新邮件")
+                    success_count += 1
+                    continue  # 不执行设备动作，继续下一轮决策
+                
+                # 5. 处理wait动作：等待一段时间
+                if action_type == 'wait':
+                    wait_duration = decision.get('action_data', {}).get('duration', 2)
+                    self.logger.info(f"[GuiAgent] 执行等待动作: {wait_duration}秒")
+                    time.sleep(wait_duration)
+                    success_count += 1
+                    continue  # 不生成设备事件，继续下一轮决策
+                
+                # 5.1 处理滑块验证码技能
+                if action_type == 'solve_slider_captcha':
+                    from .guiagent_core.skills import solve_slider_captcha
+                    
+                    action_data = decision.get('action_data', {})
+                    captcha_region = action_data.get('captcha_region', [])
+                    slider_position = action_data.get('slider_position', [])
+                    
+                    # 坐标转换: 归一化坐标 [0-1000] -> 绝对像素坐标
+                    display_info = self.device.get_display_info()
+                    width = display_info.get('width', 1080)
+                    height = display_info.get('height', 1920)
+                    
+                    def to_abs(coords, is_region=False):
+                        if not coords:
+                            return []
+                        if is_region and len(coords) == 4:
+                            return [
+                                int(coords[0] * width / 1000),
+                                int(coords[1] * height / 1000),
+                                int(coords[2] * width / 1000),
+                                int(coords[3] * height / 1000)
+                            ]
+                        elif len(coords) == 2:
+                            return [int(coords[0] * width / 1000), int(coords[1] * height / 1000)]
+                        return coords
+                    
+                    region_abs = to_abs(captcha_region, is_region=True)
+                    slider_abs = to_abs(slider_position)
+                    
+                    self.logger.info(f"[GuiAgent] 执行滑块验证码技能: region={region_abs}, slider={slider_abs}")
+                    success, msg = solve_slider_captcha(
+                        device=self.device,
+                        captcha_region=region_abs,
+                        slider_position=slider_abs,
+                        debug=True
+                    )
+                    
+                    # 将结果反馈给上下文
+                    if success:
+                        self.context.add_user_message(f"滑块验证码操作已完成: {msg}")
+                        self.logger.info(f"[GuiAgent] 滑块验证码成功: {msg}")
+                    else:
+                        self.context.add_user_message(f"滑块验证码操作失败: {msg}，请尝试其他方式或手动处理")
+                        self.logger.warning(f"[GuiAgent] 滑块验证码失败: {msg}")
+                    success_count += 1
+                    time.sleep(2)  # 等待验证结果
+                    continue  # 技能已执行，继续下一轮决策
+                
+                # 5.2 处理图片验证码技能
+                if action_type == 'solve_image_captcha':
+                    from .guiagent_core.skills import solve_image_captcha
+                    from .core import PlatformFactory
+                    
+                    action_data = decision.get('action_data', {})
+                    captcha_region = action_data.get('captcha_region', [])
+                    input_field = action_data.get('input_field', [])
+                    
+                    # 坐标转换
+                    display_info = self.device.get_display_info()
+                    width = display_info.get('width', 1080)
+                    height = display_info.get('height', 1920)
+                    
+                    def to_abs(coords, is_region=False):
+                        if not coords:
+                            return []
+                        if is_region and len(coords) == 4:
+                            return [
+                                int(coords[0] * width / 1000),
+                                int(coords[1] * height / 1000),
+                                int(coords[2] * width / 1000),
+                                int(coords[3] * height / 1000)
+                            ]
+                        elif len(coords) == 2:
+                            return [int(coords[0] * width / 1000), int(coords[1] * height / 1000)]
+                        return coords
+                    
+                    region_abs = to_abs(captcha_region, is_region=True)
+                    field_abs = to_abs(input_field)
+                    
+                    self.logger.info(f"[GuiAgent] 执行图片验证码技能: region={region_abs}, input_field={field_abs}")
+                    success, captcha_text = solve_image_captcha(
+                        device=self.device,
+                        captcha_region=region_abs,
+                        debug=True
+                    )
+                    
+                    if success and captcha_text:
+                        # 先点击输入框
+                        if field_abs:
+                            platform = self.device.get_platform_name()
+                            TouchEvent = PlatformFactory.get_event_class(platform, 'touch')
+                            tap_event = TouchEvent(x=field_abs[0], y=field_abs[1])
+                            self.device.send_event(tap_event)
+                            time.sleep(0.5)
+                        
+                        # 输入验证码文本
+                        platform = self.device.get_platform_name()
+                        SetTextEvent = PlatformFactory.get_event_class(platform, 'set_text')
+                        text_event = SetTextEvent(text=captcha_text)
+                        self.device.send_event(text_event)
+                        
+                        self.context.add_user_message(f"图片验证码已识别并输入: {captcha_text}")
+                        self.logger.info(f"[GuiAgent] 图片验证码成功: {captcha_text}")
+                    else:
+                        self.context.add_user_message(f"图片验证码识别失败: {captcha_text}，请尝试其他方式")
+                        self.logger.warning(f"[GuiAgent] 图片验证码失败: {captcha_text}")
+                    success_count += 1
+                    time.sleep(1)
+                    continue  # 技能已执行，继续下一轮决策
+
+                if action_type == 'login_ios':
+                    from .guiagent_core.utils import login_ios
+                    login_success, login_message = login_ios(self.device)
+                    if login_success:
+                        self.context.add_user_message(f"iOS登录脚本执行成功: {login_message}")
+                        self.logger.info(f"[GuiAgent] login_ios 成功: {login_message}")
+                        success_count += 1
+                    else:
+                        self.context.add_user_message(f"iOS登录脚本失败: {login_message}，请根据截屏内容完成登录步骤")
+                        self.logger.warning(f"[GuiAgent] login_ios 失败: {login_message}")
+                        error_steps += 1
+                    time.sleep(2)
+                    continue  # 技能已执行，继续下一轮决策
+                
+                # 6. 转换为平台事件并执行
+                event = self._convert_decision_to_event(decision, before_state)
+                if event:
+                    self.logger.debug(f"[GuiAgent] 执行事件: {type(event).__name__}")
+
+                    # 执行前保存状态
+                    before_exec_state = self.device.get_current_state()
+
+                    # 执行动作
+                    self.device.send_event(event)
+                    time.sleep(2)  # 等待UI更新
+
+                    # 执行后获取状态
+                    after_exec_state = self.device.get_current_state()
+
+                    # 验证执行效果
+                    verify_success, verify_msg = self._verify_action_effect(
+                        before_exec_state, after_exec_state, action_type
+                    )
+
+                    if verify_success:
+                        self.context.add_execution_feedback(True)
+                        success_count += 1
+                    else:
+                        self.context.add_execution_feedback(False, verify_msg)
+                        self.logger.warning(f"[GuiAgent] 执行验证失败: {verify_msg}")
+                        error_steps += 1
+
+                    # 记录state transition到UTG
+                    if self.utg:
+                        if before_state and after_exec_state:
+                            self.utg.add_transition(event, before_state, after_exec_state, is_guiagent_event=True)
+                            self.logger.debug(f"[GuiAgent] 记录state transition: {before_state.state_str[:16]}... -> {after_exec_state.state_str[:16]}...")
+                        before_state = after_exec_state  # 更新before_state为下一步
+                else:
+                    self.logger.warning(f"[GuiAgent] 无法生成事件 (action_type: {action_type})")
+                    executed_steps -= 1
+                    error_steps += 1
+                    self.context.add_execution_feedback(False, f"无法生成 {action_type} 事件")
+                    time.sleep(2)
+                    continue
+            
+            # 判断是否成功（有执行过有效操作）
+            success = success_count > 0
+            
+            # 打印Agent步数
+            self.logger.info(f"[GuiAgent] 任务执行完成, Agent步数={success_count}")
+            
+            # 检查login/register是否因超出步数限制而失败
+            if category in ('login', 'register') and executed_steps >= MAX_STEPS and not task_finished:
+                category_name = "登录" if category == "login" else "注册"
+                guiagent_message = f"{category_name}失败: 超出最大步数限制({MAX_STEPS}步)"
+                self.logger.warning(f"[GuiAgent] {guiagent_message}")
+            
+            # 只有当agent明确报告任务完成时，才标记界面已处理
+            # 这样处理失败的界面下次还会重试
+            if task_finished:
+                self.logger.info(f"[GuiAgent] 处理成功 (类别: {category}, 执行了 {success_count} 个有效操作)")
+                if current_state:
+                    self.mark_state_processed(current_state, category)
+            else:
+                if success:
+                    self.logger.warning(f"[GuiAgent] 任务未完成但执行了操作 (类别: {category}, 执行了 {success_count} 个操作, 下次将重试)")
+                else:
+                    self.logger.warning(f"[GuiAgent] 处理失败 (类别: {category}, 未执行任何有效操作, 下次将重试)")
+            
+            return success, guiagent_message, stuck_reason_code
+                
+        except KeyboardInterrupt:
+            raise
+        except Exception as e:
+            self.logger.error(f"[GuiAgent] 处理异常: {e}")
+            import traceback
+            self.logger.debug(traceback.format_exc())
+            return False, f"GuiAgent处理异常: {e}", None
+        finally:
+            # 确保执行flag被重置
+            if self.input_manager:
+                self.input_manager.is_guiagent_executing = False
+                self.logger.debug("[GuiAgent] 重置执行flag")
+    
+    def _convert_decision_to_event(self, decision: Dict[str, Any], current_state=None):
+        """
+        将GuiAgent决策转换为平台事件
+
+        注意：guiagent_core返回的是归一化坐标[0-1000]，需要转换为绝对像素坐标
+        使用PlatformFactory获取平台特定的事件类，支持Android和iOS
+        """
+        from .core import PlatformFactory
+
+        action_type = decision['action_type']
+        action_data = decision['action_data']
+
+        # 获取设备平台名称
+        platform = self.device.get_platform_name()
+
+        # 获取分辨率用于坐标转换
+        display_info = self.device.get_display_info()
+        width = display_info.get('width', 1080)
+        height = display_info.get('height', 1920)
+        resolution = (width, height)
+
+        def normalize_to_absolute(coords):
+            """归一化坐标 [0-1000] -> 绝对像素坐标"""
+            if not coords or len(coords) != 2:
+                return coords
+
+            # 只有在非 absolute_mode 时才进行归一化转换
+            if self.decision_maker and self.decision_maker.absolute_mode:
+                return [int(coords[0]), int(coords[1])]
+
+            x, y = coords
+            abs_x = int(x * resolution[0] / 1000)
+            abs_y = int(y * resolution[1] / 1000)
+            self.logger.debug(f"[GuiAgent] 坐标转换: ({x:.1f}, {y:.1f}) -> ({abs_x}, {abs_y})")
+            return [abs_x, abs_y]
+
+        try:
+            if action_type == 'tap' or action_type == 'click':
+                if 'target' in action_data and action_data['target']:
+                    x, y = normalize_to_absolute(action_data['target'])
+
+                    # 尝试找到最接近的 clickable view，使用精确的 view center
+                    if current_state:
+                        closest_view = self._find_closest_view(current_state, x, y)
+
+                        if closest_view:
+                            self.logger.info(f"[GuiAgent] 使用 view 对象替代坐标点击")
+                            self._last_action_coords = (x, y)  # 记录原始坐标用于绿圈标记
+                            TouchEvent = PlatformFactory.get_event_class(platform, 'touch')
+                            return TouchEvent(view=closest_view)
+
+                    self.logger.info(f"[GuiAgent] 生成TouchEvent: ({x}, {y})")
+                    self._last_action_coords = (x, y)
+                    TouchEvent = PlatformFactory.get_event_class(platform, 'touch')
+                    return TouchEvent(x=x, y=y)
+
+            elif action_type == 'long_tap':
+                if 'target' in action_data and action_data['target']:
+                    x, y = normalize_to_absolute(action_data['target'])
+                    self.logger.debug(f"[GuiAgent] 生成LongTouchEvent: ({x}, {y})")
+                    self._last_action_coords = (x, y)
+                    LongTouchEvent = PlatformFactory.get_event_class(platform, 'long_touch')
+                    return LongTouchEvent(x=x, y=y)
+
+            elif action_type in ('drag', 'swipe'):
+                if 'start' in action_data and 'end' in action_data:
+                    start_x, start_y = normalize_to_absolute(action_data['start'])
+                    end_x, end_y = normalize_to_absolute(action_data['end'])
+                    self.logger.debug(f"[GuiAgent] 生成SwipeEvent: ({start_x},{start_y}) -> ({end_x},{end_y})")
+                    self._last_action_coords = (start_x, start_y)
+                    SwipeEvent = PlatformFactory.get_event_class(platform, 'swipe')
+                    return SwipeEvent(start_x=start_x, start_y=start_y,
+                                     end_x=end_x, end_y=end_y)
+
+            elif action_type == 'type':
+                if 'text' in action_data and action_data['text']:
+                    text = action_data['text']
+                    self.logger.debug(f"[GuiAgent] 生成SetTextEvent: '{text}'")
+                    SetTextEvent = PlatformFactory.get_event_class(platform, 'set_text')
+                    return SetTextEvent(text=text)
+
+            elif action_type == 'key_press':
+                if 'key' in action_data and action_data['key']:
+                    key = action_data['key']
+                    self.logger.debug(f"[GuiAgent] 生成KeyEvent: {key}")
+                    KeyEvent = PlatformFactory.get_event_class(platform, 'key')
+                    return KeyEvent(key_name=key)
+
+        except Exception as e:
+            self.logger.error(f"[GuiAgent] 事件转换失败: {e}")
+            return None
+
+        return None
+
+    def _verify_action_effect(self, before_state, after_state, action_type: str) -> Tuple[bool, str]:
+        """
+        验证动作是否产生预期效果
+
+        Args:
+            before_state: 执行前的设备状态
+            after_state: 执行后的设备状态
+            action_type: 动作类型
+
+        Returns:
+            (是否成功, 失败原因)
+        """
+        if not before_state or not after_state:
+            return True, ""
+
+        # 检查状态是否变化
+        state_changed = before_state.state_str != after_state.state_str
+
+        if not state_changed:
+            if action_type in ('tap', 'click', 'long_tap'):
+                return False, "界面未发生变化，可能点击位置不准确或点击无效区域"
+            elif action_type == 'type':
+                return False, "输入未生效，可能未正确聚焦输入框"
+            elif action_type in ('drag', 'swipe'):
+                return False, "滑动未生效，可能滑动距离不足或方向错误"
+
+        return True, ""
+
+    def _find_closest_view(self, state, target_x: int, target_y: int, threshold: int = 50):
+        """
+        在 state.views 中找到最接近目标坐标的 clickable view
+
+        Args:
+            state: 当前设备状态
+            target_x: 目标 x 坐标
+            target_y: 目标 y 坐标
+            threshold: 最大距离阈值（像素）
+
+        Returns:
+            最接近的 view 对象，如果没有找到则返回 None
+        """
+        if not state or not hasattr(state, 'views'):
+            return None
+
+        closest_view = None
+        min_distance = float('inf')
+
+        for view in state.views:
+            if not view.get('clickable') or not view.get('visible'):
+                continue
+
+            bounds = view.get('bounds', [[0, 0], [0, 0]])
+            center_x = (bounds[0][0] + bounds[1][0]) // 2
+            center_y = (bounds[0][1] + bounds[1][1]) // 2
+
+            distance = ((center_x - target_x)**2 + (center_y - target_y)**2)**0.5
+
+            if distance < min_distance and distance < threshold:
+                min_distance = distance
+                closest_view = view
+
+        if closest_view:
+            self.logger.debug(f"[GuiAgent] 找到最近的 view，距离: {min_distance:.1f}px")
+
+        return closest_view
+    
+    def _generate_instruction(self, category: str, context: Dict[str, Any] = None) -> str:
+        """
+        根据类别生成任务指令
+        :param category: 场景类别
+        :param context: 上下文信息
+        :return: 任务指令
+        """
+        base_instruction = self.instructions.get(category, "继续操作")
+        
+        # 构建完整指令，在开头添加app名称
+        app_name = self._get_app_name()
+        if app_name:
+           full_instruction = f"你将要操作的app是{app_name}。{base_instruction}"
+        else:
+            full_instruction = base_instruction
+        
+        # 如果有额外上下文，添加到指令中
+        if context and "additional_info" in context:
+            full_instruction += f"。附加信息: {context['additional_info']}"
+        
+        self.logger.debug(f"[GuiAgent] 生成指令: {full_instruction}")
+        return full_instruction
+    
+    def _get_app_name(self) -> str:
+        """
+        获取当前app的名称
+        :return: app名称
+        """
+        # 1. 优先使用初始化时传入的app_name
+        if self.app_name:
+            return self.app_name
+            
+        # 2. 尝试从apk文件名获取
+        if self.app and hasattr(self.app, 'app_path'):
+            import os
+            apk_path = self.app.app_path
+            filename = os.path.basename(apk_path)
+            if filename.endswith('.apk'):
+                return filename[:-4].replace('_', ' ')
+        return ""
+    

DroidBot/guiagent_core/__init__.py

@@ -0,0 +1,33 @@
+"""
+GuiAgent Core - Modular Decision Making Components
+
+This package provides the core GuiAgent functionality as independent modules
+that can be integrated into DroidBot without requiring full GuiAgent setup.
+
+Key Components:
+- GuiAgentDecisionMaker: Stateless LLM-based decision engine
+- ContextManager: Conversation history management
+- UniversalLLMClient: Multi-provider LLM interface
+- Prompt builders for different platforms (iOS, Android, Desktop)
+- GmailChecker: Email checking utility for receive_email action
+"""
+
+from .decision_maker import GuiAgentDecisionMaker
+from .context_manager import ContextManager
+from .llm_client import LLMClient
+from .prompt_builder import get_ios_prompt, get_android_prompt, get_verification_prompt
+from .utils import parse_uitars_action, convert_to_executor_action, GmailChecker, receive_email
+
+__all__ = [
+    'GuiAgentDecisionMaker',
+    'ContextManager',
+    'LLMClient',
+    'get_ios_prompt',
+    'get_android_prompt',
+    'get_verification_prompt',
+    'parse_uitars_action',
+    'convert_to_executor_action',
+    'GmailChecker',
+    'receive_email',
+]
+

DroidBot/guiagent_core/constants.py

@@ -0,0 +1,50 @@
+"""
+Constants Module for GuiAgent Core
+
+Configuration related to models (API keys, params) has been moved to KeyPool.
+"""
+
+from typing import List
+from pathlib import Path
+import sys
+
+ROOT_DIR = Path(__file__).resolve().parent.parent.parent
+if str(ROOT_DIR) not in sys.path:
+    sys.path.insert(0, str(ROOT_DIR))
+
+from config_loader import load_config as load_autool_config
+
+# ==============================================================================
+# Constants
+# ==============================================================================
+
+MAX_HISTORY_LENGTH = 10  # Maximum conversation history length
+IMAGE_PLACEHOLDER = "<image>"  # Placeholder for images in conversation
+RESOLUTION_PLACEHOLDER = "{{resolution}}"  # Placeholder for resolution in prompts
+
+def load_config(json_path: str = None):
+    """从JSON文件加载配置"""
+    return load_autool_config(json_path)
+
+CURRENT_MODEL_NAME = (
+    load_config().get("CURRENT_MODEL_NAME")
+    or load_config().get("model", {}).get("current_name", "")
+    or ""
+)
+# Models that use absolute coordinates (return actual pixel coordinates)
+# instead of normalized coordinates [0-1000]
+ABSOLUTE_COORD_MODELS: List[str] = ["claude-sonnet-4-5"]
+
+def is_absolute_coord_model(model_name: str) -> bool:
+    """
+    Determine if the specified model uses absolute coordinate mode.
+    
+    Args:
+        model_name: The name of the model to check
+        
+    Returns:
+        bool: True if the model uses absolute coordinates, False for normalized [0-1000]
+    """
+    if not model_name:
+        return False
+    return model_name in ABSOLUTE_COORD_MODELS

DroidBot/guiagent_core/context_manager.py

@@ -0,0 +1,217 @@
+"""
+Context Manager - 管理对话历史和构建模型请求
+
+Extracted from GuiAgent/core/context.py for modular use in DroidBot.
+"""
+import re
+import logging
+from typing import Optional, Dict, Any, List
+
+logger = logging.getLogger(__name__)
+
+# Constants
+IMAGE_PLACEHOLDER = "<image>"
+MAX_HISTORY_LENGTH = 20
+
+
+class ContextManager:
+    """
+    上下文管理器，负责：
+    1. 管理对话历史
+    2. 构建发送给模型的请求消息
+    3. 过滤和限制对话历史长度
+    """
+    
+    def __init__(self, system_prompt: str, max_history: int = MAX_HISTORY_LENGTH):
+        """
+        初始化上下文管理器
+        
+        Args:
+            system_prompt: 系统提示词
+            max_history: 最大对话历史长度
+        """
+        self._system_prompt = system_prompt
+        self.max_history = max_history
+        
+        # 对话历史：[{from: 'human'/'gpt', value: str, screenshot?: str}, ...]
+        self.conversations: List[Dict[str, Any]] = []
+        
+        # 初始指令是否已添加
+        self._has_instruction: bool = False
+
+    @property
+    def system_prompt(self) -> str:
+        return self._system_prompt
+    
+    @system_prompt.setter
+    def system_prompt(self, value: str) -> None:
+        self._system_prompt = value
+
+    def add_instruction(self, instruction: str) -> None:
+        """添加用户初始指令（仅调用一次）"""
+        if self._has_instruction:
+            return
+        
+        self.conversations.append({
+            'from': 'human',
+            'value': instruction
+        })
+        self._has_instruction = True
+
+    def add_user_message(self, message: str) -> None:
+        """
+        添加用户消息到对话历史
+        
+        Args:
+            message: 用户消息内容
+        """
+        self.conversations.append({
+            'from': 'human',
+            'value': message
+        })
+        self._prune()
+
+    def add_screenshot(self, screenshot_base64: str, width: int, height: int) -> None:
+        """
+        添加截图到对话历史
+        
+        Args:
+            screenshot_base64: 截图的 Base64 编码（不含 data:image 前缀）
+            width: 图像宽度
+            height: 图像高度
+        """
+        self.conversations.append({
+            'from': 'human',
+            'value': IMAGE_PLACEHOLDER,
+            'screenshot': screenshot_base64,
+            'size': (width, height)
+        })
+        self._prune()
+
+    def add_response(self, response: str) -> None:
+        """
+        添加模型响应到对话历史
+
+        Args:
+            response: 模型响应文本
+        """
+        # 提取摘要（移除 Reflection 部分）
+        summary = re.sub(r'Reflection:[\s\S]*?(?=Action:|$)', '', response).strip()
+
+        self.conversations.append({
+            'from': 'gpt',
+            'value': summary
+        })
+        self._prune()
+
+    def add_execution_feedback(self, success: bool, detail: str = "") -> None:
+        """
+        添加动作执行结果反馈
+
+        Args:
+            success: 执行是否成功
+            detail: 详细信息（失败原因或额外说明）
+        """
+        if success:
+            feedback = "✓ 上次操作已成功执行"
+            if detail:
+                feedback += f": {detail}"
+        else:
+            feedback = f"✗ 上次操作失败: {detail}。请分析失败原因并调整策略（如调整坐标、使用替代方案等）"
+
+        self.conversations.append({
+            'from': 'system',
+            'value': feedback
+        })
+        self._prune()
+
+    def _prune(self) -> None:
+        """限制对话历史长度，保留初始指令 + 最近对话"""
+        if len(self.conversations) <= self.max_history:
+            return
+        
+        # 保留第一条（初始指令）+ 最近的对话
+        first = self.conversations[0] if self.conversations else None
+        recent = self.conversations[-(self.max_history - 1):]
+        self.conversations = [first] + recent if first else recent
+
+    def _get_filtered(self) -> List[Dict[str, Any]]:
+        """
+        获取过滤后的对话历史：
+        - 保留所有 AI 回复
+        - 保留所有系统反馈（执行结果）
+        - 保留初始指令 + 最近3轮的完整对话（包括截图）
+        """
+        # 找出所有截图索引
+        screenshot_indices = [i for i, c in enumerate(self.conversations) if c.get('screenshot')]
+
+        if not screenshot_indices:
+            return self.conversations
+
+        # 保留最近3轮的截图及其后续对话
+        KEEP_RECENT_ROUNDS = 3
+        keep_from_idx = screenshot_indices[-KEEP_RECENT_ROUNDS] if len(screenshot_indices) >= KEEP_RECENT_ROUNDS else 0
+
+        filtered = []
+        for i, conv in enumerate(self.conversations):
+            # 始终保留初始指令
+            if i == 0:
+                filtered.append(conv)
+            # 保留最近N轮的所有对话
+            elif i >= keep_from_idx:
+                filtered.append(conv)
+            # 保留所有系统反馈（即使在旧轮次中）
+            elif conv['from'] == 'system':
+                filtered.append(conv)
+
+        return filtered
+
+    def build_messages(self) -> List[Dict[str, Any]]:
+        """
+        构建发送给模型的消息列表（OpenAI 格式）
+
+        Returns:
+            消息列表，每条消息包含 role 和 content
+        """
+        filtered = self._get_filtered()
+        messages = []
+        images = []
+
+        # 收集图像
+        for conv in filtered:
+            if conv.get('screenshot'):
+                images.append(conv['screenshot'])
+
+        image_idx = 0
+        for i, conv in enumerate(filtered):
+            if conv.get('screenshot'):
+                # 图像消息
+                messages.append({
+                    'role': 'user',
+                    'content': [{
+                        'type': 'image_url',
+                        'image_url': {'url': f"data:image/png;base64,{images[image_idx]}"}
+                    }]
+                })
+                image_idx += 1
+            elif i == 0 and conv['from'] == 'human':
+                # 第一条消息：嵌入系统提示词
+                messages.append({
+                    'role': 'system',
+                    'content': f"{self._system_prompt}\n{conv['value']}"
+                })
+            else:
+                # 普通消息和系统反馈
+                role_map = {'human': 'user', 'gpt': 'assistant', 'system': 'user'}
+                role = role_map.get(conv['from'], 'user')
+                messages.append({
+                    'role': role,
+                    'content': conv['value']
+                })
+
+        return messages
+
+    def clear(self) -> None:
+        """清空对话历史"""
+        self.conversations = []
+        self._has_instruction = False

DroidBot/guiagent_core/decision_maker.py

@@ -0,0 +1,766 @@
+"""
+Decision Maker - Core GuiAgent decision engine without device coupling
+
+Extracted from GuiAgent/core/agent.py, this module provides stateless LLM-based
+decision making for GUI automation without managing device connections.
+"""
+import base64
+import logging
+import json
+import time
+import os
+from pathlib import Path
+from typing import Dict, Any, Tuple, Optional
+from PIL import Image
+import sys
+
+# 智能导入：先尝试相对导入，失败则使用绝对导入
+# 支持：模块导入、直接运行、调试器运行等多种场景
+try:
+    from .context_manager import ContextManager
+    from .llm_client import LLMClient
+    from .prompt_builder import get_ios_prompt, get_android_prompt, get_desktop_prompt, get_verification_prompt
+    from .utils import parse_uitars_action, convert_to_executor_action, strip_base64_prefix, draw_grid_on_image, draw_last_action_marker, image_to_base64
+    from .constants import is_absolute_coord_model, CURRENT_MODEL_NAME
+except ImportError:
+    # 相对导入失败，添加项目根目录到 sys.path 并使用绝对导入
+    current_file = Path(__file__).resolve()
+    project_root = current_file.parent.parent.parent
+    if str(project_root) not in sys.path:
+        sys.path.insert(0, str(project_root))
+    from DroidBot.guiagent_core.context_manager import ContextManager
+    from DroidBot.guiagent_core.llm_client import LLMClient
+    from DroidBot.guiagent_core.prompt_builder import get_ios_prompt, get_android_prompt, get_desktop_prompt, get_verification_prompt
+    from DroidBot.guiagent_core.utils import parse_uitars_action, convert_to_executor_action, strip_base64_prefix, draw_grid_on_image, draw_last_action_marker, image_to_base64
+    from DroidBot.guiagent_core.constants import is_absolute_coord_model, CURRENT_MODEL_NAME
+
+import re
+
+logger = logging.getLogger(__name__)
+
+
+class GuiAgentDecisionMaker:
+    """
+    GuiAgent 核心决策引擎（无状态，不管理设备）
+    
+    提供基于LLM的GUI自动化决策能力，支持多平台（iOS, Android, Desktop）。
+    专为DroidBot等自动化框架设计，实现决策与执行的解耦。
+    """
+
+    # 全局步数计数器，用于记录本轮采集时的GUI Agent步数总和
+    total_steps = 0
+    
+    def __init__(
+        self,
+        platform: str,
+        resolution: Tuple[int, int] = None,
+        scale: float = 1.0,
+        absolute_mode: bool = None
+    ):
+        """
+        初始化决策引擎
+        
+        Args:
+            platform: 平台类型 ("ios", "android", "desktop")
+            resolution: 屏幕分辨率 (width, height)
+                       对于iOS: 这是截图的物理像素尺寸 (e.g., 1125x2436 for iPhone X @3x)
+                       对于Android/Desktop: 这是逻辑分辨率
+            scale: iOS scale factor (2.0 for @2x, 3.0 for @3x)，其他平台使用1.0
+            model_name: 使用的LLM模型名称
+            absolute_mode: 是否使用绝对坐标模式，如果为None则自动判断
+        """
+        self.platform = platform.lower()
+        self.resolution = resolution
+        self.scale = scale
+        self.model_name = CURRENT_MODEL_NAME
+        self.llm = LLMClient(model_name=self.model_name)
+        
+        
+        # 确定坐标模式
+        if absolute_mode is None:
+            # iOS推荐使用绝对坐标，其他平台根据模型决定
+            if self.platform == "ios":
+                self.absolute_mode = True
+            else:
+                self.absolute_mode = is_absolute_coord_model(self.model_name)
+        else:
+            self.absolute_mode = absolute_mode
+        
+        # 是否在截图上绘制网格（帮助LLM定位）
+        self.enable_grid = True
+        
+        # 日志文件路径和初始化
+        self.log_dir = Path(os.environ.get('GUIAGENT_LOG_DIR', '/tmp/guiagent_logs'))
+        self.log_file_path = None
+        self._init_log()
+        
+        # 生成系统提示词
+        self.system_prompt = self._generate_system_prompt()
+        
+        logger.info(
+            f"Initialized GuiAgentDecisionMaker: platform={platform}, "
+            f"resolution={resolution}, scale={scale}, absolute_mode={self.absolute_mode}"
+        )
+    
+    def _init_log(self) -> None:
+        """初始化日志文件（追加模式，每次会话追加记录）"""
+        try:
+            self.log_dir.mkdir(parents=True, exist_ok=True)
+            self.log_file_path = self.log_dir / f"agent_{time.strftime('%Y%m%d')}.log"
+            with open(self.log_file_path, "a", encoding="utf-8") as f:
+                f.write(f"\n{'='*60}\n")
+                f.write(f"=== Agent Session: {time.strftime('%Y-%m-%d %H:%M:%S')} ===\n")
+                f.write(f"=== Platform: {self.platform}, Resolution: {self.resolution} ===\n")
+                f.write(f"{'='*60}\n")
+            logger.debug(f"日志文件初始化: {self.log_file_path}")
+        except Exception as e:
+            logger.error(f"日志文件初始化失败: {e}")
+            self.log_file_path = None
+    
+    def _log(self, step: int, messages: list, response: str, tokens: Dict = None, error_type: str = None) -> None:
+        """
+        记录到日志文件，包含发送给模型的上下文（去掉 base64 图片内容）
+        
+        Args:
+            step: 当前步骤数
+            messages: 发送给LLM的消息列表
+            response: LLM响应文本
+            tokens: Token使用统计
+            error_type: 错误类型（如果有）
+        """
+        if not self.log_file_path:
+            return
+            
+        # 复制一份消息并脱敏 image base64，避免日志体积暴涨
+        def _sanitize_messages(msgs: list) -> list:
+            sanitized = []
+            for msg in msgs:
+                msg_copy = json.loads(json.dumps(msg))  # 简单深拷贝
+                content = msg_copy.get("content")
+                if isinstance(content, list):
+                    for item in content:
+                        if isinstance(item, dict) and item.get("type") == "image_url":
+                            if "image_url" in item:
+                                item["image_url"]["url"] = "<image_base64_omitted>"
+                msg_copy["content"] = content
+                sanitized.append(msg_copy)
+            return sanitized
+
+        try:
+            with open(self.log_file_path, "a", encoding="utf-8") as f:
+                f.write(f"\n=== Round {step} ===\n")
+                if error_type:
+                    f.write(f"Error Type: {error_type}\n")
+                f.write("Messages Sent:\n")
+                f.write(json.dumps(_sanitize_messages(messages), ensure_ascii=False, indent=2))
+                f.write("\nResponse:\n")
+                f.write(f"{response}\n")
+                if tokens:
+                    f.write(f"Tokens: in={tokens.get('input_tokens', 0)}, "
+                           f"out={tokens.get('output_tokens', 0)}\n")
+        except Exception as e:
+            logger.error(f"写入日志文件失败: {e}")
+    
+    def _generate_system_prompt(self) -> str:
+        """生成平台特定的系统提示词"""
+        if self.platform == "ios":
+            return get_ios_prompt(
+                resolution=self.resolution,
+                scale=self.scale,
+                absolute_mode=self.absolute_mode
+            )
+        elif self.platform == "android":
+            return get_android_prompt(
+                resolution=self.resolution,
+                absolute_mode=self.absolute_mode
+            )
+        elif self.platform == "desktop":
+            return get_desktop_prompt(
+                resolution=self.resolution,
+                absolute_mode=self.absolute_mode
+            )
+        else:
+            raise ValueError(f"Unsupported platform: {self.platform}")
+    
+    def decide_next_action(
+        self,
+        screenshot_path: str = None,
+        screenshot_base64: str = None,
+        width: int = None,
+        height: int = None,
+        context: ContextManager = None,
+        step: int = 0,
+        last_action_coords: tuple = None,
+        draw_grid_and_marker: bool = True
+    ) -> Dict[str, Any]:
+        """
+        基于截图决策下一步动作
+        
+        Args:
+            screenshot_path: 截图文件路径（与screenshot_base64二选一）
+            screenshot_base64: 截图的base64编码（与screenshot_path二选一）
+            width: 截图宽度（如果不提供则从图像获取）
+            height: 截图高度（如果不提供则从图像获取）
+            context: 上下文管理器（可选，用于多轮对话）
+            step: 当前步骤数（用于日志记录）
+            last_action_coords: 上次动作坐标，用于绘制标记
+            draw_grid_and_marker: 是否绘制网格和上次动作标记（卡住检测、场景验证等不依赖坐标的场景可关闭）
+        
+        Returns:
+            {
+                "action_type": str,       # 动作类型 (tap, drag, type, finished, etc.)
+                "action_data": Dict,      # 动作参数（已转换为执行器格式）
+                "raw_response": str,      # 原始LLM响应
+                "thought": str,           # LLM的思考过程
+                "tokens": Dict,           # Token使用统计
+                "is_finished": bool,      # 任务是否完成
+                "success": bool           # 决策是否成功
+            }
+        """
+        messages = []
+        response = ""
+        tokens = {}
+        
+        try:
+            # 1. 准备截图数据
+            if screenshot_base64 is None and screenshot_path:
+                screenshot_base64, width, height = self._load_and_process_screenshot(
+                    screenshot_path, 
+                    last_action_coords=last_action_coords,
+                    draw_grid_and_marker=draw_grid_and_marker
+                )
+            elif screenshot_base64:
+                # 如果提供了base64，也需要处理网格
+                if draw_grid_and_marker:
+                    screenshot_base64 = self._add_grid_to_base64(screenshot_base64)
+            
+            if not screenshot_base64:
+                self._log(step, [], "", error_type="screenshot_load_failed")
+                return self._error_result("No screenshot provided", error_type="screenshot_load_failed")
+            
+            # 获取图像尺寸
+            if width is None or height is None:
+                img_width, img_height = self._get_image_size_from_base64(screenshot_base64)
+                width = width or img_width
+                height = height or img_height
+            
+            if not width or not height:
+                self._log(step, [], "", error_type="resolution_failed")
+                return self._error_result("Failed to get image size", error_type="resolution_failed")
+            
+            # 2. 准备上下文
+            if context is None:
+                context = ContextManager(system_prompt=self.system_prompt)
+            
+            # 清理base64前缀
+            clean_base64 = strip_base64_prefix(screenshot_base64)
+            
+            # 3. 添加截图到上下文
+            context.add_screenshot(clean_base64, width, height)
+            
+            # 4. 构建消息并调用LLM
+            messages = context.build_messages()
+            
+            logger.debug(f"[Step {step}] Calling LLM with {len(messages)} messages")
+            try:
+                response, _, tokens = self.llm.query(messages)
+            except Exception as e:
+                logger.error(f"[Step {step}] LLM调用失败: {e}")
+                self._log(step, messages, f"LLM Error: {e}", error_type="llm_call_failed")
+                return self._error_result(f"LLM call failed: {e}", error_type="llm_call_failed")
+            
+            if not response:
+                self._log(step, messages, "", error_type="empty_response")
+                return self._error_result("Empty LLM response", error_type="empty_response")
+            
+            # 5. 解析动作
+            parsed = parse_uitars_action(response)
+            action_type = parsed.get('action_type', '')
+            
+            if not action_type:
+                logger.warning(f"[Step {step}] Failed to parse action from response: {response[:200]}")
+                self._log(step, messages, response, tokens, error_type="action_parse_failed")
+                return self._error_result("Failed to parse action", raw_response=response, error_type="action_parse_failed")
+            
+            # 6. 转换为执行器格式
+            executor_action = convert_to_executor_action(parsed)
+            
+            # 7. 记录成功的日志
+            self._log(step, messages, response, tokens)
+            
+            # 8. 添加响应到上下文（为下一轮决策准备）
+            context.add_response(response)
+            
+            # 增加全局步数计数
+            GuiAgentDecisionMaker.total_steps += 1
+            
+            logger.info(f"[Step {step}] 决策成功: {action_type}")
+            
+            # 9. 返回决策结果
+            return {
+                "action_type": action_type,
+                "action_data": executor_action,
+                "raw_response": response,
+                "thought": parsed.get('thought', ''),
+                "tokens": tokens or {},
+                "is_finished": action_type in ('finished', 'report_stuck_reason'),
+                "success": True
+            }
+            
+        except KeyboardInterrupt:
+            raise
+        except Exception as e:
+            logger.exception(f"[Step {step}] Error in decide_next_action: {e}")
+            self._log(step, messages, response or str(e), tokens, error_type=f"exception:{type(e).__name__}")
+            # 返回错误结果而非抛出，让调用方决定如何处理
+            return self._error_result(f"Exception: {e}", error_type=f"exception:{type(e).__name__}")
+    
+    def _load_and_process_screenshot(self, screenshot_path: str, last_action_coords: tuple = None, draw_grid_and_marker: bool = True) -> Tuple[Optional[str], int, int]:
+        """从文件加载截图，添加网格和上次动作标记，并转换为base64"""
+        try:
+            img = Image.open(screenshot_path)
+            width, height = img.size
+            
+            if draw_grid_and_marker:
+                # 添加网格（坐标标注与agent坐标系一致）
+                if self.enable_grid:
+                    # 归一化模式标注0-1000，绝对模式标注实际像素
+                    coord_range = None if self.absolute_mode else (1000, 1000)
+                    img = draw_grid_on_image(img, coord_range=coord_range)
+                    logger.debug(f"截图已添加网格: {screenshot_path}")
+                
+                # 绘制上次动作坐标的绿圈标记
+                if last_action_coords:
+                    x, y = last_action_coords
+                    img = draw_last_action_marker(img, int(x), int(y))
+                    logger.debug(f"截图已添加上次动作标记: ({x}, {y})")
+            
+            # 转换为base64
+            img_base64 = image_to_base64(img)
+            return img_base64, width, height
+        except Exception as e:
+            logger.error(f"加载和处理截图失败 {screenshot_path}: {e}")
+            return None, 0, 0
+    
+    def _add_grid_to_base64(self, screenshot_base64: str) -> str:
+        """给base64截图添加网格"""
+        if not self.enable_grid:
+            return screenshot_base64
+        
+        try:
+            import io
+            clean = strip_base64_prefix(screenshot_base64)
+            img_data = base64.b64decode(clean)
+            img = Image.open(io.BytesIO(img_data))
+            img = draw_grid_on_image(img, coord_range=None if self.absolute_mode else (1000, 1000))
+            return image_to_base64(img)
+        except Exception as e:
+            logger.error(f"添加网格失败: {e}")
+            return screenshot_base64
+    
+    def _load_screenshot_base64(self, screenshot_path: str) -> Optional[str]:
+        """从文件加载截图并转换为base64"""
+        try:
+            with open(screenshot_path, 'rb') as f:
+                img_data = f.read()
+            return base64.b64encode(img_data).decode('utf-8')
+        except Exception as e:
+            logger.error(f"Failed to load screenshot from {screenshot_path}: {e}")
+            return None
+    
+    def _get_image_size_from_base64(self, base64_str: str) -> Tuple[int, int]:
+        """从base64字符串获取图像尺寸"""
+        try:
+            import io
+            clean = strip_base64_prefix(base64_str)
+            img_data = base64.b64decode(clean)
+            img = Image.open(io.BytesIO(img_data))
+            return img.size
+        except Exception as e:
+            logger.error(f"Failed to get image size: {e}")
+            return 0, 0
+    
+    def _error_result(self, error_msg: str, raw_response: str = "", error_type: str = "unknown") -> Dict[str, Any]:
+        """生成错误结果，包含错误分类用于统计"""
+        return {
+            "action_type": "",
+            "action_data": {},
+            "raw_response": raw_response,
+            "thought": "",
+            "tokens": {},
+            "is_finished": False,
+            "success": False,
+            "error": error_msg,
+            "error_type": error_type
+        }
+    
+    def verify_screen(
+        self,
+        category: str,
+        screenshot_path: str = None,
+        screenshot_base64: str = None,
+        get_screenshot_func=None
+    ) -> bool:
+        """
+        验证当前界面是否确实属于目标场景
+        
+        Args:
+            category: 目标场景类别 (login, payment, etc.)
+            screenshot_path: 截图文件路径
+            screenshot_base64: 截图的base64编码
+            get_screenshot_func: 获取截图的回调函数，返回 (path, base64) 或 base64 字符串
+        
+        Returns:
+            如果验证通过返回 True，否则返回 False
+        """
+        logger.info(f"[验证] 正在验证当前界面是否为: {category}")
+        
+        try:
+            # 获取截图
+            if screenshot_base64 is None:
+                if screenshot_path:
+                    screenshot_base64 = self._load_screenshot_base64(screenshot_path)
+                elif get_screenshot_func:
+                    result = get_screenshot_func()
+                    if isinstance(result, tuple):
+                        screenshot_path, screenshot_base64 = result
+                    else:
+                        screenshot_base64 = result
+                    if screenshot_path and not screenshot_base64:
+                        screenshot_base64 = self._load_screenshot_base64(screenshot_path)
+            
+            if not screenshot_base64:
+                logger.warning("[验证] 无法获取截图，默认通过验证")
+                return True
+            
+            clean_base64 = strip_base64_prefix(screenshot_base64)
+            
+            # 构建验证提示词
+            verification_prompt = get_verification_prompt(category)
+            
+            # 构建单次对话消息
+            messages = [
+                {
+                    "role": "user",
+                    "content": [
+                        {"type": "text", "text": verification_prompt},
+                        {
+                            "type": "image_url",
+                            "image_url": {"url": f"data:image/png;base64,{clean_base64}"}
+                        }
+                    ]
+                }
+            ]
+            
+            # 调用模型
+            response, _, token_usage = self.llm.query(messages)
+            
+            if not response:
+                logger.warning("[验证] 模型未返回响应，默认通过验证")
+                return True
+            
+            logger.info(f"[验证响应]\n{response},Input Tokens: {token_usage.get('input_tokens', 0)},Output Tokens: {token_usage.get('output_tokens', 0)}")
+            
+            # 解析结果
+            match = re.search(r"Result:\s*(YES|NO)", response, re.IGNORECASE)
+            if match:
+                result = match.group(1).upper()
+                if result == "YES":
+                    logger.info(f"[验证成功] 确认当前为 {category} 场景")
+                    return True
+                else:
+                    logger.info(f"[验证失败] 当前界面不符合 {category} 场景描述")
+                    return False
+            
+            # 如果没找到标准格式，简单检查关键词
+            if "YES" in response.upper() and "NO" not in response.upper():
+                return True
+            
+            return False
+            
+        except Exception as e:
+            logger.error(f"[验证] 验证过程中发生异常: {e}")
+            return True  # 发生异常时默认通过，避免中断流程
+    
+    def create_context(self, instruction: str = None) -> ContextManager:
+        """
+        创建新的上下文管理器
+        
+        Args:
+            instruction: 初始任务指令（可选）
+        
+        Returns:
+            ContextManager实例
+        """
+        context = ContextManager(system_prompt=self.system_prompt)
+        if instruction:
+            context.add_instruction(instruction)
+        return context
+
+
+if __name__ == "__main__":
+    """
+    模块化测试和调试入口
+    
+    用于单独测试和调试 GuiAgent 的决策功能，无需启动完整的自动化框架。
+    直接修改下面的配置参数即可进行测试。
+    
+    使用示例:
+        python -m DroidBot.guiagent_core.decision_maker
+        或
+        python DroidBot/guiagent_core/decision_maker.py
+    """
+    import sys
+    
+    # ==================== 配置参数（根据需要修改） ====================
+    
+    # === 截图模式配置 ===
+    # 设置为 True 可从连接的设备实时捕获截图，False 则使用指定的截图文件
+    USE_DEVICE_SCREENSHOT = True
+    
+    # 截图文件路径（当 USE_DEVICE_SCREENSHOT=False 时使用）
+    SCREENSHOT_PATH = "/path/to/screenshot.png"
+    
+    # === 设备配置（当 USE_DEVICE_SCREENSHOT=True 时使用） ===
+    # Android 设备序列号（None 表示使用默认连接的设备）
+    DEVICE_SERIAL = None
+    
+    # iOS WDA 服务地址
+    WDA_URL = ""
+    
+    # 设备截图保存目录
+    DEVICE_OUTPUT_DIR = "./output/guiagent_debug"
+    
+    # === 任务配置 ===
+    # 任务指令
+    INSTRUCTION = "探索当前界面并执行合理操作"
+    
+    # 平台类型: 'android', 'ios', 'desktop'
+    PLATFORM = "ios"
+    
+    # 屏幕分辨率 (width, height)，None 表示自动从截图获取
+    RESOLUTION = None  # 例如: (1080, 2340)
+    
+    # iOS scale factor (2.0 for @2x, 3.0 for @3x)
+    SCALE = 1.0
+    
+    # 最大执行步数（用于多轮测试）
+    MAX_STEPS = 1
+    
+    # 是否强制使用绝对坐标模式
+    ABSOLUTE_MODE = None  # None 表示自动判断
+    
+    # 是否禁用截图网格线
+    DISABLE_GRID = False
+    
+    # 验证模式：验证当前界面是否属于指定场景（None 表示不验证）
+    # 例如: "login", "payment" 等
+    VERIFY_CATEGORY = None
+    
+    # 是否启用调试模式（显示详细日志和原始响应）
+    DEBUG_MODE = False
+    
+    # ================================================================
+    
+    # 配置日志
+    logging.basicConfig(
+        level=logging.DEBUG if DEBUG_MODE else logging.INFO,
+        format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+    )
+    
+    if DEBUG_MODE:
+        logger.setLevel(logging.DEBUG)
+    
+    # 初始化设备（如果使用设备截图模式）
+    device = None
+    if USE_DEVICE_SCREENSHOT:
+        logger.info(f"\n{'='*60}")
+        logger.info(f"设备截图模式已启用")
+        logger.info(f"{'='*60}\n")
+        
+        try:
+            if PLATFORM == "android":
+                logger.info("正在连接 Android 设备...")
+                try:
+                    from DroidBot.platforms.android import AndroidDevice
+                except ImportError:
+                    from platforms.android import AndroidDevice
+                
+                device = AndroidDevice(
+                    device_serial=DEVICE_SERIAL,
+                    output_dir=DEVICE_OUTPUT_DIR
+                )
+                device.set_up()
+                device.connect()
+                logger.info(f"✓ Android 设备已连接: {device.device_serial}")
+                
+            elif PLATFORM == "ios":
+                logger.info("正在连接 iOS 设备...")
+                try:
+                    from DroidBot.platforms.ios import IOSDevice
+                except ImportError:
+                    from platforms.ios import IOSDevice
+                
+                device = IOSDevice(
+                    wda_url=WDA_URL,
+                    output_dir=DEVICE_OUTPUT_DIR
+                )
+                device.set_up()
+                device.connect()
+                logger.info(f"✓ iOS 设备已连接 (WDA: {WDA_URL})")
+                
+            else:
+                logger.error(f"设备截图模式不支持平台: {PLATFORM}")
+                sys.exit(1)
+            
+            # 从设备捕获截图
+            logger.info("正在从设备捕获截图...")
+            SCREENSHOT_PATH = device.take_screenshot()
+            
+            if not SCREENSHOT_PATH:
+                logger.error("从设备捕获截图失败")
+                sys.exit(1)
+            
+            logger.info(f"✓ 截图已保存: {SCREENSHOT_PATH}")
+            
+        except Exception as e:
+            logger.error(f"设备连接或截图失败: {e}")
+            if device:
+                try:
+                    device.disconnect()
+                except:
+                    pass
+            sys.exit(1)
+    else:
+        # 文件模式：检查截图文件是否存在
+        if not os.path.exists(SCREENSHOT_PATH):
+            logger.error(f"截图文件不存在: {SCREENSHOT_PATH}")
+            logger.error(f"请修改 SCREENSHOT_PATH 参数为有效的截图文件路径")
+            logger.error(f"或者设置 USE_DEVICE_SCREENSHOT = True 从设备捕获截图")
+            sys.exit(1)
+    
+    # 创建决策引擎
+    logger.info(f"\n{'='*60}")
+    logger.info(f"初始化 GuiAgent 决策引擎")
+    logger.info(f"{'='*60}")
+    logger.info(f"平台: {PLATFORM}, 分辨率: {RESOLUTION}, Scale: {SCALE}")
+    
+    try:
+        decision_maker = GuiAgentDecisionMaker(
+            platform=PLATFORM,
+            resolution=RESOLUTION,
+            scale=SCALE,
+            absolute_mode=ABSOLUTE_MODE
+        )
+        
+        # 配置网格
+        if DISABLE_GRID:
+            decision_maker.enable_grid = False
+            logger.info("已禁用截图网格")
+        
+        # 验证模式
+        if VERIFY_CATEGORY:
+            logger.info(f"\n{'='*60}")
+            logger.info(f"验证模式: 检查当前界面是否为 {VERIFY_CATEGORY} 场景")
+            logger.info(f"{'='*60}\n")
+            
+            is_valid = decision_maker.verify_screen(
+                category=VERIFY_CATEGORY,
+                screenshot_path=SCREENSHOT_PATH
+            )
+            
+            print(f"\n{'='*60}")
+            print(f"验证结果: {'✓ 通过' if is_valid else '✗ 不通过'}")
+            print(f"{'='*60}\n")
+            
+            sys.exit(0 if is_valid else 1)
+        
+        # 决策模式
+        logger.info(f"\n{'='*60}")
+        logger.info(f"任务指令: {INSTRUCTION}")
+        logger.info(f"截图文件: {SCREENSHOT_PATH}")
+        logger.info(f"最大步数: {MAX_STEPS}")
+        logger.info(f"{'='*60}\n")
+        
+        # 创建上下文
+        context = decision_maker.create_context(instruction=INSTRUCTION)
+        
+        # 多轮决策测试
+        for step in range(MAX_STEPS):
+            logger.info(f"\n{'='*60}")
+            logger.info(f"第 {step + 1} 步决策")
+            logger.info(f"{'='*60}\n")
+            
+            # 执行决策
+            result = decision_maker.decide_next_action(
+                screenshot_path=SCREENSHOT_PATH,
+                context=context,
+                step=step + 1
+            )
+            
+            # 打印结果
+            print(f"\n{'='*60}")
+            print(f"决策结果 (步骤 {step + 1})")
+            print(f"{'='*60}")
+            print(f"成功: {result.get('success')}")
+            print(f"动作类型: {result.get('action_type')}")
+            print(f"是否完成: {result.get('is_finished')}")
+            
+            if result.get('thought'):
+                print(f"\n思考过程:")
+                print(f"  {result.get('thought')}")
+            
+            if result.get('action_data'):
+                print(f"\n动作数据:")
+                for key, value in result.get('action_data', {}).items():
+                    print(f"  {key}: {value}")
+            
+            if result.get('tokens'):
+                tokens = result.get('tokens')
+                print(f"\nToken 使用:")
+                print(f"  输入: {tokens.get('input_tokens', 0)}")
+                print(f"  输出: {tokens.get('output_tokens', 0)}")
+                print(f"  总计: {tokens.get('total', 0)}")
+            
+            if result.get('error'):
+                print(f"\n错误信息: {result.get('error')}")
+                print(f"错误类型: {result.get('error_type')}")
+            
+            if DEBUG_MODE and result.get('raw_response'):
+                print(f"\n原始响应:")
+                print(f"{result.get('raw_response')}")
+            
+            print(f"{'='*60}\n")
+            
+            # 检查是否完成或失败
+            if result.get('is_finished'):
+                logger.info(f"✓ 任务完成！")
+                break
+            
+            if not result.get('success'):
+                logger.error(f"✗ 决策失败，停止执行")
+                sys.exit(1)
+            
+            # 如果是多步测试，模拟等待
+            if step < MAX_STEPS - 1:
+                logger.info("等待 2 秒后继续下一步...")
+                time.sleep(2)
+        
+        logger.info(f"\n测试完成！日志文件: {decision_maker.log_file_path}")
+        
+    except KeyboardInterrupt:
+        logger.info("\n用户中断测试")
+        sys.exit(130)
+    except Exception as e:
+        logger.exception(f"测试过程中发生异常: {e}")
+        sys.exit(1)
+    finally:
+        # 清理设备连接
+        if device:
+            logger.info("\n正在断开设备连接...")
+            try:
+                device.disconnect()
+                logger.info("✓ 设备已断开")
+            except Exception as e:
+                logger.warning(f"设备断开时出现警告: {e}")
+

DroidBot/guiagent_core/llm_client.py

@@ -0,0 +1,124 @@
+from typing import List, Dict, Any, Optional, Tuple
+import os
+import requests
+import time
+import json
+import logging
+import traceback
+from .constants import MAX_HISTORY_LENGTH, load_config
+
+logger = logging.getLogger(__name__)
+
+class LLMClient:
+    """
+    Client for interacting with LLM models via KeyPool service.
+    Handles conversation history and API requests.
+    """
+    
+    def __init__(self, model_name: str, keypool_url: str=None, api_key: str=None):
+        """
+        Initialize LLM client.
+        
+        Args:
+            model_name: Name of the model to use (e.g., "gemini-2.5-flash-lite")
+                        Must match a model_name in KeyPool's keys.yaml
+        """
+        self.history: List[Dict[str, Any]] = []
+        
+        # Default timeout for requests
+        self.timeout: int = 120
+        
+        # KeyPool Service Configuration
+        config = load_config() if (keypool_url is None or api_key is None) else {}
+        if model_name is None:
+            model_name = (
+                config.get("CURRENT_MODEL_NAME")
+                or config.get("llm_api", {}).get("current_model_name", "")
+                or ""
+            )
+        self.model_name = str(model_name).strip()
+        if keypool_url is None:
+            keypool_url = (
+                config.get("KEY_POOL_URL")
+                or config.get("llm_api", {}).get("key_pool_url", "")
+                or ""
+            )
+        self.keypool_url = str(keypool_url).strip().rstrip("/")
+        
+        if api_key is None:
+            api_key = (
+                config.get("KEY_POOL_API_KEY")
+                or config.get("llm_api", {}).get("key_pool_api_key", "")
+                or os.environ.get("KEY_POOL_API_KEY", "")
+                or ""
+            )
+        self.api_key = str(api_key).strip()
+        
+        logger.info(f"LLMClient initialized for model: {self.model_name}")
+
+    def query(self, messages: List[Dict[str, Any]], extra_params: Optional[Dict[str, Any]] = None) -> Tuple[str, Any, Dict[str, Any]]:
+        """
+        Send a query to the LLM via KeyPool.
+        
+        Args:
+            messages: List of message dictionaries
+            extra_params: Optional dictionary of extra parameters to override defaults
+            
+        Returns:
+            Tuple[str, Any, Dict[str, Any]]: (response_content, reasoning_content, token_usage)
+        """
+        try:
+            # Prepare payload for KeyPool
+            payload = {
+                "model": self.model_name,
+                "messages": messages,
+                "stream": False
+            }
+            
+            # Add extra parameters if provided
+            if extra_params:
+                payload.update(extra_params)
+            
+            # Build headers (OpenAI-compatible /v1/chat/completions endpoint)
+            headers = {"Content-Type": "application/json"}
+            if self.api_key:
+                headers["Authorization"] = f"Bearer {self.api_key}"
+            
+            # Send request to KeyPool (OpenAI-compatible endpoint)
+            response = requests.post(
+                f"{self.keypool_url}/v1/chat/completions",
+                json=payload,
+                headers=headers,
+                timeout=self.timeout
+            )
+            
+            response.raise_for_status()
+            result = response.json()
+            
+            content = ""
+            reasoning = None
+            usage = {}
+            
+            if "choices" in result and len(result["choices"]) > 0:
+                message = result["choices"][0]["message"]
+                content = message.get("content", "")
+                reasoning = message.get("reasoning_content")
+                
+            if "usage" in result:
+                usage = result["usage"]
+            return content, reasoning, usage
+                
+        except Exception as e:
+            logger.error(f"Error querying KeyPool: {e}")
+            logger.error(traceback.format_exc())
+            return "", None, {}
+
+    def add_history(self, role: str, content: Any):
+        """Add a message to history"""
+        self.history.append({"role": role, "content": content})
+        if len(self.history) > MAX_HISTORY_LENGTH:
+            self.history.pop(0)
+
+    def clear_history(self):
+        """Clear conversation history"""
+        self.history = []

DroidBot/guiagent_core/prompt_builder.py

@@ -0,0 +1,216 @@
+"""
+Prompt Builder - 生成不同平台的系统提示词
+
+Extracted and adapted from GuiAgent/config/prompts.py for modular use in DroidBot.
+Added iOS support with scale parameter.
+"""
+
+# Resolution placeholder for dynamic replacement
+RESOLUTION_PLACEHOLDER = "<RESOLUTION>"
+
+# ==============================================================================
+# Action Space Definitions
+# ==============================================================================
+
+IOS_ACTION_SPACES = """
+tap(center='[x, y]')
+long_tap(center='[x, y]', duration_ms='1000')
+drag(start_center='[x, y]', end_center='[x, y]')
+type(content='')
+key_press(key='HOME or BACK or ENTER')
+wait()
+receive_email()
+finished()
+login_ios()
+solve_slider_captcha(captcha_region='[x1, y1, x2, y2]', slider_position='[x, y]')
+solve_image_captcha(captcha_region='[x1, y1, x2, y2]', input_field='[x, y]')
+report_stuck_reason(reason_code='0-12', message='reason description')
+"""
+
+ANDROID_ACTION_SPACES = """
+tap(center='[x, y]')
+long_tap(center='[x, y]', duration_ms='1000')
+drag(start_center='[x, y]', end_center='[x, y]')
+type(content='')
+key_press(key='HOME or BACK or ENTER')
+wait()
+receive_email()
+solve_slider_captcha(captcha_region='[x1, y1, x2, y2]', slider_position='[x, y]')
+solve_image_captcha(captcha_region='[x1, y1, x2, y2]', input_field='[x, y]')
+finished()
+report_stuck_reason(reason_code='0-12', message='reason description')
+"""
+
+WINDOWS_ACTION_SPACES = """
+click(center='[x, y]')
+double_click(center='[x, y]')
+right_click(center='[x, y]')
+drag(start_center='[x, y]', end_center='[x, y]')
+type(content='')
+key_press(key='ctrl+v or alt+tab or enter')
+scroll(start_center='[x, y]', direction='down or up or right or left')
+wait()
+receive_email()
+finished()
+"""
+
+
+def _get_coord_instruction(absolute_mode: bool, resolution_str: str, scale: float = 1.0) -> str:
+    """
+    生成坐标指令
+    
+    Args:
+        absolute_mode: 是否使用绝对坐标模式
+        resolution_str: 分辨率字符串
+        scale: iOS scale factor (仅用于 iOS)
+        
+    Returns:
+        坐标指令字符串
+    """
+    if absolute_mode:
+        coord_inst = f"- Please return the coordinate values relative to the actual resolution of the screenshot, Width and height coordinate (x,y) from left-top (0, 0) to right-bottom {resolution_str}."
+        if scale > 1.0:
+            # iOS specific: add scale information
+            coord_inst += f" Note: Device scale is {scale}x, screenshot is in physical pixels ({resolution_str}), but coordinates should be in logical points."
+        return coord_inst
+    else:
+        return f"- Width and height coordinate (x,y) from left-top (0, 0) to right-bottom (1000, 1000) Please give the coordinates as 1000-normalized relative coordinates.**"
+
+
+def _build_base_prompt(action_space: str, coord_instruction: str, extra_notes: str = "") -> str:
+    """
+    构建基础提示词模板
+
+    Args:
+        action_space: 动作空间字符串
+        coord_instruction: 坐标指令
+        extra_notes: 额外的注意事项
+
+    Returns:
+        完整的提示词字符串
+    """
+    notes = "- Write your thought in one sentence in `Thought` part."
+    if extra_notes:
+        notes += f"\n{extra_notes}"
+    notes += "\n- Check the conversation history and the current state of the application, do not repeat the same action."
+    notes += f"\n{coord_instruction}"
+    notes += "\n- You MUST provide the coordinates of the center of the target element's bounding box **"
+
+    # 添加错误恢复策略
+    notes += """
+
+## Error Recovery Strategy
+- 如果收到"✗ 上次操作失败"的反馈：
+  1. 点击未命中：参考绿色圆圈标记判断偏移方向，调整坐标（偏左则向右移10-30像素，偏上则向下移）
+  2. 界面无变化：尝试滚动查看更多内容，或使用 BACK 键返回，或尝试长按等替代操作
+  3. 输入未生效：先 tap 输入框聚焦，等待0.5秒，再 type 输入，最后按 ENTER 确认
+  4. 连续3次相同错误：调用 report_stuck_reason 或尝试完全不同的路径"""
+
+    return f"""
+You are a GUI agent. You are given a task and your action history, with screenshots. You need to perform the next action to complete the task.
+## Action Space
+{action_space.strip()}
+## Note
+{notes}
+## Output Format
+```
+Thought: ...
+Action: ...
+```
+"""
+
+
+def get_ios_prompt(resolution: tuple = None, scale: float = 1.0, absolute_mode: bool = True) -> str:
+    """
+    生成 iOS 系统提示词
+    
+    Args:
+        resolution: 屏幕分辨率 (physical_width, physical_height)，如果为 None 则使用占位符
+                   注意：这是截图的物理像素分辨率，不是逻辑分辨率
+        scale: iOS scale factor (2.0 for @2x, 3.0 for @3x)
+        absolute_mode: 是否使用绝对坐标（iOS 推荐使用 True）
+        
+    Returns:
+        系统提示词字符串
+    """
+    resolution_str = f"({resolution[0]}, {resolution[1]})" if resolution else RESOLUTION_PLACEHOLDER
+    
+    coord_instruction = _get_coord_instruction(absolute_mode, resolution_str, scale)
+    
+    extra_notes = "- Both of `Thought` part and `Action` part need to be filled out."
+    extra_notes += "\n- IMPORTANT: If you see the iOS Home Screen (SpringBoard with app icons grid), it means you have accidentally left the target app. You MUST immediately call `finished()` to end the session. Do NOT try to re-open the app or perform any other actions on the Home Screen."
+    extra_notes += "\n- IMPORTANT: 上一次点击(tap/long_tap)或拖拽(drag)的起始坐标会以绿色圆圈标记在截图上。如果你发现多次操作后仍停留在当前页面，请参考绿圈位置判断上次点击是否命中目标元素，并相应调整坐标。"
+    if scale > 1.0:
+        extra_notes += f"\n- IMPORTANT: Screenshot resolution is {resolution_str} (physical pixels). Device scale is {scale}x. When you see an element at pixel position (x, y) in the screenshot, the tap coordinate should be (x/{scale}, y/{scale}) in logical points."
+    
+    return _build_base_prompt(IOS_ACTION_SPACES, coord_instruction, extra_notes)
+
+
+def get_android_prompt(resolution: tuple = None, absolute_mode: bool = False) -> str:
+    """
+    生成 Android 系统提示词
+    
+    Args:
+        resolution: 屏幕分辨率 (width, height)，如果为 None 则使用占位符
+        absolute_mode: 是否使用绝对坐标
+        
+    Returns:
+        系统提示词字符串
+    """
+    resolution_str = f"({resolution[0]}, {resolution[1]})" if resolution else RESOLUTION_PLACEHOLDER
+    
+    coord_instruction = _get_coord_instruction(absolute_mode, resolution_str)
+    
+    extra_notes = """- Both of `Thought` part and `Action` part need to be filled out.
+- 【滑块验证码】当检测到滑块验证码（需要拖动滑块到缺口位置）时，使用 solve_slider_captcha 动作。captcha_region 是验证码图片区域的坐标 [左上角x, 左上角y, 右下角x, 右下角y]（仅包含验证码图片区域，不包含滑块），slider_position 是可拖动滑块按钮的中心坐标 [x, y]。此动作会自动识别缺口位置并拖动滑块完成验证。
+- 【图片验证码】当检测到图片验证码（如字母/数字识别码）时，使用 solve_image_captcha 动作。captcha_region 是验证码图片区域坐标，input_field 是验证码输入框的中心坐标。此动作会自动识别图片中的文字并输入到指定输入框。"""
+    return _build_base_prompt(ANDROID_ACTION_SPACES, coord_instruction, extra_notes)
+
+
+def get_desktop_prompt(resolution: tuple = None, absolute_mode: bool = False) -> str:
+    """
+    生成 Windows 桌面系统提示词
+    
+    Args:
+        resolution: 屏幕分辨率 (width, height)，如果为 None 则使用占位符
+        absolute_mode: 是否使用绝对坐标
+        
+    Returns:
+        系统提示词字符串
+    """
+    resolution_str = f"({resolution[0]}, {resolution[1]})" if resolution else RESOLUTION_PLACEHOLDER
+    coord_instruction = _get_coord_instruction(absolute_mode, resolution_str)
+    
+    return _build_base_prompt(WINDOWS_ACTION_SPACES, coord_instruction)
+
+
+def get_verification_prompt(category: str) -> str:
+    """
+    生成界面验证提示词，用于二次确认当前界面是否为目标场景。
+    
+    Args:
+        category: 目标场景类别 (login, payment, etc.)
+        
+    Returns:
+        提示词字符串
+    """
+    return f"""
+You are a GUI verification expert. Your task is to judge whether the current screen is relevant to the "{category}" task based on the screenshot.
+
+## Task
+1. Analyze the UI elements, text, and layout in the screenshot.
+2. Determine if the screen is related to "{category}".
+   - It IS related if:
+     - It is the "{category}" page itself (e.g., login form).
+     - It contains an entry point to "{category}" (e.g., a "Login" button on a welcome screen).
+     - It is a relevant intermediate step (e.g., account type selection before registering).
+     - It is a pop-up or overlay that might appear during "{category}" (e.g., permission request).
+3. If it IS related, output "YES" and a brief reason.
+4. If it is NOT related (e.g., a completely different app, a game screen when asking for login), output "NO" and the reason.
+
+## Output Format
+```
+Result: YES/NO
+Reason: ...
+```
+"""

DroidBot/guiagent_core/scene_config_loader.py

@@ -0,0 +1,192 @@
+"""
+Scene Configuration Loader
+场景配置加载器 - 支持平台特定配置覆盖
+
+功能:
+1. 从统一配置文件 scene_configs.json 加载配置
+2. 合并默认配置和平台特定覆盖配置
+3. 支持深度合并策略 (平台配置覆盖默认配置的同名场景)
+4. 支持场景移除 (通过 null 值标记)
+"""
+
+import json
+import os
+import re
+import logging
+from pathlib import Path
+from typing import Dict, Any, Optional
+from copy import deepcopy
+
+logger = logging.getLogger(__name__)
+
+# 配置文件路径
+CONFIG_FILE = Path(__file__).parent / "scene_configs.json"
+
+
+def _expand_env_vars(value):
+    """递归展开 ${ENV_VAR_NAME} 格式的环境变量引用"""
+    if isinstance(value, str):
+        def _replacer(match):
+            env_value = os.environ.get(match.group(1))
+            if env_value is None:
+                return match.group(0)
+            return env_value
+        return re.sub(r'\$\{([^}]+)\}', _replacer, value)
+    elif isinstance(value, dict):
+        return {k: _expand_env_vars(v) for k, v in value.items()}
+    elif isinstance(value, list):
+        return [_expand_env_vars(v) for v in value]
+    return value
+
+
+def deep_merge(base: Dict, override: Dict) -> Dict:
+    """
+    深度合并两个字典,override 中的值会覆盖 base 中的同名键
+    
+    Args:
+        base: 基础配置字典
+        override: 覆盖配置字典
+    
+    Returns:
+        合并后的字典
+    
+    特殊处理:
+        - 如果 override 中的值为 None,则从结果中移除该键
+        - 对于嵌套字典,递归进行深度合并
+        - 对于列表,直接覆盖(不合并列表内容)
+    """
+    result = deepcopy(base)
+    
+    for key, value in override.items():
+        # 特殊处理: None 值表示移除该场景
+        if value is None:
+            result.pop(key, None)
+            continue
+        
+        # 如果两者都是字典,递归合并
+        if key in result and isinstance(result[key], dict) and isinstance(value, dict):
+            result[key] = deep_merge(result[key], value)
+        else:
+            # 否则直接覆盖
+            result[key] = deepcopy(value)
+    
+    return result
+
+
+def load_scene_config(platform: str = "android") -> Dict[str, Any]:
+    """
+    加载场景配置,自动合并默认配置和平台覆盖配置
+    
+    Args:
+        platform: 平台名称 (android/ios/web/windows)
+    
+    Returns:
+        合并后的配置字典,包含 keywords 和 instructions
+        
+    Raises:
+        FileNotFoundError: 配置文件不存在
+        json.JSONDecodeError: 配置文件格式错误
+        KeyError: 缺少必要的配置项
+    
+    Example:
+        >>> config = load_scene_config("android")
+        >>> print(config["keywords"]["login"])
+        ['登录', '登陆', 'login', ...]
+    """
+    # 1. 检查配置文件是否存在
+    if not CONFIG_FILE.exists():
+        raise FileNotFoundError(f"配置文件不存在: {CONFIG_FILE}")
+    
+    # 2. 加载统一配置文件
+    try:
+        with open(CONFIG_FILE, 'r', encoding='utf-8') as f:
+            all_configs = json.load(f)
+    except json.JSONDecodeError as e:
+        logger.error(f"配置文件格式错误: {e}")
+        raise
+    
+    # 3. 验证配置文件结构
+    if "default" not in all_configs:
+        raise KeyError("配置文件缺少 'default' 节点")
+    
+    # 4. 获取默认配置
+    default_config = all_configs["default"]
+    
+    # 验证默认配置完整性
+    if "keywords" not in default_config or "instructions" not in default_config:
+        raise KeyError("默认配置缺少 'keywords' 或 'instructions' 字段")
+    
+    # 5. 获取平台覆盖配置 (如果存在)
+    platform_config = all_configs.get(platform, {})
+    
+    # 6. 深度合并配置 (平台配置优先)
+    merged_config = {
+        "keywords": deep_merge(
+            default_config.get("keywords", {}),
+            platform_config.get("keywords", {})
+        ),
+        "instructions": deep_merge(
+            default_config.get("instructions", {}),
+            platform_config.get("instructions", {})
+        ),
+        "step_limits": deep_merge(
+            default_config.get("step_limits", {}),
+            platform_config.get("step_limits", {})
+        )
+    }
+    
+    # 7. 展开环境变量引用 (格式: ${VAR_NAME})
+    merged_config = _expand_env_vars(merged_config)
+
+    # 8. 记录日志
+    scenes = list(merged_config["keywords"].keys())
+    if platform_config.get("keywords") or platform_config.get("instructions") or platform_config.get("step_limits"):
+        overridden_scenes = (
+            set(platform_config.get("keywords", {}).keys())
+            | set(platform_config.get("instructions", {}).keys())
+            | set(platform_config.get("step_limits", {}).keys())
+        )
+        logger.info(f"[SceneConfig] 平台 '{platform}' 加载配置成功, 场景数: {len(scenes)}, 覆盖场景: {overridden_scenes}")
+    else:
+        logger.info(f"[SceneConfig] 平台 '{platform}' 使用默认配置, 场景数: {len(scenes)}")
+    
+    return merged_config
+
+
+def get_available_scenes(platform: str = "android") -> list:
+    """
+    获取指定平台可用的场景列表
+    
+    Args:
+        platform: 平台名称
+    
+    Returns:
+        场景名称列表
+    """
+    config = load_scene_config(platform)
+    return list(config["keywords"].keys())
+
+
+# 测试代码 (仅在直接运行此模块时执行)
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO)
+    
+    print("=" * 60)
+    print("场景配置加载器测试")
+    print("=" * 60)
+    
+    # 测试默认配置加载
+    print("\n[测试 1] 加载 Android 平台默认配置:")
+    android_config = load_scene_config("android")
+    print(f"  可用场景: {list(android_config['keywords'].keys())}")
+    print(f"  login 关键词数量: {len(android_config['keywords']['login'])}")
+    
+    # 测试其他平台
+    for platform in ["ios", "web", "windows"]:
+        print(f"\n[测试 2] 加载 {platform} 平台配置:")
+        config = load_scene_config(platform)
+        print(f"  可用场景: {list(config['keywords'].keys())}")
+    
+    print("\n" + "=" * 60)
+    print("测试完成!")
+    print("=" * 60)

DroidBot/guiagent_core/scene_configs.json

@@ -0,0 +1,150 @@
+{
+    "_comment": "GuiAgent场景配置 - 统一配置文件 (所有平台配置集中管理)",
+    "_version": "1.1.0",
+    "_description": "默认配置包含所有场景的关键词、提示词和步数限制，平台配置仅需覆盖需要定制的场景",
+    "default": {
+        "keywords": {
+            "login": [
+                "登录",
+                "登陆",
+                "login",
+                "signin",
+                "sign in",
+                "log in",
+                "sign-in",
+                "log-in",
+                "already have an account",
+                "with Google",
+                "with google",
+                "Continue with Google",
+                "continuewithgoogle",
+                "Log in with Google",
+                "loginwithgoogle",
+                "Continue with Phone Number",
+                "continuewithphonenumber"
+            ],
+            "register": [
+                "注册",
+                "注册账号",
+                "register",
+                "signup",
+                "sign up",
+                "sign-up",
+                "create account",
+                "create an account",
+                "create new account",
+                "register"
+            ],
+            "ad": [
+                "广告",
+                "推广",
+                "宣传",
+                "跳过",
+                "advertisement",
+                "promotion"
+            ],
+            "captcha": [
+                "验证码",
+                "校验码",
+                "CAPTCHA",
+                "verification"
+            ],
+            "payment": [
+                "支付",
+                "付款",
+                "购买",
+                "payment",
+                "pay"
+            ],
+            "game": [
+                "开始游戏",
+                "启动游戏",
+                "start game",
+                "play"
+            ],
+            "search": [
+                "搜索",
+                "查找",
+                "查询",
+                "search"
+            ]
+        },
+        "instructions": {
+            "login": "完成登录操作，首先尝试通过谷歌账户直接登录，如果存在已经登陆好的邮箱，直接使用且不要使用后面的tpshosai邮箱，若无法通过谷歌账号登录，则尝试输入手机号 ${SCENE_PHONE} 进行登录，或者使用邮箱${SCENE_EMAIL}、密码${SCENE_PASSWORD}、用户名${SCENE_USERNAME}完成注册，其他信息自行填写，无论是手机号还是邮箱，均通过调用receive_email()函数获取验证码（手机验证码也通过此途径获取），根据内容完成验证，登录成功后进入应用主界面，而后结束对话",
+            "register": "完成注册操作，首先尝试通过邮箱${SCENE_EMAIL}、密码${SCENE_PASSWORD}、用户名${SCENE_USERNAME}完成注册，其他信息自行填写，若不行，输入手机号 ${SCENE_PHONE} 进行注册。 无论是手机号还是邮箱，均通过调用receive_email()函数获取验证码（手机验证码也通过此途径获取），根据内容完成验证，登录成功后进入应用主界面，而后结束对话",
+            "payment": "请先点击结算按钮，弹出付款弹窗后退出支付操作，直到回到非支付界面，而后结束对话",
+            "game": "开始游戏，点击开始游戏按钮或相关操作进入游戏对局，而后结束对话",
+            "ad": "关闭当前出现的广告/推广/宣传内容，或使用Back键返回到上一个主界面或正常操作界面，而后结束对话",
+            "captcha": "输入验证码完成当前操作，如果需要获取新的验证码，请尝试点击重新发送或刷新按钮，而后结束对话",
+            "search": "在搜索框中输入'tplink'进行搜索，而后结束对话",
+            "game_initial": "你需要进入游戏的游玩流程，即关闭阻碍的弹窗信息，点击 继续游戏 或者 开始游戏 或者 有着进入游戏对局类似意义的图标和按钮，如果第一次游玩需要填写信息则填写信息，进入游戏对局后即完成任务。",
+            "explore_stuck": "应用探索疑似卡住，请按以下逻辑处理：\n1. 【试探与交互】当前信息不足或遇到简单阻碍时，允许进行试探性操作（如点击勾选框、寻找关闭按钮、滑动页面）。若是未勾选隐私政策、可跳过的内购推广或引导等简单场景，请自行尝试点击通过，不要立即报错。\n2. 【谨慎研判】若经过试探（如仔细寻找并点击“X”、“跳过”、“暂不”等）仍无法推进，分析界面并调用 report_stuck_reason 反馈。\n注意以下易误判场景：\n- reason_code 1 (登录注册需人工辅助)：【严格限制】仅当遇到需要真实手机号接收短信验证码、人脸识别、复杂图形验证码等 AI 绝对无法绕过的硬性限制时才使用。不要因为未勾选协议或普通的输入框判定为1。\n- reason_code 4 (地区网络限制)：【严格限制】界面必须出现明确的文字提示（如“不在服务区”或“Not available in your country”），严禁将普通的加载超时或白屏误判为地区限制。\n- reason_code 5 (需付费才能使用)：【严格限制】仅针对“强制付费墙”（即没有任何关闭、跳过、试用等入口，不付费绝对无法进入主界面或继续使用的情况）。遇到普通的VIP订阅弹窗或内购推广页面，必须先尝试寻找关闭或跳过按钮，切勿直接报错。\n\nreason_code 列表: 1=严格的人工辅助(验证码/人脸); 2=启动异常(黑屏/闪退/卡启动); 3=无明显异常; 4=明确的地区限制(需文字证据); 5=强制付费墙(无任何跳过选项); 6=虚拟手机号无效; 7=虚拟身份信息无效; 8=注册校验失败; 9=非开放注册(需邀请码); 10=应用停服下架; 11=需实体证件(银行卡/驾照); 12=模拟器处于root下无法使用; 0=其他异常(message需详细说明)。\n\n如果在试探后成功突破卡点，则无需调用报错，继续正常探索即可。",
+            "stuck_escape": "应用探索卡住但无异常。请尝试脱离当前卡住状态：1) 如果是系统弹窗（权限/通知/追踪），点击允许或关闭；2) 如果是应用内弹窗（广告/促销/评分/引导），关闭或跳过；3) 如果是加载中/转圈，等待或点击取消；4) 如果页面无操作入口，尝试返回、滑动或点击空白区域；5) 如果是确认对话框，点击确认或取消。处理完毕后调用 finished 结束任务。"
+        },  
+        "step_limits": {
+            "_comment": "各场景最大执行步数，未配置的场景使用 default_steps",
+            "default_steps": 20, 
+            "login": 40,
+            "register": 40,
+            "ad": 10,
+            "captcha": 20,
+            "payment": 20,
+            "game": 20,
+            "game_initial": 20,
+            "search": 20,
+            "explore_stuck": 10,
+            "stuck_escape": 15
+        }
+    },
+    "android": {
+        "_comment": "Android 平台覆盖配置 - 仅写需要覆盖的场景，未覆盖的自动使用 default 配置",
+        "keywords": {
+            "search": [],
+            "game": []
+        },
+        "instructions": {}
+    },
+    "ios": {
+        "_comment": "iOS 平台覆盖配置 - 仅写需要覆盖的场景，未覆盖的自动使用 default 配置",
+        "keywords": {
+            "search": [],
+            "game": [],
+            "login": [
+                "登录",
+                "登陆",
+                "账号继续",
+                "帐号继续",
+                "login",
+                "signin",
+                "sign in",
+                "log in",
+                "sign-in",
+                "log-in",
+                "already have an account",
+                "AppleID",
+                "appleid",
+                "with AppleID",
+                "with appleid",
+                "Continue with AppleID",
+                "continue with appleid",
+                "Log in with AppleID",
+                "log in with appleid"
+            ]
+        },
+        "instructions": {
+            "login": "完成登录操作，首先尝试点击类似通过Apple账号直接登录的按钮，当看到系统弹出的Apple ID密码登录对话框（要求通过密码登录）时，调用login_ios()函数来自动输入密码完成登录，不要自己手动操作密码输入。只有在收到login_ios()返回的错误信息后，才尝试自己手动在密码输入框中输入密码 ${IOS_PASSWORD} 并点击登录，注意不要在输入框已输入了密码的情况下(此时密码输入框会显示密码隐藏后的11个黑点)重复输入导致密码错误。若无法通过Apple账号登录，则使用邮箱${SCENE_EMAIL}、密码${SCENE_PASSWORD}、用户名${SCENE_USERNAME}完成注册或登录，其他信息自行填写，通过调用receive_email()函数获取验证码邮件，根据内容完成验证。如果登录后还需要验证手机号，则尝试选择区域爱沙尼亚（estonia）输入手机号 ${SCENE_PHONE} 进行登录，无论是手机号还是邮箱，均通过调用receive_email()函数获取验证码（手机验证码也通过此途径获取）登录成功后进入应用主界面，就结束对话调用finished",
+            "register": "完成注册操作,尝试通过Apple账号直接注册，若无法通过Apple账号注册，则使用邮箱${SCENE_EMAIL}、密码${SCENE_PASSWORD}、用户名${SCENE_USERNAME}完成注册，其他信息自行填写，通过调用receive_email()函数获取验证码邮件，根据内容完成验证，登录成功后进入应用主界面，而后结束对话调用finished",
+            "stuck_handler": "应用可能卡在某个页面无法继续操作。请分析当前截图，判断页面状态：如果是系统弹窗（权限/通知/追踪），点击允许或关闭；如果是应用内弹窗（广告/促销/评分），关闭它；如果是登录/注册页面，尝试通过Apple账号登录或跳过；如果是引导页，点击跳过/继续/下一步；如果是加载失败，点击重试或返回；如果页面正常尝试点击页面中最可能正常继续使用应用的按键，然后结束任务。处理完毕后结束任务调用finished。"
+        }
+    },
+    "web": {
+        "_comment": "Web 平台覆盖配置 - 仅写需要覆盖的场景，未覆盖的自动使用 default 配置",
+        "keywords": {},
+        "instructions": {}
+    },
+    "windows": {
+        "_comment": "Windows 平台覆盖配置 - 仅写需要覆盖的场景，未覆盖的自动使用 default 配置",
+        "keywords": {},
+        "instructions": {}
+    }
+}

DroidBot/guiagent_core/skills/__init__.py

@@ -0,0 +1,15 @@
+"""
+ Skills Package
+
+专为  平台设计的验证码处理技能。
+"""
+
+from .slider_captcha import SliderCaptchaSkill, solve_slider_captcha
+from .image_captcha import ImageCaptchaSkill, solve_image_captcha
+
+__all__ = [
+    'SliderCaptchaSkill',
+    'solve_slider_captcha',
+    'ImageCaptchaSkill',
+    'solve_image_captcha'
+]

DroidBot/guiagent_core/skills/image_captcha/__init__.py

@@ -0,0 +1,9 @@
+"""
+Image Captcha Skill
+
+图片验证码技能模块。
+"""
+
+from .image_skill import ImageCaptchaSkill, solve_image_captcha
+
+__all__ = ['ImageCaptchaSkill', 'solve_image_captcha']

DroidBot/guiagent_core/skills/image_captcha/image_skill.py

@@ -0,0 +1,126 @@
+"""
+图片验证码识别技能模块
+
+当 Agent 检测到图片验证码时，可以调用此技能自动完成验证。
+需要配置 Anti-Captcha API KEY。
+"""
+
+import os
+import time
+import logging
+import numpy as np
+from typing import Tuple, Any, Optional
+
+import cv2
+from PIL import Image
+from anticaptchaofficial.imagecaptcha import imagecaptcha
+
+logger = logging.getLogger(__name__)
+
+# === 配置 ===
+API_KEY = "cc4ce25dda8c040d6f38f9b00a5a44c9"
+
+class Config:
+    """配置参数"""
+    DEBUG = True
+    DEBUG_OUTPUT_DIR = "./tmp"
+
+
+class ImageCaptchaSkill:
+    """
+     图片验证码识别技能
+    
+    使用 Anti-Captcha 服务识别图片验证码。
+    """
+    
+    def __init__(self, debug: bool = False):
+        self.debug = debug
+
+    def execute(
+        self, 
+        device: Any,
+        captcha_region: list
+    ) -> Tuple[bool, str]:
+        """
+        执行图片验证码识别
+        
+        Args:
+            device: Device 实例
+            captcha_region: [x1, y1, x2, y2] 验证码区域坐标（绝对像素坐标）
+            
+        Returns:
+            (success, result_text) - 如果成功，result_text 是识别的验证码文本
+        """
+        try:
+            # 1. 验证参数
+            if not captcha_region or len(captcha_region) != 4:
+                return False, "Error: captcha_region 需要 4 个坐标值 [x1, y1, x2, y2]"
+                
+            x1, y1, x2, y2 = [int(v) for v in captcha_region]
+            
+            if x2 <= x1 or y2 <= y1:
+                return False, f"Error: Invalid region coords: ({x1},{y1})-({x2},{y2})"
+
+            logger.info(f"图片验证码技能启动: 区域=({x1},{y1})-({x2},{y2})")
+            print(f"[ImageCaptcha] 验证码区域: ({x1},{y1})-({x2},{y2})")
+
+            # 2. 使用 device 截图
+            screenshot_path = device.take_screenshot()
+            if not screenshot_path:
+                return False, "Error: 截图失败"
+            
+            # 加载截图并裁剪验证码区域
+            screenshot = Image.open(screenshot_path)
+            captcha_img = screenshot.crop((x1, y1, x2, y2))
+            
+            # 保存临时文件用于上传
+            os.makedirs(Config.DEBUG_OUTPUT_DIR, exist_ok=True)
+            temp_path = os.path.join(Config.DEBUG_OUTPUT_DIR, "_temp_captcha.png")
+            captcha_img.save(temp_path)
+            
+            if self.debug:
+                print(f"[ImageCaptcha] 验证码截图已保存: {temp_path}")
+
+            # 3. 调用 Anti-Captcha 识别
+            print("正在上传图片到 Anti-Captcha...")
+            solver = imagecaptcha()
+            solver.set_verbose(1 if self.debug else 0)
+            solver.set_key(API_KEY)
+            
+            captcha_text = solver.solve_and_return_solution(temp_path)
+            
+            if captcha_text != 0:
+                logger.info(f"识别成功: {captcha_text}")
+                print(f"[ImageCaptcha] 识别成功! 结果: {captcha_text}")
+                return True, str(captcha_text)
+            else:
+                error_msg = f"识别失败: {solver.error_code}"
+                logger.error(error_msg)
+                print(f"[ImageCaptcha] {error_msg}")
+                return False, error_msg
+
+        except Exception as e:
+            logger.error(f"图片验证码操作异常: {e}")
+            import traceback
+            traceback.print_exc()
+            return False, f"Error: {e}"
+
+
+def solve_image_captcha(
+    device: Any,
+    captcha_region: list,
+    debug: bool = False
+) -> Tuple[bool, str]:
+    """
+    便捷函数：执行图片验证码识别
+    
+    Args:
+        device: Device 实例
+        captcha_region: [x1, y1, x2, y2] 验证码区域坐标
+        debug: 是否保存调试图片
+        
+    Returns:
+        (success, captcha_text) - 如果成功，captcha_text 是识别的文本
+    """
+    skill = ImageCaptchaSkill(debug=debug)
+    return skill.execute(device, captcha_region)

DroidBot/guiagent_core/skills/slider_captcha/__init__.py

@@ -0,0 +1,9 @@
+"""
+Slider Captcha Skill
+
+滑块验证码技能模块。
+"""
+
+from .slider_skill import SliderCaptchaSkill, solve_slider_captcha
+
+__all__ = ['SliderCaptchaSkill', 'solve_slider_captcha']

DroidBot/guiagent_core/skills/slider_captcha/slider_detector.py

@@ -0,0 +1,864 @@
+import base64
+import os
+import random
+import time
+from pathlib import Path
+from typing import List, Tuple, Union
+
+import cv2
+import numpy as np
+import onnxruntime as ort
+from shapely.geometry import Polygon
+
+CONF_THRESHOLD = 0.5
+
+IOU_THRESHOLD = 0.8
+
+Y_IOU_THRESHOLD = 0.85
+
+
+class Slider:
+
+    def __init__(self):
+        """
+        Initialize the instance segmentation model using an ONNX model.
+        """
+        root_dir = os.path.dirname(__file__)
+        slider_model_path = os.path.join(root_dir, 'models', 'slider.onnx')
+
+        self.session = ort.InferenceSession(
+            slider_model_path,
+            providers=["CUDAExecutionProvider", "CPUExecutionProvider"] if ort.get_device() == 'GPU' else [
+                "CPUExecutionProvider"],
+        )
+
+        self.classes = {0: 's'}
+
+    def predict(self, img: np.ndarray, conf: float = 0.25, iou: float = 0.7,
+                imgsz: Union[int, Tuple[int, int]] = 640) -> List:
+        """
+        Run inference on the input image using the ONNX model.
+        """
+        imgsz = (imgsz, imgsz) if isinstance(imgsz, int) else imgsz
+        prep_img = self.preprocess(img, imgsz)
+        outs = self.session.run(None, {self.session.get_inputs()[0].name: prep_img})
+        return self.postprocess(img, prep_img, outs, conf=conf, iou=iou)
+
+    @staticmethod
+    def letterbox(img: np.ndarray, new_shape: Tuple[int, int] = (640, 640)) -> np.ndarray:
+        """
+        Resize and pad image while maintaining aspect ratio.
+        Returns exactly new_shape sized image.
+        """
+        shape = img.shape[:2]  # current shape [height, width]
+
+        # Calculate ratio and new dimensions
+        r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])
+        new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))
+
+        # Ensure new dimensions are at least 1 and not larger than target
+        new_unpad = (max(1, min(new_unpad[0], new_shape[1])),
+                     max(1, min(new_unpad[1], new_shape[0])))
+
+        # Resize if needed
+        if shape[::-1] != new_unpad:
+            img = cv2.resize(img, new_unpad, interpolation=cv2.INTER_LINEAR)
+
+        # Calculate padding
+        dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1]
+        dw, dh = float(dw), float(dh)
+
+        # Divide padding into 2 sides
+        top, bottom = int(round(dh / 2)), int(round(dh / 2))
+        left, right = int(round(dw / 2)), int(round(dw / 2))
+
+        # Add padding
+        img = cv2.copyMakeBorder(img, top, bottom, left, right, cv2.BORDER_CONSTANT, value=(114, 114, 114))
+
+        # Final check to ensure exact size (might need crop if rounding caused overflow)
+        if img.shape[0] != new_shape[0] or img.shape[1] != new_shape[1]:
+            img = cv2.resize(img, new_shape, interpolation=cv2.INTER_LINEAR)
+
+        return img
+
+    def preprocess(self, img: np.ndarray, new_shape: Tuple[int, int]) -> np.ndarray:
+        """
+        Preprocess the input image before feeding it into the model.
+        """
+        img = self.letterbox(img, new_shape)
+        img = img[..., ::-1].transpose([2, 0, 1])[None]
+        img = np.ascontiguousarray(img)
+        img = img.astype(np.float32) / 255
+        return img
+
+    def postprocess(self, img: np.ndarray, prep_img: np.ndarray, outs: List, conf: float = 0.25,
+                    iou: float = 0.7) -> List:
+        """
+        Post-process model predictions to extract meaningful results.
+        """
+        preds, protos = outs
+        preds = self.non_max_suppression(preds, conf, iou, nc=len(self.classes))
+
+        results = []
+        for i, pred in enumerate(preds):
+            pred[:, :4] = self.scale_boxes(prep_img.shape[2:], pred[:, :4], img.shape)
+            masks = self.process_mask(protos[i], pred[:, 6:], pred[:, :4], img.shape[:2])
+            results.append([pred[:, :6], masks])
+
+        return results
+
+    def process_mask(self, protos: np.ndarray, masks_in: np.ndarray, bboxes: np.ndarray,
+                     shape: Tuple[int, int]) -> np.ndarray:
+        c, mh, mw = protos.shape
+        masks = (masks_in @ protos.reshape(c, -1)).reshape(-1, mh, mw)
+        masks = self.scale_masks(masks, shape)
+        masks = self.crop_mask(masks, bboxes)
+        return masks > 0.0
+
+    @staticmethod
+    def masks_to_segments(masks: Union[np.ndarray,], strategy: str = "largest") -> List[np.ndarray]:
+        """
+        将二值Mask转换为多边形边界点(segments)，不使用多边形简化
+
+        参数:
+            masks: 输入的二值Mask，可以是numpy数组或torch张量
+                  形状为(batch_size, height, width)或(height, width)
+            strategy: 处理多个轮廓的策略:
+                     'all' - 合并所有轮廓
+                     'largest' - 只保留最大轮廓
+                     'none' - 返回所有轮廓不合并
+
+        返回:
+            包含多边形点集的列表，每个元素是(N,2)的numpy数组
+        """
+        # 转换输入为numpy数组
+
+        masks_np = masks.astype("uint8")
+
+        # 处理单张mask的情况
+        if masks_np.ndim == 2:
+            masks_np = masks_np[np.newaxis, ...]
+
+        segments = []
+
+        for mask in masks_np:
+            # 查找轮廓 (OpenCV 4.x返回格式)
+            contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+
+            if not contours:  # 没有找到轮廓
+                segments.append(np.zeros((0, 2), dtype=np.float32))
+                continue
+
+            # 根据策略处理多个轮廓
+            if strategy == "all" and len(contours) > 1:
+                # 合并所有轮廓，保留所有点
+                contour = np.concatenate([x.reshape(-1, 2) for x in contours])
+            elif strategy == "largest":
+                # 选择最长的轮廓，保留所有点
+                contour = max(contours, key=lambda x: cv2.arcLength(x, closed=True))
+                contour = contour.reshape(-1, 2)
+            else:  # 'none'策略或其他情况
+                # 不合并轮廓，保留所有点
+                contour = contours[0].reshape(-1, 2)
+
+            segments.append(contour.astype(np.float32))
+
+        return segments[0] if masks_np.shape[0] == 1 else segments
+
+    @staticmethod
+    def draw_segments(image, boxes, masks,
+                      mask_alpha=0.5, box_thickness=2, draw_labels=True):
+
+        """
+        在图像上绘制预测框和掩膜
+
+        参数:
+            image: 原始图像 (numpy数组, BGR格式)
+            boxes: 预测框列表, 格式为 [[x1, y1, x2, y2, score, class_id], ...]
+            masks: 掩膜列表, 每个掩膜为二值图像 (0或255)
+            box_color: 框的颜色 (BGR格式), 如果为None则随机生成
+            mask_alpha: 掩膜透明度 (0-1)
+            box_thickness: 框的线宽
+            draw_labels: 是否绘制类别和置信度标签
+
+        返回:
+            绘制后的图像
+        """
+        # 创建输出图像的副本
+        output = image.copy()
+
+        # 如果没有提供boxes和masks，直接返回原图
+        if boxes is None and masks is None:
+            return output
+
+        # 绘制masks
+        if masks is not None:
+            # 创建一个空的彩色掩膜图像
+            color_mask = np.zeros_like(image)
+
+            for i, mask in enumerate(masks):
+                # 为每个mask生成随机颜色或使用指定颜色
+
+                color = (random.randint(0, 255), random.randint(0, 255), random.randint(0, 255))
+                # 将二值mask转换为彩色mask
+                mask = mask.astype(bool)
+                color_mask[mask] = color
+
+            # 将彩色掩膜与原始图像混合
+            output = cv2.addWeighted(output, 1, color_mask, mask_alpha, 0)
+
+        # 绘制boxes
+        if boxes is not None:
+            for box in boxes:
+                x1, y1, x2, y2, score, class_id = box[:6]  # 只取前6个值，兼容不同格式
+
+                # 为每个box生成随机颜色或使用指定颜色
+
+                color = (random.randint(0, 255), random.randint(0, 255), random.randint(0, 255))
+
+                # 绘制矩形框
+                cv2.rectangle(output, (int(x1), int(y1)), (int(x2), int(y2)), color, box_thickness)
+
+                # 绘制标签
+                if draw_labels:
+                    label = f"{int(class_id)}: {score:.2f}"
+                    (label_width, label_height), _ = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.5, 1)
+
+                    # 绘制标签背景
+                    cv2.rectangle(output, (int(x1), int(y1) - label_height - 5),
+                                  (int(x1) + label_width, int(y1)), color, -1)
+                    # 绘制标签文本
+                    cv2.putText(output, label, (int(x1), int(y1) - 5),
+                                cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 1, cv2.LINE_AA)
+
+        return output
+
+    @staticmethod
+    def image_to_array(source: Union[str, Path, bytes, np.ndarray] = None):
+        if isinstance(source, str) and source.startswith('data:image'):
+            # 从Base64字符串读取
+            header, encoded = source.split(',', 1)
+            data = base64.b64decode(encoded)
+            np_arr = np.frombuffer(data, np.uint8)
+            return cv2.imdecode(np_arr, cv2.IMREAD_COLOR)
+        elif isinstance(source, (str, Path)):
+            # 从文件路径读取
+            return cv2.imread(str(source))
+        elif isinstance(source, bytes):
+            # 从字节流读取
+            np_arr = np.frombuffer(source, np.uint8)
+            return cv2.imdecode(np_arr, cv2.IMREAD_COLOR)
+        elif isinstance(source, np.ndarray):
+            # 如果已经是 numpy 数组，直接使用
+            return source
+        else:
+            raise TypeError("Unsupported source type. Only str, Path, bytes, or numpy.ndarray are supported.")
+
+    @staticmethod
+    def normalize_points(points):
+        """
+        将点集归一化到以原点为中心
+        :param points: 点集
+        :return: 归一化后的点集
+        """
+        # 计算质心
+        centroid = np.mean(points, axis=0)
+        # 将质心移到原点
+        normalized_points = points - centroid
+        return normalized_points
+
+    @staticmethod
+    def y_iou(segment1, segment2):
+        # 计算交集
+        start = max(segment1[0], segment2[0])
+        end = min(segment1[1], segment2[1])
+        intersection = max(0, end - start)  # 确保没有负值（无重叠时返回0）
+
+        # 计算并集
+        len1 = segment1[1] - segment1[0]
+        len2 = segment2[1] - segment2[0]
+        union = len1 + len2 - intersection
+
+        # 计算 IoU
+        iou = intersection / union if union != 0 else 0  # 避免除以0
+        return iou
+
+    def polygon_iou(self, poly1, poly2):
+        """
+        计算两个多边形的 IoU
+        :param poly1: 多边形1的顶点坐标，格式为 [[x1,y1], [x2,y2], ..., [xn,yn]]
+        :param poly2: 多边形2的顶点坐标，格式同上
+        :return: IoU 值（范围 [0, 1]）
+        """
+        # 归一化处理到原点
+        p1 = self.normalize_points(poly1)
+        p2 = self.normalize_points(poly2)
+
+        poly1 = Polygon(p1).buffer(0)  # buffer(0) 修复无效多边形（如自相交）
+        poly2 = Polygon(p2).buffer(0)
+        # poly2 = Polygon(normalize_points(poly2))
+
+        # if not poly1.is_valid or not poly2.is_valid:
+        #     return 0.0  # 无效多边形（如面积为零）
+
+        # 计算交集和并集面积
+        intersect = poly1.intersection(poly2).area
+        union = poly1.union(poly2).area
+
+        # 计算 IoU
+        iou = intersect / union if union > 0 else 0.0
+        return iou
+
+    def pick_out_mask(self, boxes: list, segments):
+        # boxes, masks 为两个列表，找出box值最小的一个
+        box_slider = min(boxes, key=lambda x: x[0])
+        box_slider_index = boxes.index(box_slider)
+        segment_slider = segments[box_slider_index]
+
+        box_sample = boxes[:box_slider_index] + boxes[box_slider_index + 1:]
+        segment_sample = segments[:box_slider_index] + segments[box_slider_index + 1:]
+
+        # 先按照y值iou过滤
+        box_filtered = []
+        segment_filtered = []
+
+        for index, box in enumerate(box_sample):
+            if self.y_iou([box_slider[1], box_slider[3]], [box[1], box[3]]) > Y_IOU_THRESHOLD:
+                box_filtered.append(box)
+                segment_filtered.append(segment_sample[index])
+        # 如果通过y轴iou没有过滤掉有效值，则从所有box中选择iou最大的一个
+        if not box_filtered:
+            box_filtered = box_sample
+            segment_filtered = segment_sample
+
+        if len(box_filtered) == 1:
+            return box_filtered[0], segment_filtered[0]
+
+        iou_flag = 0
+        iou_index = 0
+        for index, segment in enumerate(segment_filtered):
+            segment_iou = self.polygon_iou(segment_slider, segment)
+            if segment_iou > iou_flag:
+                iou_flag = segment_iou
+                iou_index = index
+
+        return box_filtered[iou_index], segment_filtered[iou_index]
+
+    def identify(self, source: Union[str, Path, bytes, np.ndarray], conf=CONF_THRESHOLD, iou=IOU_THRESHOLD, show=False):
+        box_list = []
+        mask_ndarray = None
+
+        original_image: np.ndarray = self.image_to_array(source)
+        results = self.predict(original_image, conf=conf, iou=iou, imgsz=640)
+
+        if results:
+            boxes, masks = results[0]
+            if len(boxes) == 0:
+                pass
+            elif len(boxes) == 1:
+                box_list = boxes[0].tolist()
+                mask_ndarray = masks[0]
+
+            else:
+                segments = self.masks_to_segments(masks)
+                box_list, _ = self.pick_out_mask(boxes.tolist(), segments)
+                mask_ndarray = masks[boxes.tolist().index(box_list)]
+
+        # 仅展示目标缺口
+        if show and box_list and mask_ndarray is not None:
+            sample = self.draw_segments(original_image, [box_list, ], [mask_ndarray, ])
+            cv2.imshow('result', sample)
+            cv2.waitKey(0)
+            cv2.destroyAllWindows()
+
+        if box_list:
+            box = box_list[:4]
+            box_conf = float(box_list[4])
+        else:
+            box = []
+            box_conf = 0.0
+        return box, box_conf
+
+    def identify_offset(self, source: Union[str, Path, bytes, np.ndarray], conf=CONF_THRESHOLD, iou=IOU_THRESHOLD,
+                        show=False):
+        """
+        通过滑块图或者全图获取offset
+        """
+        box_list = []
+        mask_ndarray = None
+
+        original_image: np.ndarray = self.image_to_array(source)
+        results = self.predict(original_image, conf=conf, iou=iou, imgsz=640)
+
+        if results:
+            boxes, masks = results[0]
+            if len(boxes) == 0:
+                pass
+            elif len(boxes) == 1:
+                box_list = boxes[0].tolist()
+                mask_ndarray = masks[0]
+
+            else:
+                # 如果有多个目标，则选择X值最小的目标
+                box_left = min(boxes, key=lambda x: x[0])
+                box_list = box_left.tolist()
+                mask_ndarray = masks[boxes.tolist().index(box_list)]
+
+        # 仅展示目标缺口
+        if show and box_list and mask_ndarray is not None:
+            sample = self.draw_segments(original_image, [box_list, ], [mask_ndarray, ])
+            cv2.imshow('result', sample)
+            cv2.waitKey(0)
+            cv2.destroyAllWindows()
+
+        if box_list:
+            box = box_list[:4]
+            box_conf = float(box_list[4])
+            offset = box[0]
+        else:
+            offset = 0
+            box_conf = 0.0
+
+        return offset, box_conf
+
+    def identify_gap(self, source: Union[str, Path, bytes, np.ndarray], conf=CONF_THRESHOLD, iou=IOU_THRESHOLD,
+                     show=False):
+        """
+        Identify the gap and calculate the distance from the slider.
+        Returns: distance, confidence, slider_box, gap_box
+        """
+        original_image: np.ndarray = self.image_to_array(source)
+        results = self.predict(original_image, conf=conf, iou=iou, imgsz=640)
+
+        distance = 0
+        box_conf = 0.0
+        slider_box = []
+        gap_box = []
+
+        if results:
+            boxes, masks = results[0]
+            if len(boxes) >= 2:
+                # Sort boxes by X coordinate
+                sorted_boxes = sorted(boxes.tolist(), key=lambda x: x[0])
+                
+                # Heuristic: Slider is usually the leftmost object
+                slider_box = sorted_boxes[0]
+                
+                # Heuristic: Gap is the second leftmost (or we could use Y-IoU matching if needed)
+                # For now, taking the next object as the gap seems reliable based on tests.
+                gap_box = sorted_boxes[1]
+                
+                slider_x = slider_box[0]
+                gap_x = gap_box[0]
+                
+                distance = gap_x - slider_x
+                box_conf = (float(slider_box[4]) + float(gap_box[4])) / 2
+            elif len(boxes) == 1:
+                # Fallback: if only one box, assume it's the gap (no slider visible?) or slider?
+                # This matches original identify_offset behavior for single box
+                gap_box = boxes[0].tolist()
+                distance = gap_box[0]
+                box_conf = float(gap_box[4])
+
+        if show and slider_box and gap_box:
+             # Draw results
+             sample = original_image.copy()
+             sx1, sy1, sx2, sy2 = map(int, slider_box[:4])
+             gx1, gy1, gx2, gy2 = map(int, gap_box[:4])
+             
+             cv2.rectangle(sample, (sx1, sy1), (sx2, sy2), (255, 0, 0), 2)
+             cv2.rectangle(sample, (gx1, gy1), (gx2, gy2), (0, 255, 0), 2)
+             cv2.line(sample, (sx1, sy1), (gx1, sy1), (0, 0, 255), 2)
+             
+             cv2.imshow('result', sample)
+             cv2.waitKey(0)
+             cv2.destroyAllWindows()
+
+        return distance, box_conf, slider_box, gap_box
+
+    def scale_boxes(self, img1_shape: Tuple[int, int], boxes: np.ndarray, img0_shape: Tuple[int, int],
+                    ratio_pad: Union[Tuple, None] = None, padding: bool = True, xywh: bool = False):
+        """
+        Rescale bounding boxes from one image shape to another.
+
+        Rescales bounding boxes from img1_shape to img0_shape, accounting for padding and aspect ratio changes.
+        Supports both xyxy and xywh box formats.
+
+        Args:
+            img1_shape (tuple): Shape of the source image (height, width).
+            boxes (np.ndarray): Bounding boxes to rescale in format (N, 4).
+            img0_shape (tuple): Shape of the target image (height, width).
+            ratio_pad (tuple, optional): Tuple of (ratio, pad) for scaling. If None, calculated from image shapes.
+            padding (bool): Whether boxes are based on YOLO-style augmented images with padding.
+            xywh (bool): Whether box format is xywh (True) or xyxy (False).
+
+        Returns:
+            (np.ndarray): Rescaled bounding boxes in the same format as input.
+        """
+        if ratio_pad is None:
+            gain = min(img1_shape[0] / img0_shape[0], img1_shape[1] / img0_shape[1])
+            pad = (
+                round((img1_shape[1] - img0_shape[1] * gain) / 2),
+                round((img1_shape[0] - img0_shape[0] * gain) / 2),
+            )
+        else:
+            gain = ratio_pad[0][0]
+            pad = ratio_pad[1]
+
+        if padding:
+            boxes[..., 0] -= pad[0]
+            boxes[..., 1] -= pad[1]
+            if not xywh:
+                boxes[..., 2] -= pad[0]
+                boxes[..., 3] -= pad[1]
+        boxes[..., :4] /= gain
+        return self.clip_boxes(boxes, img0_shape)
+
+    @staticmethod
+    def get_covariance_matrix(boxes: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
+        """
+        Generate covariance matrix from oriented bounding boxes.
+
+        Args:
+            boxes (np.ndarray): A tensor of shape (N, 5) representing rotated bounding boxes, with xywhr format.
+
+        Returns:
+            (np.ndarray): Covariance matrices corresponding to original rotated bounding boxes.
+        """
+        gbbs = np.concatenate((np.power(boxes[:, 2:4], 2) / 12, boxes[:, 4:]), axis=-1)
+        a, b, c = np.split(gbbs, [1, 2], axis=-1)
+        cos = np.cos(c)
+        sin = np.sin(c)
+        cos2 = np.power(cos, 2)
+        sin2 = np.power(sin, 2)
+        return a * cos2 + b * sin2, a * sin2 + b * cos2, (a - b) * cos * sin
+
+    def batch_probiou(self, obb1: np.ndarray, obb2: np.ndarray, eps: float = 1e-7) -> np.ndarray:
+        """
+        Calculate the probabilistic IoU between oriented bounding boxes.
+
+        Args:
+            obb1 (np.ndarray): A tensor of shape (N, 5) representing ground truth obbs, with xywhr format.
+            obb2 (np.ndarray): A tensor of shape (M, 5) representing predicted obbs, with xywhr format.
+            eps (float, optional): A small value to avoid division by zero.
+
+        Returns:
+            (np.ndarray): A tensor of shape (N, M) representing obb similarities.
+        """
+        x1, y1 = np.split(obb1[..., :2], 2, axis=-1)
+        x2, y2 = (np.expand_dims(x.squeeze(-1), 0) for x in np.split(obb2[..., :2], 2, axis=-1))
+        a1, b1, c1 = self.get_covariance_matrix(obb1)
+        a2, b2, c2 = (np.expand_dims(x.squeeze(-1), 0) for x in self.get_covariance_matrix(obb2))
+
+        t1 = (
+                     ((a1 + a2) * np.power(y1 - y2, 2) + (b1 + b2) * np.power(x1 - x2, 2)) / (
+                     (a1 + a2) * (b1 + b2) - np.power(c1 + c2, 2) + eps)
+             ) * 0.25
+        t2 = (((c1 + c2) * (x2 - x1) * (y1 - y2)) / ((a1 + a2) * (b1 + b2) - np.power(c1 + c2, 2) + eps)) * 0.5
+
+        term1_log = (a1 * b1 - np.power(c1, 2)).clip(0)
+        term2_log = (a2 * b2 - np.power(c2, 2)).clip(0)
+
+        denominator = 4 * np.sqrt(term1_log * term2_log) + eps
+        t3_numerator = (a1 + a2) * (b1 + b2) - np.power(c1 + c2, 2)
+        # 确保 log 的输入为正值
+        t3_arg = np.clip(t3_numerator / denominator + eps, eps, None)
+        t3 = np.log(t3_arg) * 0.5
+
+        bd = (t1 + t2 + t3).clip(eps, 100.0)
+        hd = np.sqrt(1.0 - np.exp(-bd) + eps)
+        return 1 - hd
+
+    def nms_rotated(self, boxes: np.ndarray, scores: np.ndarray, threshold: float = 0.45):
+        """
+        Perform NMS on oriented bounding boxes using probiou and fast-nms.
+
+        Args:
+            boxes (np.ndarray): Rotated bounding boxes with shape (N, 5) in xywhr format.
+            scores (np.ndarray): Confidence scores with shape (N,).
+            threshold (float): IoU threshold for NMS.
+
+        Returns:
+            (np.ndarray): Indices of boxes to keep after NMS.
+        """
+        sorted_idx = np.argsort(scores)[::-1]
+        boxes = boxes[sorted_idx]
+        ious = self.batch_probiou(boxes, boxes)
+
+        # 使用更高效的方式创建上三角矩阵
+        n = boxes.shape[0]
+        ious[np.tril_indices(n)] = 0  # 将下三角和对角线置零
+
+        pick = np.where((ious >= threshold).sum(axis=0) <= 0)[0]
+        return sorted_idx[pick]
+
+    def clip_boxes(self, boxes: np.ndarray, shape: Tuple[int, int]):
+        """
+        Clip bounding boxes to image boundaries.
+
+        Args:
+            boxes (np.ndarray): Bounding boxes to clip.
+            shape (tuple): Image shape as (height, width).
+
+        Returns:
+            (np.ndarray): Clipped bounding boxes.
+        """
+        boxes[..., [0, 2]] = np.clip(boxes[..., [0, 2]], 0, shape[1])
+        boxes[..., [1, 3]] = np.clip(boxes[..., [1, 3]], 0, shape[0])
+        return boxes
+
+    @staticmethod
+    def xywh2xyxy(x: np.ndarray):
+        """
+        Convert bounding box coordinates from (x, y, width, height) format to (x1, y1, x2, y2) format.
+
+        Args:
+            x (np.ndarray): Input bounding box coordinates in (x, y, width, height) format.
+
+        Returns:
+            (np.ndarray): Bounding box coordinates in (x1, y1, x2, y2) format.
+        """
+        assert x.shape[-1] == 4, f"input shape last dimension expected 4 but input shape is {x.shape}"
+        y = np.empty_like(x, dtype=np.float32)
+        xy = x[..., :2]
+        wh = x[..., 2:] / 2
+        y[..., :2] = xy - wh
+        y[..., 2:] = xy + wh
+        return y
+
+    @staticmethod
+    def crop_mask(masks: np.ndarray, boxes: np.ndarray):
+        """
+        Crop masks to bounding box regions.
+
+        Args:
+            masks (np.ndarray): Masks with shape (N, H, W).
+            boxes (np.ndarray): Bounding box coordinates with shape (N, 4) in relative point form.
+
+        Returns:
+            (np.ndarray): Cropped masks.
+        """
+        _, h, w = masks.shape
+        # 确保 boxes 的维度正确
+        boxes = boxes[:, :, None] if boxes.ndim == 2 else boxes
+        x1, y1, x2, y2 = np.split(boxes, 4, axis=1)
+        r = np.arange(w, dtype=x1.dtype)[None, None, :]
+        c = np.arange(h, dtype=x1.dtype)[None, :, None]
+
+        return masks * ((r >= x1) * (r < x2) * (c >= y1) * (c < y2))
+
+    def process_mask_np(self, protos: np.ndarray, masks_in: np.ndarray, bboxes: np.ndarray, shape: Tuple[int, int],
+                        upsample: bool = False):
+        """
+        Apply masks to bounding boxes using mask head output.
+
+        Args:
+            protos (np.ndarray): Mask prototypes with shape (mask_dim, mask_h, mask_w).
+            masks_in (np.ndarray): Mask coefficients with shape (N, mask_dim) where N is number of masks after NMS.
+            bboxes (np.ndarray): Bounding boxes with shape (N, 4) where N is number of masks after NMS.
+            shape (tuple): Input image size as (height, width).
+            upsample (bool): Whether to upsample masks to original image size.
+
+        Returns:
+            (np.ndarray): A binary mask array of shape [n, h, w], where n is the number of masks after NMS, and h and w
+                are the height and width of the input image. The mask is applied to the bounding boxes.
+        """
+        c, mh, mw = protos.shape
+        ih, iw = shape
+
+        masks = (masks_in @ protos.reshape(c, -1)).reshape(-1, mh, mw)
+        width_ratio = mw / iw
+        height_ratio = mh / ih
+
+        downsampled_bboxes = bboxes.copy()
+        downsampled_bboxes[:, 0] *= width_ratio
+        downsampled_bboxes[:, 2] *= width_ratio
+        downsampled_bboxes[:, 3] *= height_ratio
+        downsampled_bboxes[:, 1] *= height_ratio
+
+        masks = self.crop_mask(masks, downsampled_bboxes)
+        if upsample:
+            masks = cv2.resize(masks.transpose((1, 2, 0)),
+                               (shape[1], shape[0]),
+                               interpolation=cv2.INTER_LINEAR).transpose((2, 0, 1))
+
+        return masks > 0.0
+
+    @staticmethod
+    def scale_masks(masks: np.ndarray, shape: Tuple[int, int], padding: bool = True):
+        """
+        Rescale segment masks to target shape.
+        Args:
+            masks (np.ndarray): Masks with shape (N, H, W).
+            shape (tuple): Target height and width as (height, width).
+            padding (bool): Whether masks are based on YOLO-style augmented images with padding.
+        Returns:
+            (np.ndarray): Rescaled masks with shape (N, H_new, W_new).
+        """
+        mh, mw = masks.shape[1:]
+        gain = min(mh / shape[0], mw / shape[1])
+        pad = [mw - shape[1] * gain, mh - shape[0] * gain]
+
+        if padding:
+            pad[0] /= 2
+            pad[1] /= 2
+
+        top, left = (int(round(pad[1])), int(round(pad[0]))) if padding else (0, 0)
+        bottom, right = (
+            mh - int(round(pad[1])),
+            mw - int(round(pad[0])),
+        )
+
+        # Crop the masks first
+        masks_cropped = masks[:, top:bottom, left:right]
+
+        # 向量化 resize 操作
+        resized_masks = np.zeros((masks_cropped.shape[0], shape[0], shape[1]), dtype=masks_cropped.dtype)
+        for i, mask in enumerate(masks_cropped):
+            resized_masks[i] = cv2.resize(mask, (shape[1], shape[0]), interpolation=cv2.INTER_LINEAR)
+
+        return resized_masks
+
+    def non_max_suppression(
+            self,
+            prediction: np.ndarray,
+            conf_thres: float = 0.25,
+            iou_thres: float = 0.45,
+            classes=None,
+            agnostic: bool = False,
+            multi_label: bool = False,
+            labels=(),
+            max_det: int = 300,
+            nc: int = 0,
+            max_time_img: float = 0.05,
+            max_nms: int = 30000,
+            max_wh: int = 7680,
+            in_place: bool = True,
+            rotated: bool = False,
+            end2end: bool = False,
+            return_idxs: bool = False,
+    ):
+        """
+        Perform non-maximum suppression (NMS) on prediction results.
+        """
+        assert 0 <= conf_thres <= 1, f"Invalid Confidence threshold {conf_thres}, valid values are between 0.0 and 1.0"
+        assert 0 <= iou_thres <= 1, f"Invalid IoU {iou_thres}, valid values are between 0.0 and 1.0"
+
+        if isinstance(prediction, (list, tuple)):
+            prediction = prediction[0]
+        if classes is not None:
+            classes = np.array(classes)
+
+        if prediction.shape[-1] == 6 or end2end:
+            output = [pred[pred[:, 4] > conf_thres][:max_det] for pred in prediction]
+            if classes is not None:
+                output = [pred[np.any(pred[:, 5:6] == classes, axis=1)] for pred in output]
+            return output
+
+        bs = prediction.shape[0]
+        nc = nc or (prediction.shape[1] - 4)
+        extra = prediction.shape[1] - nc - 4
+        mi = 4 + nc
+        xc = np.amax(prediction[:, 4:mi], axis=1) > conf_thres
+        xinds = np.stack([np.arange(len(i)) for i in xc])[..., None]
+
+        time_limit = 2.0 + max_time_img * bs
+        multi_label &= nc > 1
+
+        prediction = np.transpose(prediction, (0, 2, 1))
+        if not rotated:
+            if in_place:
+                prediction[..., :4] = self.xywh2xyxy(prediction[..., :4])
+            else:
+                prediction = np.concatenate((self.xywh2xyxy(prediction[..., :4]), prediction[..., 4:]), axis=-1)
+
+        t = time.time()
+        output = [np.zeros((0, 6 + extra), dtype=np.float32)] * bs
+        keepi = [np.zeros((0, 1), dtype=np.int64)] * bs
+        for xi, (x, xk) in enumerate(zip(prediction, xinds)):
+            filt = xc[xi]
+            x, xk = x[filt], xk[filt]
+
+            # 增强 labels 的健壮性
+            if labels and len(labels) > xi and len(labels[xi]) and not rotated:
+                lb = np.array(labels[xi])
+                if lb.size > 0:
+                    v = np.zeros((len(lb), nc + extra + 4), dtype=np.float32)
+                    v[:, :4] = self.xywh2xyxy(lb[:, 1:5])
+                    v[range(len(lb)), lb[:, 0].astype(np.int64) + 4] = 1.0
+                    x = np.concatenate((x, v), axis=0)
+
+            if not x.shape[0]:
+                continue
+
+            box, cls, mask = np.split(x, [4, 4 + nc], axis=1)
+
+            if multi_label:
+                i, j = np.where(cls > conf_thres)
+                x = np.concatenate((box[i], x[i, 4 + j, None], j[:, None].astype(np.float32), mask[i]), axis=1)
+                xk = xk[i]
+            else:
+                conf = np.amax(cls, axis=1, keepdims=True)
+                j = np.argmax(cls, axis=1, keepdims=True)
+                filt = conf.squeeze(-1) > conf_thres
+                x = np.concatenate((box, conf, j.astype(np.float32), mask), axis=1)[filt]
+                xk = xk[filt]
+
+            if classes is not None:
+                filt = np.any(x[:, 5:6] == classes, axis=1)
+                x, xk = x[filt], xk[filt]
+
+            n = x.shape[0]
+            if not n:
+                continue
+            if n > max_nms:
+                filt = np.argsort(x[:, 4])[::-1][:max_nms]
+                x, xk = x[filt], xk[filt]
+
+            c = x[:, 5:6] * (0 if agnostic else max_wh)
+            scores = x[:, 4]
+
+            if rotated:
+                boxes = np.concatenate((x[:, :2] + c, x[:, 2:4], x[:, -1:]), axis=-1)
+                i = self.nms_rotated(boxes, scores, iou_thres)
+            else:
+                boxes = x[:, :4] + c
+                # Custom NMS for numpy
+                i = []
+                if boxes.shape[0] > 0:
+                    y1, x1, y2, x2 = boxes[:, 1], boxes[:, 0], boxes[:, 3], boxes[:, 2]
+                    area = (x2 - x1) * (y2 - y1)
+                    order = scores.argsort()[::-1]
+                    while order.size > 0:
+                        idx = order[0]
+                        i.append(idx)
+                        xx1 = np.maximum(x1[idx], x1[order[1:]])
+                        yy1 = np.maximum(y1[idx], y1[order[1:]])
+                        xx2 = np.minimum(x2[idx], x2[order[1:]])
+                        yy2 = np.minimum(y2[idx], y2[order[1:]])
+                        w = np.maximum(0.0, xx2 - xx1)
+                        h = np.maximum(0.0, yy2 - yy1)
+                        inter = w * h
+                        iou = inter / (area[idx] + area[order[1:]] - inter)
+                        order = order[np.where(iou <= iou_thres)[0] + 1]
+                i = np.array(i)
+
+            i = i[:max_det]
+
+            output[xi], keepi[xi] = x[i], xk[i].reshape(-1)
+            if (time.time() - t) > time_limit:
+                break
+
+        return (output, keepi) if return_idxs else output
+
+
+if __name__ == "__main__":
+    """
+    单缺口
+    """
+    model = Slider()
+    # base64 图片测试
+    # base64_image = 'xxx'
+    # res = model.identify(source=base64_image, show=True)
+    res = model.identify(source='img_example.png', show=True)
+    print('results', res)

DroidBot/guiagent_core/skills/slider_captcha/slider_skill.py

@@ -0,0 +1,261 @@
+"""
+滑块验证码技能模块
+
+当 Agent 检测到滑动验证码时，可以调用此技能自动完成验证。
+
+使用方式:
+    1. Agent 识别到滑动验证码后，调用 solve_slider_captcha 动作
+    2. 传入验证码区域坐标和滑块位置
+    3. 技能自动截图、识别、拖拽
+"""
+
+import os
+import cv2
+import time
+import logging
+import numpy as np
+from typing import Dict, Any, Tuple, Optional, Union
+from PIL import Image
+
+from .slider_detector import Slider
+
+logger = logging.getLogger(__name__)
+
+
+# ============== 配置类 ==============
+class Config:
+    """滑块识别配置参数"""
+    
+    # 是否开启调试模式（保存中间处理图片）
+    DEBUG = True    
+    
+    # 调试图片保存目录
+    DEBUG_OUTPUT_DIR = "./tmp"
+
+
+# ============== 识别算法 ==============
+def detect_gap_position(img_input: Union[str, np.ndarray]) -> dict:
+    """
+    检测滑块验证码中滑块和缺口的位置
+    
+    使用 ONNX 模型进行检测
+    
+    Args:
+        img_input: 图片路径(str)或numpy数组
+        
+    Returns:
+        包含滑块和缺口位置信息的字典
+    """
+    # 实例化模型
+    slider_model = Slider()
+    
+    # 识别缺口
+    # identify_gap 返回: distance, box_conf, slider_box, gap_box
+    distance, conf, slider_box, gap_box = slider_model.identify_gap(img_input, show=False)
+    
+    # 构造返回结果
+    result = {
+        'distance': int(distance) if distance else 0,
+        'confidence': conf,
+        'slider': None,
+        'gap': None,
+        'slider_left_x': 0,
+        'gap_left_x': 0
+    }
+    
+    if slider_box and len(slider_box) >= 4:
+        x1, y1, x2, y2 = map(int, slider_box[:4])
+        result['slider'] = {
+            'x': x1, 
+            'y': y1, 
+            'w': x2 - x1, 
+            'h': y2 - y1,
+            'conf': slider_box[4] if len(slider_box) > 4 else 0
+        }
+        result['slider_left_x'] = x1
+        
+    if gap_box and len(gap_box) >= 4:
+        x1, y1, x2, y2 = map(int, gap_box[:4])
+        result['gap'] = {
+            'x': x1, 
+            'y': y1, 
+            'w': x2 - x1, 
+            'h': y2 - y1,
+            'conf': gap_box[4] if len(gap_box) > 4 else 0
+        }
+        result['gap_left_x'] = x1
+    
+    # 保存调试信息
+    if Config.DEBUG:
+        try:
+            os.makedirs(Config.DEBUG_OUTPUT_DIR, exist_ok=True)
+            
+            if isinstance(img_input, str):
+                debug_img = cv2.imread(img_input)
+            elif isinstance(img_input, np.ndarray):
+                debug_img = img_input.copy()
+            else:
+                debug_img = None
+                
+            if debug_img is not None:
+                if result['slider']:
+                    s = result['slider']
+                    cv2.rectangle(debug_img, (s['x'], s['y']), (s['x']+s['w'], s['y']+s['h']), (255, 0, 0), 2)
+                    cv2.putText(debug_img, "Slider", (s['x'], s['y']-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 0, 0), 2)
+                    
+                if result['gap']:
+                    g = result['gap']
+                    cv2.rectangle(debug_img, (g['x'], g['y']), (g['x']+g['w'], g['y']+g['h']), (0, 255, 0), 2)
+                    cv2.putText(debug_img, f"Gap ({distance})", (g['x'], g['y']-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
+                
+                if result['slider'] and result['gap']:
+                    sx = result['slider']['x']
+                    gx = result['gap']['x']
+                    cy = result['slider']['y'] + result['slider']['h'] // 2
+                    cv2.line(debug_img, (sx, cy), (gx, cy), (0, 0, 255), 2)
+                
+                output_path = os.path.join(Config.DEBUG_OUTPUT_DIR, "debug__slider.png")
+                cv2.imwrite(output_path, debug_img)
+                print(f"[SliderCaptcha] Debug image saved to {output_path}")
+                
+        except Exception as e:
+            logger.error(f"保存调试图片失败: {e}")
+
+    return result
+
+
+# ============== 技能类 ==============
+class SliderCaptchaSkill:
+    """
+     滑块验证码技能
+    
+    当 Agent 在界面上识别到滑动验证码时，可以调用此技能自动完成验证。
+    使用 ADB 进行截图和拖拽操作。
+    """
+    
+    def __init__(self, debug: bool = False):
+        """
+        Args:
+            debug: 是否保存调试图片
+        """
+        self.debug = debug
+    
+    def execute(
+        self, 
+        device: Any,
+        captcha_region: list, 
+        slider_position: list
+    ) -> Tuple[bool, str]:
+        """
+        执行滑块验证码破解
+        
+        Args:
+            device: Device 实例
+            captcha_region: [x1, y1, x2, y2] 验证码区域的左上角和右下角坐标（绝对像素坐标）
+            slider_position: [x, y] 滑块按钮的中心坐标（绝对像素坐标）
+        
+        Returns:
+            Tuple[bool, str]: (是否成功, 反馈消息)
+        """
+        try:
+            # 1. 验证参数
+            if not captcha_region or len(captcha_region) != 4:
+                return False, "Error: captcha_region 需要 4 个坐标值 [x1, y1, x2, y2]"
+            
+            if not slider_position or len(slider_position) != 2:
+                return False, "Error: slider_position 需要 2 个坐标值 [x, y]"
+            
+            x1, y1, x2, y2 = [int(v) for v in captcha_region]
+            slider_x, slider_y = [int(v) for v in slider_position]
+            
+            if x2 <= x1 or y2 <= y1:
+                return False, f"Error: Invalid region coords: ({x1},{y1})-({x2},{y2}), width/height must be positive"
+            
+            if (x2 - x1) < 10 or (y2 - y1) < 10:
+                return False, f"Error: Region too small: {x2-x1}x{y2-y1}"
+
+            logger.info(f"滑块验证码技能启动: 验证码区域=({x1},{y1})-({x2},{y2}), 滑块位置=({slider_x},{slider_y})")
+            print(f"[SliderCaptcha] 验证码区域: ({x1},{y1})-({x2},{y2})")
+            print(f"[SliderCaptcha] 滑块位置: ({slider_x},{slider_y})")
+            
+            # 2. 使用 device 截图
+            screenshot_path = device.take_screenshot()
+            if not screenshot_path:
+                return False, "Error: 截图失败"
+            
+            # 加载截图并裁剪验证码区域
+            screenshot = Image.open(screenshot_path)
+            captcha_img = screenshot.crop((x1, y1, x2, y2))
+            
+            # 转换为 OpenCV 格式 (PIL -> numpy BGR)
+            captcha_cv = cv2.cvtColor(np.array(captcha_img), cv2.COLOR_RGB2BGR)
+            
+            if self.debug:
+                os.makedirs(Config.DEBUG_OUTPUT_DIR, exist_ok=True)
+                cv2.imwrite(os.path.join(Config.DEBUG_OUTPUT_DIR, "_captcha_crop.png"), captcha_cv)
+            
+            # 3. 识别滑动距离
+            Config.DEBUG = self.debug
+            result = detect_gap_position(captcha_cv)
+            distance = result['distance']
+            
+            if distance <= 0:
+                return False, f"Error: 无法识别滑动距离 (distance={distance})"
+            
+            logger.info(f"识别到滑动距离: {distance} 像素")
+            print(f"[SliderCaptcha] 识别滑动距离: {distance} 像素")
+            
+            # 4. 计算拖拽终点
+            end_x = slider_x + distance
+            end_y = slider_y  # Y 保持不变
+            
+            logger.info(f"执行拖拽: ({slider_x},{slider_y}) -> ({end_x},{end_y})")
+            print(f"[SliderCaptcha] 拖拽: ({slider_x},{slider_y}) -> ({end_x},{end_y})")
+            
+            # 5. 使用 ADB 执行拖拽操作
+            # 模拟人类拖拽：分段拖拽，使用较长的持续时间
+            # view_drag(start_xy, end_xy, duration_ms)
+            
+            # 拖拽时间（毫秒），模拟人类操作
+            drag_duration = 800  # 800ms
+            
+            intermediate_x = end_x
+            device.view_drag(
+                start_xy=(slider_x, slider_y),
+                end_xy=(intermediate_x, end_y),
+                duration=drag_duration
+            )
+            
+            logger.info("滑块验证码操作完成")
+            print("[SliderCaptcha] 操作完成")
+            
+            return True, f"Success: 滑块验证码操作完成，滑动距离 {distance} 像素"
+            
+        except Exception as e:
+            logger.error(f"滑块验证码操作失败: {e}")
+            import traceback
+            traceback.print_exc()
+            return False, f"Error: 滑块验证码操作失败 - {e}"
+
+
+# ============== 简易接口 ==============
+def solve_slider_captcha(
+    device: Any,
+    captcha_region: list,
+    slider_position: list,
+    debug: bool = False
+) -> Tuple[bool, str]:
+    """
+    便捷函数：执行滑块验证码破解
+    
+    Args:
+        device: Device 实例
+        captcha_region: [x1, y1, x2, y2] 验证码区域坐标
+        slider_position: [x, y] 滑块按钮中心坐标
+        debug: 是否保存调试图片
+        
+    Returns:
+        (success, message)
+    """
+    skill = SliderCaptchaSkill(debug=debug)
+    return skill.execute(device, captcha_region, slider_position)

DroidBot/guiagent_core/utils.py

@@ -0,0 +1,880 @@
+"""
+Utility Functions - 动作解析、坐标转换、图像处理
+
+Extracted from GuiAgent/core/utils/common.py for modular use in DroidBot.
+"""
+import re
+import base64
+import io
+import os
+import logging
+from typing import Optional, Dict, Any, Tuple, List
+
+try:
+    from PIL import Image
+except ImportError:
+    Image = None
+
+logger = logging.getLogger(__name__)
+
+# ==============================================================================
+# 动作解析
+# ==============================================================================
+
+# 兼容多种参数写法
+ACTION_PARAM_PATTERN = re.compile(
+    r"(\w+)\s*[:=]\s*(?:'([^']*)'|\"([^\"]*)\"|`([^`]*)`|\[([^\]]*)\]|(\d+,\d+)|([^\s,]+))"
+)
+
+
+def _extract_thought(prediction: str) -> str:
+    """提取 Thought 段落"""
+    match = re.search(r'Thought:\s*(.+?)(?=Action:|$)', prediction, re.DOTALL)
+    return match.group(1).strip() if match else ''
+
+
+def _extract_action_and_params(prediction: str) -> Tuple[str, Dict[str, str]]:
+    """
+    提取 Action 类型和参数字符串，并解析成字典
+    
+    支持格式：
+    1. Action: click(center='[x, y]')
+    2. <|begin_of_box|>tap(center='[511, 426]')<|end_of_box|>
+    3. 长文本后直接跟 Action: tap(center='[596, 359]')
+    
+    Returns:
+        (action_type, inputs)
+    """
+    action_content = prediction
+    
+    # 提取 <|begin_of_box|>...<|end_of_box|> 中的内容
+    box_match = re.search(r'<\|begin_of_box\|>(.*?)<\|end_of_box\|>', prediction, re.DOTALL)
+    if box_match:
+        action_content = box_match.group(1).strip()
+    
+    # 定义有效的动作类型列表
+    valid_actions = ['tap', 'click', 'long_tap', 'drag', 'swipe', 'type', 'key_press', 
+                     'wait', 'finished', 'report_stuck_reason', 'receive_email', 'double_click', 
+                     'right_click', 'scroll', 'solve_slider_captcha', 'solve_image_captcha',
+                     'login_ios']
+    
+    action_type: Optional[str] = None
+    params_str: Optional[str] = None
+
+    # 1) 从 "Action: xxx ..." 行中解析
+    line_match = re.search(r'Action:\s*(\w+)(.*)', action_content)
+    if line_match:
+        matched_action_type = line_match.group(1)
+        tail = line_match.group(2).strip()
+
+        if matched_action_type in valid_actions:
+            action_type = matched_action_type
+
+            # 带括号形式
+            paren_match = re.match(r'^\(\s*(.*)\s*\)$', tail, re.DOTALL)
+            if paren_match:
+                params_str = paren_match.group(1).strip()
+            else:
+                # 无括号形式
+                params_str = tail.strip()
+
+    # 2) 兜底查找 tap(...)/click(...) 等函数调用
+    if action_type is None:
+        action_pattern = r'\b(' + '|'.join(valid_actions) + r')\s*\(\s*([^)]*?)\s*\)'
+        fallback_match = re.search(action_pattern, action_content, re.DOTALL)
+        if fallback_match:
+            action_type = fallback_match.group(1)
+            params_str = fallback_match.group(2).strip()
+
+    if action_type is None:
+        return '', {}
+
+    inputs: Dict[str, str] = {}
+    if params_str:
+        for m in re.finditer(ACTION_PARAM_PATTERN, params_str):
+            key = m.group(1)
+            raw_val = next((g for g in m.groups()[1:] if g is not None), "")
+            # 如果是方括号或逗号分隔的坐标，补回括号
+            if m.group(5):  # 方括号
+                value = f"[{raw_val}]"
+            elif m.group(6):  # 逗号分隔坐标
+                value = f"[{raw_val}]"
+            else:
+                value = raw_val
+            inputs[key] = value.strip()
+
+    return action_type, inputs
+
+
+def parse_uitars_action(prediction: str) -> Dict[str, Any]:
+    """
+    解析 UITars 风格的动作
+    
+    输入格式：
+        Thought: ...
+        Action: click(center='[x, y]')
+    
+    Returns:
+        {'action_type': str, 'inputs': dict, 'thought': str}
+    """
+    action_type, inputs = _extract_action_and_params(prediction)
+    return {
+        'action_type': action_type,
+        'inputs': inputs,
+        'thought': _extract_thought(prediction)
+    }
+
+
+def convert_to_executor_action(parsed: Dict[str, Any]) -> Dict[str, Any]:
+    """
+    将解析后的动作转换为执行器格式
+    
+    保持原始动作名称，不做平台特定的映射。
+    坐标转换在各 executor 内部完成。
+    """
+    action_type = parsed.get('action_type', '')
+    inputs = parsed.get('inputs', {})
+    
+    # 动作映射（主要处理别名）
+    action_map = {
+        # 通用动作
+        'type': 'type',
+        'wait': 'wait',
+        'finished': 'finished',
+        'report_stuck_reason': 'report_stuck_reason',
+        'login_ios': 'login_ios',
+        'receive_email': 'receive_email',
+        'key_press': 'key_press',
+        'hold_key': 'key_press',
+        
+        # 平台特定动作（保持原样）
+        'click': 'click',
+        'double_click': 'double_click',
+        'right_click': 'right_click',
+        'drag': 'drag',
+        'scroll': 'scroll',
+        'tap': 'tap',
+        'long_tap': 'long_tap',
+        'swipe': 'swipe',
+    }
+    
+    result = {
+        'action': action_map.get(action_type, action_type),
+        'thought': parsed.get('thought', '')
+    }
+
+    def _parse_center(box_str: str) -> Tuple[Optional[float], Optional[float]]:
+        """解析坐标中心点，并处理归一化坐标兜底"""
+        if not box_str:
+            return None, None
+        try:
+            nums = [float(n.strip()) for n in box_str.strip('[]').split(',') if n.strip()]
+            if len(nums) == 2:
+                x, y = nums[0], nums[1]
+            elif len(nums) >= 4:
+                x1, y1, x2, y2 = nums[0], nums[1], nums[2], nums[3]
+                x, y = (x1 + x2) / 2, (y1 + y2) / 2
+            else:
+                return None, None
+            
+            # 坐标兜底：如果x和y都小于1，说明LLM返回的是0-1归一化坐标，需要乘以1000
+            if x < 1 and y < 1:
+                logger.debug(f"[坐标兜底] 检测到0-1归一化坐标 ({x:.4f}, {y:.4f})，乘以1000转换")
+                x = x * 1000
+                y = y * 1000
+            
+            return x, y
+        except (ValueError, IndexError):
+            return None, None
+        return None, None
+    
+    # 单点坐标（支持 center 和 point 参数）
+    center_key = 'center' if 'center' in inputs else ('point' if 'point' in inputs else None)
+    
+    if center_key:
+        x, y = _parse_center(inputs[center_key])
+        if x is not None:
+            result['target'] = [x, y]
+    elif 'start_center' in inputs:
+        x, y = _parse_center(inputs['start_center'])
+        if x is not None:
+            result['target'] = [x, y]
+    
+    # 拖拽终点
+    if 'end_center' in inputs:
+        ex, ey = _parse_center(inputs['end_center'])
+        if ex is not None:
+            result['start'] = result.get('target')
+            result['end'] = [ex, ey]
+    
+    # 文本
+    if 'content' in inputs:
+        result['text'] = inputs['content']
+    elif 'value' in inputs:
+        result['text'] = inputs['value']
+    
+    # 按键
+    if 'key' in inputs:
+        result['key'] = inputs['key']
+    
+    # 方向
+    if 'direction' in inputs:
+        result['direction'] = inputs['direction']
+    
+    # 滑动幅度
+    if 'amount' in inputs:
+        try:
+            result['amount'] = float(inputs['amount'])
+        except (ValueError, TypeError):
+            pass
+    
+    # 验证码技能参数
+    if 'captcha_region' in inputs:
+        # 解析 [x1, y1, x2, y2] 格式
+        try:
+            region_str = inputs['captcha_region'].strip('[]')
+            region_nums = [float(n.strip()) for n in region_str.split(',') if n.strip()]
+            if len(region_nums) == 4:
+                result['captcha_region'] = region_nums
+        except (ValueError, IndexError):
+            pass
+    
+    if 'slider_position' in inputs:
+        try:
+            pos_str = inputs['slider_position'].strip('[]')
+            pos_nums = [float(n.strip()) for n in pos_str.split(',') if n.strip()]
+            if len(pos_nums) == 2:
+                result['slider_position'] = pos_nums
+        except (ValueError, IndexError):
+            pass
+    
+    if 'input_field' in inputs:
+        try:
+            field_str = inputs['input_field'].strip('[]')
+            field_nums = [float(n.strip()) for n in field_str.split(',') if n.strip()]
+            if len(field_nums) == 2:
+                result['input_field'] = field_nums
+        except (ValueError, IndexError):
+            pass
+    
+    # report_stuck_reason 参数
+    if 'reason_code' in inputs:
+        result['reason_code'] = inputs['reason_code'].strip()
+    if 'message' in inputs:
+        result['message'] = inputs['message'].strip()
+    
+    return result
+
+
+# ==============================================================================
+# 图像处理
+# ==============================================================================
+
+def draw_grid_on_image(image, grid_color: Tuple[int, int, int, int] = (255, 0, 0, 128), grid_divisions: int = 10,
+                       coord_range: Tuple[int, int] = None):
+    """
+    在图像上绘制网格线并标注坐标数字，帮助LLM精确理解坐标位置
+    
+    Args:
+        image: PIL Image对象
+        grid_color: 网格线颜色 (R, G, B, A)，默认为红色半透明
+        grid_divisions: 网格分割数量，默认为10，即绘制10x10的网格
+        coord_range: 坐标范围 (max_x, max_y)，默认为 (1000, 1000) 即归一化坐标
+                    如果为None则使用图像实际像素尺寸
+    
+    Returns:
+        绘制了网格和坐标标注的图像
+    """
+    if Image is None:
+        logger.warning("PIL未安装，无法绘制网格")
+        return image
+    
+    try:
+        from PIL import ImageDraw, ImageFont
+        
+        # 创建一个可以绘制的副本
+        img_copy = image.copy()
+        
+        # 获取图像尺寸
+        width, height = img_copy.size
+        
+        # 确定坐标范围
+        if coord_range is None:
+            coord_max_x, coord_max_y = width, height
+        else:
+            coord_max_x, coord_max_y = coord_range
+        
+        # 计算网格大小（像素）
+        grid_width = width // grid_divisions
+        grid_height = height // grid_divisions
+        
+        draw = ImageDraw.Draw(img_copy)
+        
+        # 尝试加载字体，动态计算合适的字体大小
+        font_size = max(10, min(width, height) // 40)
+        # 依次尝试 macOS / Windows / Linux 字体路径
+        _font_candidates = [
+            "/System/Library/Fonts/Helvetica.ttc",                          # macOS
+            "C:/Windows/Fonts/arial.ttf",                                   # Windows
+            "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",             # Linux (Debian/Ubuntu)
+        ]
+        font = None
+        for _font_path in _font_candidates:
+            try:
+                font = ImageFont.truetype(_font_path, font_size)
+                break
+            except Exception:
+                continue
+        if font is None:
+            font = ImageFont.load_default()
+        
+        # 标注颜色（与网格线同色但更不透明）
+        label_color = (grid_color[0], grid_color[1], grid_color[2], min(255, grid_color[3] + 80))
+        # 标注背景色（半透明白色，提高可读性）
+        bg_color = (255, 255, 255, 160)
+        
+        # 绘制垂直线 + 顶部X坐标标注
+        for i in range(1, grid_divisions):
+            x = i * grid_width
+            draw.line([(x, 0), (x, height)], fill=grid_color, width=1)
+            
+            # 计算对应的坐标值
+            coord_x = int(i * coord_max_x / grid_divisions)
+            label = str(coord_x)
+            
+            # 获取文本尺寸
+            bbox = draw.textbbox((0, 0), label, font=font)
+            text_w = bbox[2] - bbox[0]
+            text_h = bbox[3] - bbox[1]
+            
+            # 在网格线顶部标注（带背景）
+            label_x = x - text_w // 2
+            label_y = 2
+            draw.rectangle([label_x - 1, label_y, label_x + text_w + 1, label_y + text_h + 1], fill=bg_color)
+            draw.text((label_x, label_y), label, fill=label_color, font=font)
+        
+        # 绘制水平线 + 左侧Y坐标标注
+        for i in range(1, grid_divisions):
+            y = i * grid_height
+            draw.line([(0, y), (width, y)], fill=grid_color, width=1)
+            
+            # 计算对应的坐标值
+            coord_y = int(i * coord_max_y / grid_divisions)
+            label = str(coord_y)
+            
+            # 获取文本尺寸
+            bbox = draw.textbbox((0, 0), label, font=font)
+            text_w = bbox[2] - bbox[0]
+            text_h = bbox[3] - bbox[1]
+            
+            # 在网格线左侧标注（带背景）
+            label_x = 2
+            label_y = y - text_h // 2
+            draw.rectangle([label_x - 1, label_y, label_x + text_w + 1, label_y + text_h + 1], fill=bg_color)
+            draw.text((label_x, label_y), label, fill=label_color, font=font)
+        
+        logger.debug(f"在图像上绘制了 {grid_divisions}x{grid_divisions} 网格（坐标范围: {coord_max_x}x{coord_max_y}）")
+        return img_copy
+    except Exception as e:
+        logger.error(f"绘制网格失败: {e}")
+        return image
+
+
+def draw_last_action_marker(image, x: int, y: int, 
+                            color: Tuple[int, int, int, int] = (0, 255, 0, 200),
+                            radius: int = 5, width: int = 3):
+    """
+    在截图上绘制上次动作位置的绿色圆圈标记
+    
+    Args:
+        image: PIL Image对象
+        x: 标记的x坐标（绝对像素坐标）
+        y: 标记的y坐标（绝对像素坐标）
+        color: 圆圈颜色 (R, G, B, A)，默认绿色
+        radius: 圆圈半径，默认5像素
+        width: 圆圈线宽，默认3像素
+    
+    Returns:
+        绘制了标记的图像
+    """
+    if Image is None:
+        logger.warning("PIL未安装，无法绘制标记")
+        return image
+    
+    try:
+        from PIL import ImageDraw, ImageFont
+        
+        img_copy = image.copy()
+        draw = ImageDraw.Draw(img_copy)
+        
+        # 绘制空心圆圈
+        bbox = [x - radius, y - radius, x + radius, y + radius]
+        draw.ellipse(bbox, outline=color, width=width)
+        # 绘制坐标数字
+        draw.text((x, y), f"({x}, {y})", fill=color, font=ImageFont.load_default())    
+        logger.debug(f"在截图上绘制了上次动作标记: ({x}, {y})")
+        return img_copy
+    except Exception as e:
+        logger.error(f"绘制动作标记失败: {e}")
+        return image
+
+
+def image_to_base64(image) -> str:
+    """
+    将 PIL 图像转换为 Base64 编码字符串
+    
+    Args:
+        image: PIL Image对象
+    
+    Returns:
+        Base64 编码字符串
+    """
+    if Image is None:
+        return ""
+    
+    try:
+        img_byte_arr = io.BytesIO()
+        image.save(img_byte_arr, format='PNG')
+        img_byte_arr = img_byte_arr.getvalue()
+        return base64.b64encode(img_byte_arr).decode('utf-8')
+    except Exception as e:
+        logger.error(f"图像转Base64失败: {e}")
+        return ""
+
+
+def get_image_size(image_base64: str) -> Tuple[int, int]:
+    """获取图像尺寸"""
+    if Image is None:
+        return 0, 0
+    
+    try:
+        clean = re.sub(r'^data:image/\w+;base64,', '', image_base64)
+        img = Image.open(io.BytesIO(base64.b64decode(clean)))
+        return img.size
+    except Exception:
+        return 0, 0
+
+
+def strip_base64_prefix(base64_str: str) -> str:
+    """移除 Base64 的 data URI 前缀"""
+    return re.sub(r'^data:image/\w+;base64,', '', base64_str)
+
+
+# ==============================================================================
+# Gmail 邮件检查器 (从 GuiAgent/core/utils/gmail_checker.py 迁移)
+# ==============================================================================
+
+class GmailChecker:
+    """
+    Gmail 邮件检查器，用于 receive_email 动作的邮件获取
+    
+    需要 Google API 相关依赖:
+    - google-auth-oauthlib
+    - google-api-python-client
+    """
+    
+    SCOPES = ["https://www.googleapis.com/auth/gmail.readonly"]
+    
+    def __init__(self, credentials_path: str = None, token_path: str = None):
+        base_dir = os.path.dirname(os.path.abspath(__file__))
+        self.credentials_path = credentials_path or os.path.join(base_dir, "credentials.json")
+        self.token_path = token_path or os.path.join(base_dir, "token.json")
+        self.creds = None
+        self.service = None
+        self._initialized = False
+        
+    def _ensure_initialized(self) -> bool:
+        """延迟初始化 Gmail 服务"""
+        if self._initialized:
+            return self.service is not None
+            
+        self._initialized = True
+        
+        try:
+            import os.path
+            from google.auth.transport.requests import Request
+            from google.oauth2.credentials import Credentials
+            from google_auth_oauthlib.flow import InstalledAppFlow
+            from googleapiclient.discovery import build
+        except ImportError as e:
+            logger.warning(f"Gmail API 依赖未安装: {e}")
+            return False
+        
+        try:
+            # 加载存储的凭证
+            if os.path.exists(self.token_path):
+                self.creds = Credentials.from_authorized_user_file(self.token_path, self.SCOPES)
+            
+            # 如果没有有效凭证，让用户登录
+            if not self.creds or not self.creds.valid:
+                if self.creds and self.creds.expired and self.creds.refresh_token:
+                    self.creds.refresh(Request())
+                else:
+                    flow = InstalledAppFlow.from_client_secrets_file(
+                        self.credentials_path, self.SCOPES
+                    )
+                    self.creds = flow.run_local_server(port=0)
+                
+                # 保存凭证以便下次使用
+                with open(self.token_path, "w") as token:
+                    token.write(self.creds.to_json())
+            
+            # 创建 Gmail 服务
+            self.service = build("gmail", "v1", credentials=self.creds)
+            logger.info("Gmail 服务初始化成功")
+            return True
+            
+        except Exception as e:
+            logger.error(f"Gmail 服务初始化失败: {e}")
+            return False
+    
+    def get_email_content(self, message_id: str) -> Optional[Dict[str, Any]]:
+        """根据邮件ID获取邮件内容"""
+        if not self._ensure_initialized():
+            return None
+            
+        try:
+            from googleapiclient.errors import HttpError
+            
+            message = self.service.users().messages().get(
+                userId="me", id=message_id, format="full"
+            ).execute()
+            
+            # 提取邮件头信息
+            headers = message["payload"]["headers"]
+            subject = ""
+            sender = ""
+            date = ""
+            
+            for header in headers:
+                if header["name"] == "Subject":
+                    subject = header["value"]
+                elif header["name"] == "From":
+                    sender = header["value"]
+                elif header["name"] == "Date":
+                    date = header["value"]
+            
+            # 提取邮件正文
+            def extract_message_text(part):
+                if "parts" in part:
+                    for subpart in part["parts"]:
+                        text = extract_message_text(subpart)
+                        if text:
+                            return text
+                elif part["mimeType"] == "text/plain":
+                    if "data" in part["body"]:
+                        return base64.urlsafe_b64decode(part["body"]["data"]).decode("utf-8")
+                elif part["mimeType"] == "text/html":
+                    if "data" in part["body"]:
+                        html_content = base64.urlsafe_b64decode(part["body"]["data"]).decode("utf-8")
+                        return f"[HTML内容]\n{html_content}"
+                return ""
+            
+            message_text = extract_message_text(message["payload"])
+            
+            # 过滤空行
+            if message_text:
+                lines = message_text.splitlines()
+                filtered_lines = [line for line in lines if line.strip()]
+                message_text = '\n'.join(filtered_lines)
+            
+            return {
+                "id": message_id,
+                "sender": sender,
+                "subject": subject,
+                "date": date,
+                "content": message_text
+            }
+            
+        except Exception as e:
+            logger.error(f"获取邮件内容时出错: {e}")
+            return None
+    
+    def is_email_within_timeframe(self, message: Dict, minutes: int = 2) -> bool:
+        """判断邮件是否在指定时间范围内"""
+        import time as time_module
+        try:
+            internal_date = int(message.get('internalDate', 0)) / 1000
+            current_time = time_module.time()
+            time_diff_minutes = (current_time - internal_date) / 60
+            return time_diff_minutes <= minutes
+        except Exception as e:
+            logger.warning(f"判断邮件时间时出错: {e}")
+            return False
+    
+    def wait_for_new_email(
+        self, 
+        polling_interval: int = 10, 
+        time_window: int = 2, 
+        max_retries: int = 10
+    ) -> List[Dict[str, Any]]:
+        """
+        阻塞等待新邮件，直到收到指定时间窗口内的新邮件后返回所有符合条件的邮件
+        
+        Args:
+            polling_interval: 检查新邮件的时间间隔（秒）
+            time_window: 新邮件的时间窗口（分钟），默认2分钟
+            max_retries: 最大重试次数
+        
+        Returns:
+            符合条件的邮件列表，每封邮件包含 id、sender、subject、date 和 content 字段
+        """
+        import time as time_module
+        
+        if not self._ensure_initialized():
+            logger.error("Gmail 服务未初始化，无法等待邮件")
+            return []
+        
+        logger.info(f"开始监听新邮件，检查间隔: {polling_interval}秒，时间窗口: {time_window}分钟")
+        retry_count = 0
+        
+        while retry_count < max_retries:
+            try:
+                # 检查是否需要刷新凭证
+                from google.auth.transport.requests import Request
+                if self.creds and self.creds.expired and self.creds.refresh_token:
+                    self.creds.refresh(Request())
+                    with open(self.token_path, "w") as token:
+                        token.write(self.creds.to_json())
+                
+                # 计算时间窗口的起始时间戳
+                current_timestamp = int(time_module.time())
+                after_timestamp = current_timestamp - (time_window * 60)
+                query = f"after:{after_timestamp}"
+                
+                # 获取最新的一批邮件，使用 q 参数过滤时间
+                results = self.service.users().messages().list(userId="me", maxResults=10, q=query).execute()
+                messages = results.get("messages", [])
+                
+                valid_emails = []
+                if messages:
+                    for msg_summary in messages:
+                        message_id = msg_summary["id"]
+                        message = self.service.users().messages().get(
+                            userId="me", id=message_id, format="metadata"
+                        ).execute()
+                        
+                        if self.is_email_within_timeframe(message, time_window):
+                            email_content = self.get_email_content(message_id)
+                            if email_content:
+                                valid_emails.append(email_content)
+                        else:
+                            # 假设邮件按时间倒序，遇到超时的可以停止检查
+                            # 但为了保险，可以根据具体需求决定是否 break。
+                            # 这里简单起见不 break，检查完前10封。
+                            pass
+
+                    if valid_emails:
+                        logger.info(f"收到 {len(valid_emails)} 封 {time_window} 分钟内的邮件")
+                        return valid_emails
+                    else:
+                        logger.debug(f"最近的邮件都不在 {time_window} 分钟时间窗口内，继续等待...")
+                else:
+                    logger.debug("没有找到邮件，继续等待...")
+                
+                retry_count += 1
+                time_module.sleep(polling_interval)
+                
+            except Exception as e:
+                logger.error(f"监听邮件时出错: {e}")
+                retry_count += 1
+                time_module.sleep(polling_interval)
+        
+        logger.warning(f"已重试 {max_retries} 次，未收到新邮件")
+        return []
+
+
+def receive_email(
+    token_path: str = "token.json",
+    credentials_path: str = "credentials.json",
+    polling_interval: int = 10,
+    time_window: int = 2,
+    max_retries: int = 10
+) -> List[Dict[str, Any]]:
+    """
+    receive_email 动作的便捷函数
+    
+    Args:
+        token_path: Gmail token 文件路径
+        credentials_path: Gmail credentials 文件路径
+        polling_interval: 检查间隔（秒）
+        time_window: 时间窗口（分钟）
+        max_retries: 最大重试次数
+    
+    Returns:
+        邮件内容列表
+    """
+    logger.info("开始接收新邮件")
+    
+    try:
+        gmail_checker = GmailChecker(
+            token_path=token_path,
+            credentials_path=credentials_path
+        )
+        
+        new_emails = gmail_checker.wait_for_new_email(
+            polling_interval=polling_interval,
+            time_window=time_window,
+            max_retries=max_retries
+        )
+        
+        return new_emails if new_emails else []
+        
+    except Exception as e:
+        logger.error(f"执行 receive_email 失败: {e}")
+        return []
+
+def login_ios(device) -> tuple:
+    """
+    iOS Apple ID 密码登录脚本
+    
+    在 agent 识别到系统弹出的 Apple ID 登录对话框时调用，
+    通过 WDA 直接操作完成密码输入和登录，避免 agent 操作系统级 UI 出错。
+    
+    完整流程：从点击"通过密码登录"按钮开始，覆盖不同页面调用的场景。
+    
+    Args:
+        device: IOSDevice 实例，需要有 _wda_client 属性
+    Returns:
+        (success: bool, message: str)
+    """
+    import time as time_module
+    
+    APPLE_PASSWORD = "tsIcmustyy1"
+    
+    # 获取 WDA 客户端
+    wda_client = getattr(device, '_wda_client', None)
+    if wda_client is None:
+        logger.error("[login_ios] 无法获取 WDA 客户端")
+        return False, "无法获取 WDA 客户端"
+    
+    try:
+        # === 第1步：查找并点击"通过密码登录"相关按钮 ===
+        # 覆盖中英文多种表述，确保从不同页面调用都能处理
+        password_login_labels = [
+            "通过密码登录", "用密码登录", "使用密码登录", "密码登录",
+            "Use Password", "Sign In with Password", "Use Password…",
+            "Use Password...",
+        ]
+        
+        clicked_password_btn = False
+        for label in password_login_labels:
+            try:
+                if wda_client(label=label).click_exists(timeout=1.0):
+                    logger.info(f"[login_ios] 点击了密码登录按钮: {label}")
+                    clicked_password_btn = True
+                    time_module.sleep(1.5)  # 等待密码输入界面加载
+                    break
+            except Exception as e:
+                logger.debug(f"[login_ios] 查找按钮 '{label}' 异常: {e}")
+                continue
+        
+        if not clicked_password_btn:
+            # 可能已经在密码输入页面，尝试直接查找密码输入框
+            logger.info("[login_ios] 未找到密码登录按钮，尝试直接查找密码输入框")
+        
+        # === 第2步：查找并操作密码输入框 ===
+        password_field = None
+        # 尝试多次查找，等待 UI 加载
+        for attempt in range(5):
+            try:
+                # SecureTextField 是 iOS 密码输入框的控件类型
+                secure_fields = wda_client(type='SecureTextField')
+                if secure_fields.exists:
+                    password_field = secure_fields
+                    logger.info(f"[login_ios] 找到密码输入框 (尝试 {attempt + 1})")
+                    break
+            except Exception as e:
+                logger.debug(f"[login_ios] 查找密码输入框异常 (尝试 {attempt + 1}): {e}")
+            time_module.sleep(1.0)
+        
+        if password_field is None:
+            logger.warning("[login_ios] 未找到密码输入框")
+            return False, "未找到密码输入框，可能当前页面不在密码登录界面"
+        
+        # 点击密码输入框使其获得焦点
+        try:
+            password_field.tap()
+            time_module.sleep(0.5)
+        except Exception as e:
+            logger.warning(f"[login_ios] 点击密码输入框失败: {e}")
+        
+        # 清除已有内容（避免重复输入导致密码错误）
+        try:
+            current_value = password_field.get().value
+            if current_value:
+                # 全选并删除
+                password_field.clear_text()
+                time_module.sleep(0.3)
+                logger.debug("[login_ios] 已清除密码输入框内容")
+        except Exception as e:
+            logger.debug(f"[login_ios] 清除内容时异常（可继续）: {e}")
+        
+        # 输入密码
+        try:
+            password_field.set_text(APPLE_PASSWORD)
+            time_module.sleep(0.5)
+            logger.info("[login_ios] 密码输入完成")
+        except Exception as e:
+            logger.error(f"[login_ios] 密码输入失败: {e}")
+            return False, f"密码输入失败: {e}"
+        
+        # === 第3步：点击登录按钮 ===
+        sign_in_labels = [
+            "登录", "Sign In", "sign in", "登入", "确定", "OK",
+        ]
+        
+        clicked_sign_in = False
+        for label in sign_in_labels:
+            try:
+                if wda_client(label=label).click_exists(timeout=1.0):
+                    logger.info(f"[login_ios] 点击了登录按钮: {label}")
+                    clicked_sign_in = True
+                    break
+            except Exception as e:
+                logger.debug(f"[login_ios] 查找登录按钮 '{label}' 异常: {e}")
+                continue
+        
+        if not clicked_sign_in:
+            # 尝试通过 type=Button 查找登录按钮
+            try:
+                for label in sign_in_labels:
+                    if wda_client(label=label, type='Button').click_exists(timeout=1.0):
+                        logger.info(f"[login_ios] 通过 Button 类型点击登录: {label}")
+                        clicked_sign_in = True
+                        break
+            except Exception as e:
+                logger.debug(f"[login_ios] Button 类型查找异常: {e}")
+        
+        if not clicked_sign_in:
+            logger.warning("[login_ios] 未找到登录按钮，密码已输入但无法提交")
+            return False, "密码已输入但未找到登录按钮"
+        
+        # === 第4步：等待并检查登录结果 ===
+        time_module.sleep(3)  # 等待登录处理
+        
+        # 检查密码输入框是否已消失（消失说明登录成功或进入下一步）
+        try:
+            if not wda_client(type='SecureTextField').exists:
+                logger.info("[login_ios] 登录成功：密码输入框已消失")
+                return True, "Apple ID 密码登录成功"
+            else:
+                # 密码框仍存在，可能密码错误
+                logger.warning("[login_ios] 密码通过后输入框仍存在，可能登录需要额外信息填写")
+                return False, "登录可能失败：密码通过后输入框仍存在，可能登录需要额外信息填写"
+        except Exception as e:
+            logger.debug(f"[login_ios] 检查登录结果异常: {e}")
+            # 异常时保守认为成功（可能页面已跳转导致元素不存在）
+            return True, "登录操作已执行（无法确认结果）"
+        
+    except Exception as e:
+        logger.error(f"[login_ios] 执行异常: {e}")
+        return False, f"登录脚本异常: {e}"
+
+if __name__ == "__main__":
+    new_emails = receive_email()
+    if new_emails:
+        for email in new_emails:
+            print(email)
+    else:
+        print("没有新邮件")

DroidBot/input_manager.py

@@ -0,0 +1,352 @@
+import json
+import logging
+import time
+
+from .core import EventLog
+from .core.abstract_input_event import EventType
+from .input_policy import POLICY_NONE, POLICY_MEMORY_GUIDED, POLICY_MANUAL, MemoryGuidedPolicy, NoneInputPolicy, ManualPolicy
+from .traffic_monitor import TrafficMonitor
+from .exceptions import FATAL_EXCEPTIONS, InputInterruptedException, AppCrashException, AppNeedUpdateException, AppLaunchErrorException
+DEFAULT_POLICY = POLICY_MEMORY_GUIDED
+DEFAULT_EVENT_INTERVAL = 1
+DEFAULT_EVENT_COUNT = 100000000
+DEFAULT_TIMEOUT = -1
+
+WARMUP_STEPS = 10
+STALL_STEPS_LIMIT = 33
+STOP_STALL_STEPS_LIMIT = 3 * STALL_STEPS_LIMIT
+MIN_EXPLORATION_STEPS = 300
+MAX_EXPLORATION_STEPS = 750
+BLOCK_MIN_EXPLORATION_STEPS = 100
+BLOCK_MAX_EXPLORATION_STEPS = 300
+APP_CRASH_CYCLE_LIMIT = 3  # 闪退检测: 连续关闭->拉起失败的次数阈值
+GOOGLE_PLAY_RECOVER_LIMIT = 3
+GOOGLE_PLAY_STABLE_STEP_LIMIT = 3
+REDIRECT_PULL_BACK_LIMIT = 3
+
+class InputManager(object):
+    """
+    This class manages all events to send during app running
+    """
+
+    def __init__(self, device, policy_name, random_input,
+                 event_count, event_interval,
+                 profiling_method=None,
+                 replay_output=None, enable_guiagent=False, app_name=None,
+                 pcap_callback=None,
+                 enable_app_block=False):
+        """
+        manage input event sent to the target device
+        :param device: instance of Device (device internally manages app)
+        :param policy_name: policy of ting events, string
+        :param pcap_callback: callback to push pcap file
+        :param enable_app_block: whether traffic blocking is enabled for this task
+        :return:
+        """
+        self.logger = logging.getLogger('InputEventManager')
+        self.enabled = True
+
+        self.device = device
+        self.policy_name = policy_name
+        self.random_input = random_input
+        self.events = []
+        self.policy = None
+        self.event_count = event_count
+        self.event_interval = event_interval
+        self.replay_output = replay_output
+        self.enable_guiagent = enable_guiagent
+        self.app_name = app_name
+        self.profiling_method = profiling_method
+        self.pcap_callback = pcap_callback
+        self.enable_app_block = enable_app_block
+
+        # Initialize TrafficMonitor and step counters (must be before get_input_policy)
+        self.traffic_monitor = TrafficMonitor(device, device.app_identifier, pcap_callback=self.pcap_callback)
+        self.total_exploring_steps = 0
+        self.current_stall_steps = 0
+        self.steps_since_last_stall_check = 0
+        self._force_stop = False
+        # 用户手动中断标志（KeyboardInterrupt）
+        self.user_interrupted = False
+        
+        # GuiAgent execution flag - skip foreground check and step counting during agent execution
+        self.is_guiagent_executing = False
+        
+        # 闪退检测: 追踪连续关闭->拉起失败的循环次数
+        self._crash_cycle_count = 0
+        self._awaiting_restart_result = False  # 标记是否正在等待重启结果
+        
+        # Google Play 拉回恢复状态
+        self._google_play_recover_attempts = 0
+        self._google_play_stable_steps = 0
+        
+        # 跳转目标包名记录
+        self._last_redirect_package = None
+        self._redirect_pull_back_count = 0
+        
+        # Initialize policy (uses traffic_monitor)
+        self.policy = self.get_input_policy(device)
+
+    def _reset_google_play_recovery(self):
+        self._google_play_recover_attempts = 0
+        self._google_play_stable_steps = 0
+
+    def _handle_google_play_redirect(self):
+        self._google_play_recover_attempts += 1
+        self._google_play_stable_steps = 0
+
+        self.logger.warning(
+            f"检测到 Google Play Store，尝试拉回 "
+            f"({self._google_play_recover_attempts}/{GOOGLE_PLAY_RECOVER_LIMIT})"
+        )
+
+        if self.device.pull_back_to_app():
+            self.logger.info("Google Play Store 拉回成功，进入稳定观察")
+        else:
+            self.logger.warning("Google Play Store 拉回失败")
+            time.sleep(1)
+
+        if self._google_play_recover_attempts >= GOOGLE_PLAY_RECOVER_LIMIT:
+            self.logger.error(
+                f"连续 {GOOGLE_PLAY_RECOVER_LIMIT} 次从 Google Play Store 拉回仍未稳定，判定为需更新"
+            )
+            raise AppNeedUpdateException("应用反复跳转 Google Play Store，需更新")
+
+    def _mark_google_play_recovered(self):
+        if self._google_play_recover_attempts == 0:
+            return
+
+        self._google_play_stable_steps += 1
+        if self._google_play_stable_steps < GOOGLE_PLAY_STABLE_STEP_LIMIT:
+            self.logger.info(
+                f"Google Play Store 拉回后稳定观察 "
+                f"({self._google_play_stable_steps}/{GOOGLE_PLAY_STABLE_STEP_LIMIT})"
+            )
+            return
+
+        self.logger.info("Google Play Store 拉回后已稳定，清空恢复状态")
+        self._reset_google_play_recovery()
+
+    def get_input_policy(self, device):
+        if self.policy_name == POLICY_NONE:
+            input_policy = NoneInputPolicy(device, enable_guiagent=self.enable_guiagent)
+        elif self.policy_name == POLICY_MEMORY_GUIDED:
+            input_policy = MemoryGuidedPolicy(device, self.random_input, enable_guiagent=self.enable_guiagent, app_name=self.app_name, traffic_monitor=self.traffic_monitor)
+        elif self.policy_name == POLICY_MANUAL:
+            input_policy = ManualPolicy(device, enable_guiagent=self.enable_guiagent)
+        else:
+            self.logger.warning("No valid input policy specified. Using policy \"none\".")
+            input_policy = None
+
+        return input_policy
+
+    def add_event(self, event):
+        """
+        add one event to the event list
+        :param event: the event to be added, should be subclass of AppEvent
+        :return:
+        """
+        if event is None:
+            return
+        
+        # 如果GuiAgent正在执行，跳过前台检查和步数累计，只执行事件
+        if self.is_guiagent_executing:
+            self.logger.debug(f"[GuiAgent执行中] 直接执行事件: {event.event_type}")
+            self.events.append(event)
+            event_log = EventLog(self.device, self.device._app, event, self.profiling_method)
+            event_log.start()
+            event_log.stop()
+            return
+
+        # 1. 基础限制检查
+        max_steps = BLOCK_MAX_EXPLORATION_STEPS if self.enable_app_block else MAX_EXPLORATION_STEPS
+        min_steps = BLOCK_MIN_EXPLORATION_STEPS if self.enable_app_block else MIN_EXPLORATION_STEPS
+        if self.total_exploring_steps >= max_steps:
+            self.logger.info(f"Reached max exploration steps ({max_steps}), stopping collection.")
+            raise InputInterruptedException()
+
+        # 2. 前台状态与采集终止检查
+        is_kill_app = hasattr(event, 'event_type') and event.event_type == EventType.KILL_APP
+        if not is_kill_app:
+            curr_in_foreground = self.device.is_foreground()
+            if not curr_in_foreground:
+                # C10: Web平台简化处理，无app_store/launcher概念
+                if self.device.get_platform_name() == "web":
+                    self.logger.info("Web平台: 检测到离站，尝试拉回")
+                    self.device.pull_back_to_app()
+                    return
+                
+                redirect_info = self.device.get_redirect_target_info()
+                redirect_type = redirect_info.get("type") if redirect_info else "unknown"
+                redirect_target = redirect_info.get("target") if redirect_info else None
+                
+                # 情况1: 跳转到 Google Play Store -> 需连续检测确认
+                if redirect_type == "app_store":
+                    self._handle_google_play_redirect()
+                    return
+                
+                # 情况2: 跳转到桌面 -> 闪退逻辑
+                if redirect_type in {"launcher", "unknown"}:
+                    self._last_redirect_package = None
+                    self._redirect_pull_back_count = 0
+                    
+                    if self.total_exploring_steps >= min_steps and self._force_stop:
+                        self.logger.info(f"App out of foreground and total steps ({self.total_exploring_steps}) >= min exploration steps ({min_steps}) with _force_stop. Ending.")
+                        raise InputInterruptedException()
+                    else:
+                        self.logger.info(f"App out of foreground (launcher) but not reached min exploration steps ({self.total_exploring_steps}/{min_steps}). Pulling back.")
+                        pull_back_success = self.device.pull_back_to_app()
+                        if not pull_back_success:
+                            if self._awaiting_restart_result:
+                                self._crash_cycle_count += 1
+                                self.logger.warning(f"应用重启后再次失败，闪退循环计数: {self._crash_cycle_count}/{APP_CRASH_CYCLE_LIMIT}")
+                                if self._crash_cycle_count >= APP_CRASH_CYCLE_LIMIT:
+                                    if self.enable_app_block:
+                                        self.logger.warning(f"Block任务检测到连续闪退 {self._crash_cycle_count} 次，重置计数并继续尝试（流量阻塞可能导致正常闪退）")
+                                        self._crash_cycle_count = 0
+                                        self._awaiting_restart_result = False
+                                    else:
+                                        self.logger.error(f"检测到应用闪退！连续 {self._crash_cycle_count} 次关闭->拉起失败循环")
+                                        raise AppCrashException(f"应用闪退：连续 {self._crash_cycle_count} 次关闭并重启后仍无法运行")
+                            
+                            self.logger.warning("前台拉回失败，将关闭应用并重启")
+                            from .core import PlatformFactory
+                            CloseAppEvent = PlatformFactory.get_event_class(self.device.get_platform_name(), 'kill_app')
+                            if CloseAppEvent:
+                                event = CloseAppEvent(app=self.device.app_identifier)
+                                self._awaiting_restart_result = True
+                                time.sleep(10)
+                
+                # 情况3: 跳转到其他应用 -> 记录目标并尝试拉回
+                else:
+                    self._last_redirect_package = redirect_target
+                    self.logger.warning(f"应用跳转到其他应用: {redirect_target}，尝试拉回")
+                    
+                    pull_back_success = self.device.pull_back_to_app()
+                    if pull_back_success:
+                        self._redirect_pull_back_count = 0
+                        self._last_redirect_package = None
+                        self.logger.info(f"成功拉回到目标应用")
+                    else:
+                        self._redirect_pull_back_count += 1
+                        self.logger.warning(f"拉回失败 ({self._redirect_pull_back_count}/{REDIRECT_PULL_BACK_LIMIT})，跳转目标: {self._last_redirect_package}")
+                        
+                        if self._redirect_pull_back_count >= REDIRECT_PULL_BACK_LIMIT:
+                            self.logger.error(f"连续 {REDIRECT_PULL_BACK_LIMIT} 次拉回失败，判定为启动异常，跳转目标: {self._last_redirect_package}")
+                            raise AppLaunchErrorException(f"启动异常：跳转至 {self._last_redirect_package}")
+            else:
+                if self._force_stop:
+                    self.logger.info("App is in foreground, resetting _force_stop.")
+                    self._force_stop = False
+                if self._awaiting_restart_result:
+                    self.logger.info("应用重启成功，重置闪退检测计数")
+                    self._crash_cycle_count = 0
+                    self._awaiting_restart_result = False
+                self._mark_google_play_recovered()
+                self._last_redirect_package = None
+                self._redirect_pull_back_count = 0
+
+        # 3. 统计当前步
+        self.total_exploring_steps += 1
+        self.current_stall_steps += 1
+        self.steps_since_last_stall_check += 1
+
+        # 4. 流量监控与 停滞处理
+        has_new = False
+        if self.total_exploring_steps >= WARMUP_STEPS:
+            self.traffic_monitor.update()
+            has_new, new_count = self.traffic_monitor.has_new_features()
+        if self.steps_since_last_stall_check >= STALL_STEPS_LIMIT:
+            # 监控环境健康检查
+            self.traffic_monitor.check_monitor_health()
+            self.logger.info(f"Steps reached {STALL_STEPS_LIMIT}, checking for new features...")
+            if not has_new:
+                from .core import PlatformFactory
+                if self.current_stall_steps >= STOP_STALL_STEPS_LIMIT:
+                    if self.device.check_network():
+                        self.logger.info("Network check success, force stop")
+                        self._force_stop = True
+                    else:
+                        self.logger.warning("Network check failed, retry")
+                        self._force_stop = False
+                    self.logger.info(f"No new features for {self.current_stall_steps} steps. Generating KillAppEvent.")
+                    KillAppEvent = PlatformFactory.get_event_class(self.device.get_platform_name(), 'kill_app')
+                    if KillAppEvent:
+                        event = KillAppEvent(app=self.device.app_identifier)
+                        self.current_stall_steps = 0
+                else:
+                    # 总步数达标，标志停止
+                    if self.total_exploring_steps >= min_steps:
+                        if self.device.check_network():
+                            self.logger.info("Network check success, force stop")
+                            self._force_stop = True
+                        else:
+                            self.logger.warning("Network check failed, retry")
+                            self._force_stop = False
+                    self.logger.info(f"No new features detected. Overriding with BACK event.")
+                    KeyEvent = PlatformFactory.get_event_class(self.device.get_platform_name(), 'key')
+                    if KeyEvent:
+                        event = KeyEvent(key_name="BACK")
+                
+                # 重置单次 check 计数，以便下一次 loop 继续 check
+                self.steps_since_last_stall_check = 0
+            else:
+                self.logger.info(f"New features found ({new_count}). Resetting stall counters.")
+                self.current_stall_steps = 0
+                self.steps_since_last_stall_check = 0
+                self._force_stop = False
+
+        # 5. 执行事件
+        self.logger.info(f"Step {self.total_exploring_steps}: sending {event.event_type} (stall: {self.current_stall_steps}/{STOP_STALL_STEPS_LIMIT}, check: {self.steps_since_last_stall_check}/{STALL_STEPS_LIMIT})")
+        self.events.append(event)
+        event_log = EventLog(self.device, self.device._app, event, self.profiling_method)
+        event_log.start()
+        event_log.stop()
+
+    def start(self):
+        """
+        start sending event
+        """
+        self.logger.info("start sending events, policy is %s" % self.policy_name)
+
+        try:
+            if self.policy is not None:
+                self.policy.start(self)
+            elif self.policy_name == POLICY_NONE:
+                self.device.start_app()
+                if self.event_count == 0:
+                    return
+                while self.enabled:
+                    time.sleep(1)
+            elif self.policy_name == POLICY_MANUAL:
+                self.device.start_app()
+                while self.enabled:
+                    keyboard_input = input("press ENTER to save current state, type q to exit...")
+                    if keyboard_input.startswith('q'):
+                        break
+                    state = self.device.get_current_state()
+                    if state is not None:
+                        state.save2dir()
+        except KeyboardInterrupt:
+            # 记录用户手动中断标志，并向上传播
+            self.user_interrupted = True
+            raise
+        except FATAL_EXCEPTIONS:
+            raise
+        except Exception as e:
+            self.logger.error(f'Non-fatal exception in input_manager: {e}')
+            import traceback
+            traceback.print_exc()
+        finally:
+            # 探索结束后，一次性完成所有文件写入（无论正常结束还是异常退出）
+            if self.policy and hasattr(self.policy, 'utg'):
+                self.policy.utg.finalize()
+            self.stop()
+            self.logger.info("Finish sending events")
+
+    def stop(self):
+        """
+        stop sending event
+        """
+        self.enabled = False
+

DroidBot/input_policy.py

@@ -0,0 +1,897 @@
+"""
+Input Policy Module for DroidBot
+Contains MemoryGuidedPolicy for intelligent UI exploration.
+"""
+import sys
+import json
+import logging
+import random
+import collections
+import copy
+import time
+
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn.utils.rnn import pad_sequence
+
+from .core import PlatformFactory, Platform
+from .core.abstract_input_event import BaseTouchEvent, BaseKeyEvent, EventType, BaseKillAppEvent,BaseSetTextEvent
+from .utg import UTG
+from .guiagent_bridge import GuiAgentBridge
+from .exceptions import FATAL_EXCEPTIONS, InputInterruptedException, ExplorationStuckException
+
+# 注意：日志配置现在由统一的 logging_config 模块管理
+# logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)-12s %(levelname)-8s %(message)s")
+
+# Helper function to get event classes from device
+def get_event_class(device, event_type):
+    """Get platform-specific event class from device"""
+    platform_name = device.get_platform_name()
+    platform = Platform(platform_name)
+    return PlatformFactory.get_event_class(platform, event_type)
+
+# Policy constants
+POLICY_NONE = "none"
+POLICY_MEMORY_GUIDED = "memory_guided"
+POLICY_MANUAL = "manual"
+
+# Memory-guided policy constants
+DEBUG = True
+ACTION_INEFFECTIVE = 'ineffective'
+CLOSER_ACTION_ENCOURAGEMENT = 0.01
+RANDOM_EXPLORE_PROB = 0.4
+N_ACTIONS_TRAINING = 32
+MAX_NAV_STEPS = 10
+
+# ==================== Neural Network Models ====================
+
+class TextEncoder:
+    """Text encoder using BERT or spacy for view text embedding."""
+    def __init__(self, method='spacy'):
+        self.method = method
+        self.embed_size = -1
+        self._initialized = False
+        self._nlp = None
+        self._tokenizer = None
+        self._text_encoder = None
+        if method == 'spacy':
+            self.embed_size = 300
+        if method == 'bert':
+            self.embed_size = 768
+
+    def _initialize(self):
+        """Lazy initialization of the encoder models."""
+        if self._initialized:
+            return
+        if self.method == 'spacy':
+            import spacy
+            self._nlp = spacy.load("en_core_web_md")
+        if self.method == 'bert':
+            import os
+            from transformers import BertTokenizer, BertModel
+            local_model_path = os.path.join(os.path.dirname(__file__), 'cv', 'huggingface_models')
+            self._tokenizer = BertTokenizer.from_pretrained(local_model_path)
+            self._text_encoder = BertModel.from_pretrained(local_model_path)
+        self._initialized = True
+
+    def encode(self, text):
+        self._initialize()
+        if not text:
+            return np.zeros(self.embed_size)
+        if self.method == 'spacy':
+            doc = self._nlp(text)
+            return doc.vector
+        if self.method == 'bert':
+            encoding = self._tokenizer([text], return_tensors='pt', padding=True, truncation=True)
+            input_ids = encoding['input_ids']
+            attention_mask = encoding['attention_mask']
+            text_encoder_out = self._text_encoder(input_ids, attention_mask=attention_mask)
+            text_emb = text_encoder_out['pooler_output'][0]
+            return text_emb.detach().cpu().numpy()
+
+
+class BertLayerNorm(nn.Module):
+    """TF-style LayerNorm."""
+    def __init__(self, hidden_size, eps=1e-5):
+        super(BertLayerNorm, self).__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.bias = nn.Parameter(torch.zeros(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, x):
+        u = x.mean(-1, keepdim=True)
+        s = (x - u).pow(2).mean(-1, keepdim=True)
+        x = (x - u) / torch.sqrt(s + self.variance_epsilon)
+        return self.weight * x + self.bias
+
+
+class AbsolutePositionalEncoding(nn.Module):
+    """Absolute positional encoding for UI elements."""
+    def __init__(self, d_model, pos_max=128):
+        super().__init__()
+        self.pos_max = pos_max
+        self.d_model = d_model
+        nhid = d_model
+        self.x_position_embeddings = nn.Embedding(self.pos_max, nhid)
+        self.y_position_embeddings = nn.Embedding(self.pos_max, nhid)
+        self.h_position_embeddings = nn.Embedding(self.pos_max, nhid)
+        self.w_position_embeddings = nn.Embedding(self.pos_max, nhid)
+
+    def forward(self, pos_enc):
+        l_emb = self.x_position_embeddings(pos_enc[:, 0])
+        r_emb = self.x_position_embeddings(pos_enc[:, 1])
+        t_emb = self.y_position_embeddings(pos_enc[:, 2])
+        b_emb = self.y_position_embeddings(pos_enc[:, 3])
+        w_emb = self.w_position_embeddings(pos_enc[:, 4])
+        h_emb = self.h_position_embeddings(pos_enc[:, 5])
+        pos_emb = l_emb + r_emb + t_emb + b_emb + w_emb + h_emb
+        return pos_emb
+
+
+class UIEmbedTransformer(nn.Module):
+    """Transformer-based UI element embedder."""
+    def __init__(self, nhid=64, nhead=2, nlayers=2, dropout=0.8):
+        super().__init__()
+        from torch.nn import TransformerEncoder, TransformerEncoderLayer
+        self.pos_max = 128
+        self.text_encoder = TextEncoder(method='bert')
+        dim_feedforward = 256
+        encoder_layers = TransformerEncoderLayer(nhid, nhead, dim_feedforward, dropout)
+        self.transformer_encoder = TransformerEncoder(encoder_layers, nlayers)
+        self.meta2hid = nn.Linear(12, nhid)
+        self.text2hid = nn.Linear(self.text_encoder.embed_size, nhid)
+        self.pos2hid = AbsolutePositionalEncoding(d_model=nhid, pos_max=self.pos_max)
+        self.layer_norm = BertLayerNorm(nhid)
+        self.dropout = nn.Dropout(dropout)
+
+    def forward(self, state_encs):
+        state_encs, attn_mask = self.encode_state_batch(state_encs)
+        output = self.transformer_encoder(state_encs, src_key_padding_mask=attn_mask)
+        output = output.permute(1, 0, 2)
+        return output
+
+    def encode_state(self, state, views):
+        meta_enc = torch.stack([self._encode_view_meta(state, view) for view in views])
+        pos_enc = torch.stack([self._encode_view_pos(state, view) for view in views])
+        text_enc = torch.stack([self._encode_view_text(state, view) for view in views])
+        return meta_enc, pos_enc, text_enc
+
+    def encode_state_batch(self, state_encs):
+        embs = []
+        for state_enc in state_encs:
+            meta_enc, pos_enc, text_enc = state_enc
+            meta_emb = self.meta2hid(meta_enc)
+            pos_emb = self.pos2hid(pos_enc)
+            text_emb = self.text2hid(text_enc)
+            emb = meta_emb + pos_emb + text_emb
+            emb = self.layer_norm(emb)
+            emb = self.dropout(emb)
+            embs.append(emb)
+        embs_pad = pad_sequence(embs, batch_first=False)
+        attn_mask = embs_pad.sum(axis=2).t() == 0
+        return embs_pad, attn_mask
+
+    def _encode_view_meta(self, state, view):
+        view_children = view['children'] if 'children' in view else []
+        is_parent = 1 if len(view_children) > 0 else -1
+        view_text = view['text'] if 'text' in view else None
+        is_text = 1 if view_text and len(view_text) > 0 else -1
+        enabled = 1 if 'enabled' in view and view['enabled'] else -1
+        visible = 1 if 'visible' in view and view['visible'] else -1
+        clickable = 1 if 'clickable' in view and view['clickable'] else -1
+        long_clickable = 1 if 'long_clickable' in view and view['long_clickable'] else -1
+        checkable = 1 if 'checkable' in view and view['checkable'] else -1
+        checked = 1 if 'checked' in view and view['checked'] else -1
+        selected = 1 if 'selected' in view and view['selected'] else -1
+        editable = 1 if 'editable' in view and view['editable'] else -1
+        is_password = 1 if 'is_password' in view and view['is_password'] else -1
+        scrollable = 1 if 'scrollable' in view and view['scrollable'] else -1
+        meta_enc = np.array([
+            is_parent, is_text, is_password, visible,
+            enabled, checked, selected,
+            clickable, long_clickable, checkable, editable, scrollable
+        ])
+        return torch.Tensor(meta_enc)
+
+    def _encode_view_pos(self, state, view):
+        screen_w = state.width
+        screen_h = state.height
+        [[l,t], [r,b]] = view['bounds'] if 'bounds' in view else [[0,0], [0,0]]
+        l, r, t, b = l/screen_w, r/screen_w, t/screen_h, b/screen_h
+        if l > r:
+            l, r = r, l
+        if t > b:
+            t, b = b, t
+        l = max(0, min(1, l))
+        r = max(0, min(1, r))
+        t = max(0, min(1, t))
+        b = max(0, min(1, b))
+        pos_max = self.pos_max - 1
+        l, r, t, b = int(pos_max*l), int(pos_max*r), int(pos_max*t), int(pos_max*b)
+        w = abs(l - r)
+        h = abs(t - b)
+        return torch.LongTensor(np.array([l, r, t, b, w, h]))
+
+    def _encode_view_text(self, state, view):
+        view_text = view['text'] if 'text' in view else None
+        emb = self.text_encoder.encode(view_text)
+        return torch.Tensor(emb)
+
+
+# ==================== Memory Class ====================
+
+class Memory:
+    """Memory for storing and learning from UI transitions."""
+    def __init__(self, utg, device):
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.utg = utg
+        self.device = device
+        self.known_states = collections.OrderedDict()
+        self.known_transitions = collections.OrderedDict()
+        self.known_structures = collections.OrderedDict()
+        self.model = UIEmbedTransformer()
+
+    def _memorize_state(self, state):
+        if not self.device.is_foreground():
+            return None
+        if state.state_str not in self.known_states:
+            views = state.views
+            views_str = [view['view_str'] for view in views]
+            state_enc = self.model.encode_state(state, views)
+            embedder = self.model
+            embedder.eval()
+            with torch.no_grad():
+                views_emb = self.model.forward([state_enc])
+                views_emb = views_emb.detach().cpu()[0]
+            self.known_states[state.state_str] = {
+                'state': state,
+                'views': views,
+                'views_str': views_str,
+                'state_enc': state_enc,
+                'views_emb': views_emb
+            }
+        return self.known_states[state.state_str]
+
+    def save_transition(self, action, from_state, to_state):
+        if not from_state or not to_state:
+            return
+        from_state_info = self._memorize_state(from_state)
+        if isinstance(action, BaseKillAppEvent) or isinstance(action, BaseKeyEvent) or isinstance(action, BaseSetTextEvent):
+            return
+        if not hasattr(action, 'view') or action.view is None:
+
+            return
+        action_str = action.get_event_str(state=from_state)
+        if action_str in self.known_transitions and self.known_transitions[action_str]['to_state'] == to_state:
+            return
+        if from_state_info is None:
+            return
+        view = action.view
+        if view['view_str'] not in from_state_info['views_str']:
+            self.logger.warning(f"View {view['view_str']} not found in from_state's views")
+            return
+        view_idx = from_state_info['views_str'].index(view['view_str'])
+        action_target = ACTION_INEFFECTIVE \
+            if from_state.structure_str == to_state.structure_str \
+            else to_state.structure_str
+        action_effect = action_target
+        self.known_transitions[action_str] = {
+            'from_state': from_state,
+            'to_state': to_state,
+            'action': action,
+            'view_idx': view_idx,
+            'action_effect': action_effect
+        }
+
+    def save_structure(self, state):
+        structure_str = state.structure_str
+        is_new_structure = False
+        if structure_str not in self.known_structures:
+            self.known_structures[structure_str] = []
+            is_new_structure = True
+        self.known_structures[structure_str].append(state)
+        return is_new_structure
+
+    def _select_transitions_for_training(self, size):
+        if len(self.known_transitions) <= size:
+            return list(self.known_transitions.keys())
+        effect2actions = {}
+        for k,v in self.known_transitions.items():
+            action_effect = v['action_effect']
+            if action_effect not in effect2actions:
+                effect2actions[action_effect] = []
+            effect2actions[action_effect].append(k)
+        action_strs = []
+        action_probs = []
+        prob_per_effect = 1.0 / len(effect2actions)
+        for effect in effect2actions:
+            prob_per_action = prob_per_effect / len(effect2actions[effect])
+            for action_str in effect2actions[effect]:
+                action_strs.append(action_str)
+                action_probs.append(prob_per_action)
+        action_probs = np.array(action_probs) / sum(action_probs)
+        selected_actions = np.random.choice(action_strs, size=size, replace=False, p=action_probs)
+        return selected_actions
+
+    def encode_action_pairs(self, action_strs=None):
+        if action_strs is None:
+            action_strs = list(self.known_transitions.keys())
+        state_strs = [self.known_transitions[action_str]['from_state'].state_str for action_str in action_strs]
+        state_encs = [self.known_states[state_str]['state_enc'] for state_str in state_strs]
+        action_pairs = []
+        for i, action_str1 in enumerate(action_strs):
+            state_str1 = self.known_transitions[action_str1]['from_state'].state_str
+            state_idx1 = state_strs.index(state_str1)
+            view_idx1 = self.known_transitions[action_str1]['view_idx']
+            for j, action_str2 in enumerate(action_strs[i+1:]):
+                state_str2 = self.known_transitions[action_str2]['from_state'].state_str
+                state_idx2 = state_strs.index(state_str2)
+                view_idx2 = self.known_transitions[action_str2]['view_idx']
+                action_effect1 = self.known_transitions[action_str1]['action_effect']
+                action_effect2 = self.known_transitions[action_str2]['action_effect']
+                effect_same = 1 if action_effect1 == action_effect2 else 0
+                action_pairs.append((state_idx1, view_idx1, state_idx2, view_idx2, effect_same))
+        return state_encs, action_pairs
+    
+    def get_known_actions_emb(self):
+        actions_emb = []
+        for action_str in self.known_transitions:
+            action_info = self.known_transitions[action_str]
+            state_str = action_info['from_state'].state_str
+            view_idx = action_info['view_idx']
+            if state_str not in self.known_states:
+                continue
+            action_emb = self.known_states[state_str]['views_emb'][view_idx]
+            actions_emb.append(action_emb)
+        return torch.stack(actions_emb) if len(actions_emb) > 0 else None
+
+    @staticmethod
+    def action_info_str(action_info):
+        state_activity = action_info['from_state'].foreground_page
+        view_sig = action_info['action'].view.get('signature', '')
+        action_effect = action_info['action_effect']
+        return f'{state_activity}-{view_sig}-{action_effect}'
+
+    def train_model(self):
+        if len(self.known_transitions.keys()) < 2:
+            return
+
+        embedder = self.model
+        optimizer = torch.optim.Adam(embedder.parameters(), lr=1e-3)
+        n_iterations = 10
+
+        def compute_loss(ele_embed, action_pairs):
+            pos_emb_u, pos_emb_v = [], []
+            neg_emb_u, neg_emb_v = [], []
+            for state_idx1, view_idx1, state_idx2, view_idx2, effect_same in action_pairs:
+                emb_u = ele_embed[state_idx1, view_idx1]
+                emb_v = ele_embed[state_idx2, view_idx2]
+                if effect_same:
+                    pos_emb_u.append(emb_u)
+                    pos_emb_v.append(emb_v)
+                else:
+                    neg_emb_u.append(emb_u)
+                    neg_emb_v.append(emb_v)
+            pos_score, neg_score = 0, 0
+            if len(pos_emb_u) > 0 and len(pos_emb_v) > 0:
+                pos_emb_u = torch.stack(pos_emb_u)
+                pos_emb_v = torch.stack(pos_emb_v)
+                pos_score = torch.cosine_similarity(pos_emb_u, pos_emb_v)
+                pos_score = F.logsigmoid(pos_score).mean()
+            if len(neg_emb_u) > 0 and len(neg_emb_v) > 0:
+                neg_emb_u = torch.stack(neg_emb_u)
+                neg_emb_v = torch.stack(neg_emb_v)
+                neg_score = torch.cosine_similarity(neg_emb_u, neg_emb_v)
+                neg_score = F.logsigmoid(-neg_score).mean()
+            return -pos_score - neg_score
+
+        def train():
+            embedder.train()
+            action_strs = self._select_transitions_for_training(size=N_ACTIONS_TRAINING)
+            state_encs, action_pairs = self.encode_action_pairs(action_strs)
+            ele_embed = embedder.forward(state_encs)
+            loss = compute_loss(ele_embed, action_pairs)
+            optimizer.zero_grad()
+            loss.backward()
+            optimizer.step()
+            return loss.item(), len(action_pairs)
+
+        for i in range(n_iterations):
+            epoch_start_time = time.time()
+            loss, n_pairs = train()
+            elapsed = time.time() - epoch_start_time
+            print(f'| iter: {i:3d} | time: {elapsed:6.2f}s | #pairs: {n_pairs:6d} | loss: {loss:8.4f}')
+
+        with torch.no_grad():
+            embedder.eval()
+            state_encs = [v['state_enc'] for k,v in self.known_states.items()]
+            ele_embed = embedder(state_encs)
+            ele_embed = ele_embed.detach().cpu()
+            for i, (k,v) in enumerate(self.known_states.items()):
+                self.known_states[k]['views_emb'] = ele_embed[i]
+
+    def get_unexplored_actions(self, current_state):
+        action_strs = set()
+        structure_strs = set()
+        self._memorize_state(current_state)
+        for state_str, state_info in reversed(self.known_states.items()):
+            state = state_info['state']
+            if state.structure_str in structure_strs:
+                continue
+            structure_strs.add(state.structure_str)
+            for action in state.get_possible_input():
+                if not isinstance(action, BaseTouchEvent):
+                    continue
+                action_str = action.get_event_str(state=state)
+                if action_str in action_strs:
+                    continue
+                if self.utg.is_event_explored(action, state):
+                    continue
+                action_strs.add(action_str)
+                yield state, action
+
+    def get_action_emb(self, state, action):
+        state_str = state.state_str
+        if state_str not in self.known_states:
+            return None
+        view_str = action.view['view_str']
+        if view_str not in self.known_states[state_str]['views_str']:
+            return None
+        view_idx = self.known_states[state_str]['views_str'].index(view_str)
+        return self.known_states[state_str]['views_emb'][view_idx]
+
+
+# ==================== Input Policy Classes ====================
+
+class InputPolicy(object):
+    """Base class for input policies."""
+    def __init__(self, device,enable_guiagent=False):
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.device = device
+        self.action_count = 0
+        self.enable_guiagent = enable_guiagent
+        
+
+    def start(self, input_manager):
+        """Start producing events."""
+        self.input_manager = input_manager  # Save reference for subclasses
+        
+        # 传递input_manager引用到guiagent_bridge（如果存在）
+        if hasattr(self, 'guiagent_bridge') and self.guiagent_bridge:
+            self.guiagent_bridge.input_manager = input_manager
+        
+        self.action_count = 0
+        
+        # === 启动序列 (循环外，只执行一次) ===
+        # 1. 杀掉旧进程
+        KillAppEvent = get_event_class(self.device, 'kill_app')
+        kill_event = KillAppEvent(app=self.device.app_identifier)
+        input_manager.add_event(kill_event)
+        self.action_count += 1
+        
+        # 2. 启动应用
+        self.device.start_app()
+        
+        # 3. cv模式 执行初始化任务 (登录/进入游戏)
+        if getattr(self.device, 'cv_mode', False) and self.enable_guiagent:
+            if hasattr(self.device, 'run_initial_setup'):
+                self.device.run_initial_setup()
+        
+        # === 正常探索循环 ===
+        while input_manager.enabled and self.action_count < input_manager.event_count:
+            try:
+                event = self.generate_event()
+                
+                if event is not None:
+                    input_manager.add_event(event)
+                self.action_count += 1
+            except InputInterruptedException:
+                self.logger.info("InputInterruptedException caught, stopping event generation.")
+                break
+            except FATAL_EXCEPTIONS:
+                raise
+            except Exception as e:
+                self.logger.error(f'Non-fatal exception in event loop: {e}')
+                import traceback
+                traceback.print_exc()
+                continue
+
+    def generate_event(self):
+        """Generate an event."""
+        raise NotImplementedError
+
+
+class NoneInputPolicy(InputPolicy):
+    """Do not send any event - for manual testing."""
+    def __init__(self, device, enable_guiagent=False):
+        super(NoneInputPolicy, self).__init__(device)
+        self.enable_guiagent = enable_guiagent
+
+    def generate_event(self):
+        """Generate an event."""
+        import time
+        time.sleep(2)
+        return None
+
+
+class ManualPolicy(InputPolicy):
+    """
+    Manually control the device.
+    """
+    def __init__(self, device, enable_guiagent=False):
+        super(ManualPolicy, self).__init__(device)
+        self.enable_guiagent = enable_guiagent
+
+    def generate_event(self):
+        """
+        No event is generated in manual policy
+        """
+        return None
+
+
+class UtgBasedInputPolicy(InputPolicy):
+    """State-based input policy using UTG."""
+    def __init__(self, device, random_input, enable_guiagent=False, app_name=None, traffic_monitor=None):
+        super(UtgBasedInputPolicy, self).__init__(device,enable_guiagent=enable_guiagent)
+        self.random_input = random_input
+        self.enable_guiagent = enable_guiagent
+        self.app_name = app_name
+        self.traffic_monitor = traffic_monitor  # 新增：流量监控引用
+        self.last_event = None
+        self.last_state = None
+        self.current_state = None
+        self.utg = UTG(device=device, random_input=random_input)
+        
+        # Initialize GuiAgent bridge
+        self.guiagent_bridge = None
+        if self.enable_guiagent:
+            self.guiagent_bridge = GuiAgentBridge(device, app_name=self.app_name, utg=self.utg)
+            self.logger.info("GuiAgent bridge initialized")
+        
+
+    def generate_event(self):
+        """Generate an event."""
+        # 复用上一步 EventLog.stop() 缓存的状态，避免重复调用 get_current_state
+        if hasattr(self.device, '_last_state') and self.device._last_state is not None:
+            self.current_state = self.device._last_state
+        else:
+            print("Warning: No last state available, using current state as start state.")
+            self.current_state = self.device.get_current_state()
+            # 首次调用时设置 _last_state，供 EventLog.start() 复用
+            if hasattr(self.device, '_last_state'):
+                self.device._last_state = self.current_state
+        if self.current_state is None:
+            import time
+            time.sleep(5)
+            self.logger.warning("Current state is None, waiting for 5 seconds")
+            KeyEvent = get_event_class(self.device, 'key')
+            return KeyEvent(key_name="BACK")
+
+        self.__update_utg()
+
+        event = self.generate_event_based_on_utg()
+        self.last_event = event
+        self.last_state = self.current_state
+        return event
+
+    def __update_utg(self):
+        # 在调用 UTG 方法之前，一次性获取新增域名
+        # 前10步不检测域名，等待流量文件生成，避免获取上一次测试的流量
+        new_domains_count = 0
+        new_domains_list = []
+        EARLY_PHASE_STEPS = 10
+        if self.traffic_monitor and self.action_count > EARLY_PHASE_STEPS:
+            new_domains_count, new_domains_list = self.traffic_monitor.get_new_domains_since_last_step()
+            if new_domains_count > 0:
+                self.logger.info(f"Step found {new_domains_count} new domains")
+        
+        # 将域名信息传递给 UTG
+        self.utg.add_transition(self.last_event, self.last_state, self.current_state, new_domains=new_domains_list)
+
+    def generate_event_based_on_utg(self):
+        """Generate an event based on UTG - to be overridden."""
+        raise NotImplementedError
+
+
+class MemoryGuidedPolicy(UtgBasedInputPolicy):
+    """Memory-guided exploration policy with neural network learning."""
+
+    def __init__(self, device, random_input, enable_guiagent=False, app_name=None, traffic_monitor=None):
+        super(MemoryGuidedPolicy, self).__init__(device, random_input, enable_guiagent=enable_guiagent, app_name=app_name, traffic_monitor=traffic_monitor)
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.memory = Memory(utg=self.utg, device=device)
+        self.num_actions_train = 10
+        self._nav_steps = []
+
+        self.last_login_register_step = -100  # Initialize to allow immediate trigger
+        self.guiagent_message = None  # 记录登录/注册消息（成功或失败），用于worker.report汇报
+        self.login_count = 0  # 记录进入登录场景的次数
+        self.register_count = 0  # 记录进入注册场景的次数
+        self.stuck_reason_code = 0  # 记录卡住原因代码，用于worker.report汇报
+
+        # 新状态停滞检测变量
+        self._last_state_count = 0
+        self._no_new_state_steps = 0
+        self.NO_NEW_STATE_THRESHOLD = 50
+        self.NO_NEW_STATE_STEPS_THRESHOLD = 300
+
+    def generate_event_based_on_utg(self):
+        """Generate an event based on current UTG."""
+        current_state = self.current_state
+        try:
+            self.memory.save_transition(self.last_event, self.last_state, current_state)
+        except FATAL_EXCEPTIONS:
+            raise
+        except Exception as e:
+            self.logger.error(f'failed to save transition: {e}')
+            import traceback
+            traceback.print_exc()
+            # 非致命异常，不抛出
+        if self.action_count % self.num_actions_train == 0:
+            self.memory.train_model()
+
+        self.logger.debug("Current state: %s" % current_state.state_str)
+
+        # 检测连续无新增状态
+        total_steps = self.input_manager.total_exploring_steps
+        
+        if self._check_no_new_state_stuck():
+            self.logger.warning(f"连续 {self._no_new_state_steps} 步无新增状态（总步数: {total_steps}）")
+            
+            # 在阈值以内，尝试智能体介入
+            if self.enable_guiagent and total_steps < self.NO_NEW_STATE_STEPS_THRESHOLD:
+                self.logger.info(f"触发 GuiAgent 处理探索停滞")
+                success, message, reason_code = self.guiagent_bridge.handle_with_guiagent(
+                    category="explore_stuck",
+                    context={"state": current_state, "steps": self._no_new_state_steps, "total_steps": total_steps}
+                )
+                if message:
+                    if self.guiagent_message:
+                        self.guiagent_message += f"; {message}"
+                    else:
+                        self.guiagent_message = message
+                
+                if reason_code:
+                    self.stuck_reason_code = reason_code
+                
+                if message and ("失败" in message):
+                    self.logger.error(f"GuiAgent 判断探索无法继续: {message}")
+                    raise ExplorationStuckException(f"{message}")
+                elif message and ("成功" in message):
+                    self._no_new_state_steps = 0
+                    self._last_state_count = sum(
+                        1 for state_str in self.utg.G.nodes()
+                        if self.utg.G.nodes[state_str]["state"].foreground_page
+                        and self.utg.G.nodes[state_str]["state"].foreground_page.startswith(self.device.app_identifier)
+                    )
+                    self.logger.info(f"重置卡住检测计数器，继续探索")
+                    return None
+
+            else:
+                # 超过阈值，直接抛出异常
+                self.logger.error(f"总步数 {total_steps} 超过阈值 {self.NO_NEW_STATE_STEPS_THRESHOLD}，探索停滞")
+                raise InputInterruptedException(f"总步数 {total_steps} 超过阈值 {self.NO_NEW_STATE_STEPS_THRESHOLD}，探索停滞")
+
+        nav_action = self.navigate(current_state)
+        if nav_action:
+            return nav_action
+        self._nav_steps = []
+
+        self.memory.save_structure(current_state)
+
+        if self.action_count >= self.num_actions_train \
+                and len(self._nav_steps) == 0 \
+                and np.random.uniform() > RANDOM_EXPLORE_PROB:
+            (target_state, target_action), candidates = self.pick_target(current_state)
+            if target_state:
+                if target_state.state_str == current_state.state_str:
+                    self.logger.info(f"executing action selected from {len(candidates)} candidates")
+                    
+                    # Check GuiAgent keywords before returning
+                    if self.enable_guiagent :
+                        if hasattr(target_action, 'view') and target_action.view is not None:
+                            view = target_action.view
+                            view_text = self.guiagent_bridge.get_text_within_bounds(view, current_state.views)
+                            if view_text:
+                                is_editable = view.get('editable', False)
+                                if is_editable:
+                                    self.logger.info(f"即将操作的控件是可输入的: {view_text}. 执行自定义输入序列。")
+                                    SetTextEvent = get_event_class(self.device, 'set_text')
+                                    return SetTextEvent(text='tp', view=view)
+                                category = self.guiagent_bridge.check_keywords(view_text)
+                                if category:
+                                    # Special handling for login/register cooldown
+                                    if category in ["login", "register"]:
+                                        current_step = self.input_manager.total_exploring_steps if hasattr(self, 'input_manager') else 0
+                                        if current_step - self.last_login_register_step < 50:
+                                            self.logger.info(f"Skipping {category} due to cooldown (last: {self.last_login_register_step}, current: {current_step})")
+                                            return target_action
+                                        else:
+                                            self.last_login_register_step = current_step
+
+                                    self.logger.info(f"即将操作的控件包含关键词: {view_text}, 类别: {category}")
+                                    if category == 'login':
+                                        self.login_count += 1
+                                        self.logger.info(f"进入登录场景，累计次数: {self.login_count}")
+                                    elif category == 'register':
+                                        self.register_count += 1
+                                        self.logger.info(f"进入注册场景，累计次数: {self.register_count}")
+                                    success, guiagent_message, _ = self.guiagent_bridge.handle_with_guiagent(category, {"view": view, "state": current_state})
+                                    if guiagent_message:
+                                        if self.guiagent_message:
+                                            self.guiagent_message += f"; {guiagent_message}"
+                                        else:
+                                            self.guiagent_message = guiagent_message
+                                    if success:
+                                        new_state = self.device.get_current_state()
+                                        if new_state and new_state.state_str != current_state.state_str:
+                                            self.logger.info("GuiAgent处理成功，状态已改变")
+                                            return None
+                                    else:
+                                        self.logger.warning("GuiAgent处理失败，继续使用DroidBot策略执行原操作")
+                    
+                    return target_action
+                self._nav_steps = self.get_shortest_nav_steps(current_state, target_state, target_action)
+                nav_action = self.navigate(current_state)
+                if nav_action:
+                    return nav_action
+        self._nav_steps = []
+        
+        self.logger.info("trying random action")
+        possible_events = current_state.get_possible_input()
+        # self.logger.info(possible_events)
+        random.shuffle(possible_events)
+        
+        # 当系统界面等无可交互元素时，possible_events 可能为空
+        if not possible_events:
+            self.logger.warning("possible_events 为空，执行 BACK 返回")
+            KeyEvent = get_event_class(self.device, 'key')
+            return KeyEvent(key_name="BACK")
+        
+        selected_event = possible_events[0]
+        if self.enable_guiagent:
+            if hasattr(selected_event, 'view') and selected_event.view is not None:
+                view = selected_event.view
+                view_text = self.guiagent_bridge.get_text_within_bounds(view, current_state.views)
+                if view_text:
+                    is_editable = view.get('editable', False)
+                    if is_editable:
+                        self.logger.info(f"即将操作的控件是可输入的: {view_text}. 执行自定义输入序列。")
+                        SetTextEvent = get_event_class(self.device, 'set_text')
+                        return SetTextEvent(text='test', view=view)
+                    category = self.guiagent_bridge.check_keywords(view_text)
+                    if category:
+                        # Special handling for login/register cooldown
+                        if category in ["login", "register"]:
+                            current_step = self.input_manager.total_exploring_steps if hasattr(self, 'input_manager') else 0
+                            if current_step - self.last_login_register_step < 100:
+                                self.logger.info(f"Skipping {category} due to cooldown (last: {self.last_login_register_step}, current: {current_step})")
+                                return selected_event
+                            else:
+                                self.last_login_register_step = current_step
+
+                        self.logger.info(f"即将操作的控件包含关键词: {view_text}, 类别: {category}")
+                        if category == 'login':
+                            self.login_count += 1
+                            self.logger.info(f"进入登录场景，累计次数: {self.login_count}")
+                        elif category == 'register':
+                            self.register_count += 1
+                            self.logger.info(f"进入注册场景，累计次数: {self.register_count}")
+                        success, guiagent_message, _ = self.guiagent_bridge.handle_with_guiagent(category, {"view": view, "state": current_state})
+                        if guiagent_message:
+                            if self.guiagent_message:
+                                self.guiagent_message += f"; {guiagent_message}"
+                            else:
+                                self.guiagent_message = guiagent_message
+                        if success:
+                            new_state = self.device.get_current_state()
+                            if new_state and new_state.state_str != current_state.state_str:
+                                self.logger.info("GuiAgent处理成功，状态已改变")
+                                return None
+                        else:
+                            self.logger.warning("GuiAgent处理失败，继续使用DroidBot策略执行原操作")
+        
+        return selected_event
+
+    def pick_target(self, current_state):
+        state_action_pairs = list(self.memory.get_unexplored_actions(current_state))
+        best_target = None, None
+        best_score = -np.inf
+        known_actions_emb = self.memory.get_known_actions_emb()
+        if known_actions_emb is None:
+            return best_target, state_action_pairs
+        scores = []
+        for i, (state, action) in enumerate(state_action_pairs):
+            action_emb = self.memory.get_action_emb(state, action)
+            similarities = torch.cosine_similarity(action_emb.repeat((known_actions_emb.size(0), 1)), known_actions_emb)
+            max_sim, max_sim_idx = similarities.max(0)
+            score = -max_sim
+            if state.state_str == current_state.state_str:
+                score += CLOSER_ACTION_ENCOURAGEMENT
+            if DEBUG:
+                action_info_str = f'{state.foreground_page}-{action.view.get("signature", "")}'
+                scores.append((i, score, action_info_str, similarities, action_emb))
+            if score > best_score:
+                best_score = score
+                best_target = state, action
+        return best_target, state_action_pairs
+
+    def _check_no_new_state_stuck(self):
+        """
+        检测是否连续多步无新增状态
+        只统计属于当前应用的状态（foreground_page 以 app_identifier 开头）
+        :return: True 如果连续 NO_NEW_STATE_THRESHOLD 步无新增状态
+        """
+        app_identifier = self.device.app_identifier
+        current_state_count = sum(
+            1 for state_str in self.utg.G.nodes()
+            if self.utg.G.nodes[state_str]["state"].foreground_page
+            and self.utg.G.nodes[state_str]["state"].foreground_page.startswith(app_identifier)
+        )
+        
+        if current_state_count > self._last_state_count:
+            self._last_state_count = current_state_count
+            self._no_new_state_steps = 0
+            self.logger.debug(f"发现新状态，应用内状态总数: {current_state_count}，重置计数器")
+        else:
+            self._no_new_state_steps += 1
+            self.logger.debug(f"无新增状态，连续步数: {self._no_new_state_steps}/{self.NO_NEW_STATE_THRESHOLD}")
+        
+        return self._no_new_state_steps >= self.NO_NEW_STATE_THRESHOLD
+
+    def navigate(self, current_state):
+        if self._nav_steps and len(self._nav_steps) > 0:
+            nav_state, nav_action = self._nav_steps[0]
+            self._nav_steps = self._nav_steps[1:]
+            nav_action_ = self._get_nav_action(current_state, nav_state, nav_action)
+            if nav_action_:
+                self.logger.info(f"navigating, {len(self._nav_steps)} steps left")
+                return nav_action_
+            else:
+                self.logger.warning("navigation failed")
+                self.utg.remove_transition(self.last_event, self.last_state, nav_state)
+
+    def _get_nav_action(self, current_state, nav_state, nav_action):
+        try:
+            if current_state.structure_str != nav_state.structure_str:
+                return None
+            if not isinstance(nav_action, BaseTouchEvent):
+                return nav_action
+            if nav_action.__class__.__name__ == 'ScrollEvent':
+                return copy.deepcopy(nav_action)
+            nav_view = nav_action.view
+            nav_view_idx = nav_state.views.index(nav_view)
+            new_view = current_state.views[nav_view_idx]
+            new_action = copy.deepcopy(nav_action)
+            new_action.view = new_view
+            return new_action
+        except Exception as e:
+            self.logger.error(f'exception during _get_nav_action: {e}')
+            return nav_action
+
+
+    def get_shortest_nav_steps(self, current_state, target_state, target_action):
+        normal_nav_steps = self.utg.get_G2_nav_steps(current_state, target_state)
+        restart_nav_steps = self.utg.get_G2_nav_steps(self.utg.first_state, target_state)
+        normal_nav_steps_len = len(normal_nav_steps) if normal_nav_steps else MAX_NAV_STEPS
+        restart_nav_steps_len = len(restart_nav_steps) + 1 if restart_nav_steps else MAX_NAV_STEPS
+        if normal_nav_steps_len >= MAX_NAV_STEPS and restart_nav_steps_len >= MAX_NAV_STEPS:
+            self.logger.warning(f'cannot find a path to {target_state.structure_str} {target_state.foreground_page}')
+            target_state_str = target_state.state_str
+            self.memory.known_states.pop(target_state_str, None)
+            action_strs_to_remove = []
+            for action_str in self.memory.known_transitions:
+                action_from_state = self.memory.known_transitions[action_str]['from_state']
+                action_to_state = self.memory.known_transitions[action_str]['to_state']
+                if action_from_state.state_str == target_state_str or action_to_state.state_str == target_state_str:
+                    action_strs_to_remove.append(action_str)
+            for action_str in action_strs_to_remove:
+                self.memory.known_transitions.pop(action_str, None)
+            return None
+        elif normal_nav_steps_len >= MAX_NAV_STEPS:
+            return None
+        else:
+            nav_steps = normal_nav_steps
+        return nav_steps + [(target_state, target_action)]

DroidBot/platforms/__init__.py

@@ -0,0 +1,41 @@
+# Platforms module
+# Contains platform-specific implementations
+
+# Import Android platform (always available)
+try:
+    from .android import AndroidDevice, AndroidDeviceState
+    _android_available = True
+except ImportError:
+    _android_available = False
+
+# Import iOS platform (optional)
+try:
+    from .ios import IOSDevice, IOSDeviceState
+    _ios_available = True
+except ImportError:
+    _ios_available = False
+
+# Import Web platform (optional)
+try:
+    from .web import WebDevice, WebDeviceState, WebApp
+    _web_available = True
+except ImportError:
+    _web_available = False
+
+# Import Windows platform (optional)
+try:
+    from .windows import WindowsDevice, WindowsDeviceState
+    _windows_available = True
+except ImportError:
+    _windows_available = False
+
+# Build __all__ dynamically based on available platforms
+__all__ = []
+if _android_available:
+    __all__.extend(['AndroidDevice', 'AndroidDeviceState'])
+if _ios_available:
+    __all__.extend(['IOSDevice', 'IOSDeviceState'])
+if _web_available:
+    __all__.extend(['WebDevice', 'WebDeviceState', 'WebApp'])
+if _windows_available:
+    __all__.extend(['WindowsDevice', 'WindowsDeviceState'])

DroidBot/platforms/android/__init__.py

@@ -0,0 +1,72 @@
+# Android platform implementation
+from .android_device import AndroidDevice
+from .android_device_state import AndroidDeviceState
+from .android_app import AndroidApp
+from .android_intent import AndroidIntent
+from .android_input_event import (
+    AndroidTouchEvent,
+    AndroidLongTouchEvent,
+    AndroidSwipeEvent,
+    AndroidScrollEvent,
+    AndroidSetTextEvent,
+    AndroidKeyEvent,
+    AndroidIntentEvent,
+    AndroidKillAppEvent,
+)
+
+__all__ = [
+    'AndroidDevice',
+    'AndroidDeviceState',
+    'AndroidApp',
+    'AndroidIntent',
+    'AndroidTouchEvent',
+    'AndroidLongTouchEvent',
+    'AndroidSwipeEvent',
+    'AndroidScrollEvent',
+    'AndroidSetTextEvent',
+    'AndroidKeyEvent',
+    'AndroidIntentEvent',
+    'AndroidKillAppEvent',
+]
+
+# Register Android platform with the factory
+def register_android_platform():
+    """Register Android platform with PlatformFactory"""
+    from ...core.platform_factory import PlatformFactory, Platform
+    from .android_device import AndroidDevice
+    from .android_device_state import AndroidDeviceState
+    from .android_input_event import (
+        AndroidTouchEvent,
+        AndroidLongTouchEvent,
+        AndroidSwipeEvent,
+        AndroidScrollEvent,
+        AndroidSetTextEvent,
+        AndroidKeyEvent,
+        AndroidIntentEvent,
+        AndroidKillAppEvent,
+    )
+    
+    event_classes = {
+        'touch': AndroidTouchEvent,
+        'long_touch': AndroidLongTouchEvent,
+        'swipe': AndroidSwipeEvent,
+        'scroll': AndroidScrollEvent,
+        'set_text': AndroidSetTextEvent,
+        'key': AndroidKeyEvent,
+        'intent': AndroidIntentEvent,
+        'kill_app': AndroidKillAppEvent,
+    }
+    
+    PlatformFactory.register_platform(
+        Platform.ANDROID,
+        AndroidDevice,
+        AndroidDeviceState,
+        event_classes
+    )
+
+# Auto-register on import
+try:
+    register_android_platform()
+except ImportError:
+    # Platform classes may not be fully initialized yet
+    pass

DroidBot/platforms/android/adapters/__init__.py

@@ -0,0 +1,33 @@
+# Android adapters module
+# Re-export adapters from the original location for backward compatibility
+
+# Import adapters from original location
+from .adb import ADB
+from ....exceptions import ADBException
+from .droidbot_app import DroidBotAppConn
+from .logcat import Logcat
+from .minicap import Minicap
+from .process_monitor import ProcessMonitor
+from .telnet import TelnetConsole
+from .logcat import Logcat
+from .user_input_monitor import UserInputMonitor
+from .droidbot_ime import DroidBotIme
+
+# CV module
+try:
+    from DroidBot.cv import cv
+except ImportError:
+    cv = None
+
+__all__ = [
+    'ADB',
+    'ADBException', 
+    'DroidBotAppConn',
+    'Logcat',
+    'Minicap',
+    'ProcessMonitor',
+    'TelnetConsole',
+    'UserInputMonitor',
+    'DroidBotIme',
+    'cv',
+]

DroidBot/platforms/android/adapters/adb.py

@@ -0,0 +1,415 @@
+# This is the interface for adb
+import subprocess
+import logging
+import re
+import time
+try:
+    from shlex import quote # Python 3
+except ImportError:
+    from pipes import quote # Python 2
+
+from ....exceptions import ADBException
+
+
+class ADB(object):
+    """
+    interface of ADB
+    send adb commands via this, see:
+    http://developer.android.com/tools/help/adb.html
+    """
+
+    DOWN_AND_UP = 2
+    MODEL_PROPERTY = "ro.product.model"
+    VERSION_SDK_PROPERTY = 'ro.build.version.sdk'
+    VERSION_RELEASE_PROPERTY = 'ro.build.version.release'
+
+    def __init__(self, device=None):
+        """
+        initiate a ADB connection from serial no
+        the serial no should be in output of `adb devices`
+        :param device: instance of Device
+        :return:
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        if device is None:
+            from DroidBot.platforms.android import AndroidDevice as Device
+            device = Device()
+        self.device = device
+
+        self.cmd_prefix = ['adb', "-s", device.serial]
+        
+        self._display_info_cache = None
+        self._display_info_cache_time = 0
+        self._sdk_version_cache = None
+        self.last_error = None
+
+    def _check_adb_connection(self):
+        """检查 adb 连接状态"""
+        try:
+            result = subprocess.check_output(
+                ['adb', '-s', self.device.serial, 'get-state'], 
+                stderr=subprocess.STDOUT, 
+                timeout=5
+            ).strip()
+            if isinstance(result, bytes):
+                result = result.decode()
+            return result.startswith('device')
+        except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as e:
+            return False
+
+    def run_cmd(self, extra_args):
+        """
+        run an adb command and return the output
+        :return: output of adb command
+        @param extra_args: arguments to run in adb
+        """
+        if isinstance(extra_args, str) or isinstance(extra_args, str):
+            extra_args = extra_args.split()
+        if not isinstance(extra_args, list):
+            msg = "invalid arguments: %s\nshould be list or str, %s given" % (extra_args, type(extra_args))
+            self.logger.warning(msg)
+            raise ADBException(msg)
+
+        # 执行命令前检查连接状态
+        if not self._check_adb_connection():
+            error_msg = f"ADB device {self.device.serial} is not connected"
+            self.logger.error(error_msg)
+            raise ADBException(error_msg)
+
+        args = [] + self.cmd_prefix
+        args += extra_args
+
+        self.logger.debug('command:')
+        self.logger.debug(args)
+        
+        try:
+            r = subprocess.check_output(args, stderr=subprocess.STDOUT, timeout=30).strip()
+        except subprocess.CalledProcessError as e:
+            error_msg = f"ADB command failed: {' '.join(args)}\nOutput: {e.output.decode('utf-8') if e.output else ''}"
+            self.logger.error(error_msg)
+            self.last_error = error_msg
+            return ""
+        except subprocess.TimeoutExpired as e:
+            error_msg = f"ADB command timeout: {' '.join(args)}"
+            self.logger.error(error_msg)
+            self.last_error = error_msg
+            raise ADBException(error_msg)
+            
+        if not isinstance(r, str):
+            r = r.decode()
+        self.logger.debug('return:')
+        self.logger.debug(r)
+        return r
+
+    def shell(self, extra_args):
+        """
+        run an `adb shell` command
+        @param extra_args:
+        @return: output of adb shell command
+        """
+        if isinstance(extra_args, str) or isinstance(extra_args, str):
+            extra_args = extra_args.split()
+        if not isinstance(extra_args, list):
+            msg = "invalid arguments: %s\nshould be list or str, %s given" % (extra_args, type(extra_args))
+            self.logger.warning(msg)
+            raise ADBException(msg)
+
+        shell_extra_args = ['shell'] + [ quote(arg) for arg in extra_args ]
+        return self.run_cmd(shell_extra_args)
+    
+    def shell_grep(self, cmd):
+        """执行真实的 ADB 命令"""
+        # 执行命令前检查连接状态
+        if not self._check_adb_connection():
+            error_msg = f"ADB device {self.device.serial} is not connected"
+            self.logger.error(error_msg)
+            raise ADBException(error_msg)
+        
+        try:
+            # 这里的 shell=True 是为了支持管道符 | grep
+            full_cmd = f"adb -s {self.device.serial} shell \"{cmd}\""
+            result = subprocess.check_output(full_cmd, shell=True, stderr=subprocess.STDOUT, timeout=30)
+            return result.decode('utf-8')
+        except subprocess.TimeoutExpired as e:
+            error_msg = f"ADB shell_grep command timeout: {full_cmd}"
+            self.logger.error(error_msg)
+            self.last_error = error_msg
+            raise ADBException(error_msg)
+        except subprocess.CalledProcessError as e:
+            if e.returncode == 1:
+                # grep 没有匹配到任何内容，返回空字符串
+                return ""
+            error_msg = f"ADB shell_grep failed: {cmd}\nOutput: {e.output.decode('utf-8') if e.output else ''}"
+            self.logger.error(error_msg)
+            self.last_error = error_msg
+            raise ADBException(error_msg)
+        except Exception as e:
+            error_msg = f"ADB shell_grep unexpected error: {e}"
+            self.logger.error(error_msg)
+            self.last_error = error_msg
+            raise ADBException(error_msg)
+        
+    def check_connectivity(self):
+        """
+        check if adb is connected
+        :return: True for connected
+        """
+        r = self.run_cmd("get-state")
+        return r.startswith("device")
+
+    def connect(self):
+        """
+        connect adb
+        """
+        self.logger.debug("connected")
+
+    def disconnect(self):
+        """
+        disconnect adb
+        """
+        print("[CONNECTION] %s is disconnected" % self.__class__.__name__)
+
+    def get_property(self, property_name):
+        """
+        get the value of property
+        @param property_name:
+        @return:
+        """
+        return self.shell(["getprop", property_name])
+
+    def get_model_number(self):
+        """
+        Get device model number. e.g. SM-G935F
+        """
+        return self.get_property(ADB.MODEL_PROPERTY)
+
+    def get_sdk_version(self):
+        """
+        Get version of SDK, e.g. 18, 20
+        """
+        if self._sdk_version_cache is not None:
+            return self._sdk_version_cache
+        self._sdk_version_cache = int(self.get_property(ADB.VERSION_SDK_PROPERTY))
+        return self._sdk_version_cache
+
+    def get_release_version(self):
+        """
+        Get release version, e.g. 4.3, 6.0
+        """
+        return self.get_property(ADB.VERSION_RELEASE_PROPERTY)
+
+
+    # The following methods are originally from androidviewclient project.
+    # https://github.com/dtmilano/AndroidViewClient.
+    def get_display_info(self, use_cache=True, cache_ttl=2.0):
+        """
+        Gets C{mDefaultViewport} and then C{deviceWidth} and C{deviceHeight} values from dumpsys.
+        This is a method to obtain display dimensions and density
+        @param use_cache: whether to use cached display info
+        @param cache_ttl: cache time-to-live in seconds (default 2.0s)
+        """
+        current_time = time.time()
+        if use_cache and self._display_info_cache is not None:
+            if current_time - self._display_info_cache_time < cache_ttl:
+                return self._display_info_cache
+        
+        display_info = {}
+        logical_display_re = re.compile(".*DisplayViewport{valid=true, .*orientation=(?P<orientation>\d+),"
+                                        " .*deviceWidth=(?P<width>\d+), deviceHeight=(?P<height>\d+).*")
+        dumpsys_display_result = self.shell("dumpsys display")
+        if dumpsys_display_result is not None:
+            for line in dumpsys_display_result.splitlines():
+                m = logical_display_re.search(line, 0)
+                if m:
+                    for prop in ['width', 'height', 'orientation']:
+                        display_info[prop] = int(m.group(prop))
+
+        if 'width' not in display_info or 'height' not in display_info:
+            physical_display_re = re.compile('Physical size: (?P<width>\d+)x(?P<height>\d+)')
+            m = physical_display_re.search(self.shell('wm size'))
+            if m:
+                for prop in ['width', 'height']:
+                    display_info[prop] = int(m.group(prop))
+
+        if 'width' not in display_info or 'height' not in display_info:
+            # This could also be mSystem or mOverscanScreen
+            display_re = re.compile('\s*mUnrestrictedScreen=\((?P<x>\d+),(?P<y>\d+)\) (?P<width>\d+)x(?P<height>\d+)')
+            # This is known to work on older versions (i.e. API 10) where mrestrictedScreen is not available
+            display_width_height_re = re.compile('\s*DisplayWidth=(?P<width>\d+) *DisplayHeight=(?P<height>\d+)')
+            for line in self.shell('dumpsys window').splitlines():
+                m = display_re.search(line, 0)
+                if not m:
+                    m = display_width_height_re.search(line, 0)
+                if m:
+                    for prop in ['width', 'height']:
+                        display_info[prop] = int(m.group(prop))
+
+        if 'orientation' not in display_info:
+            surface_orientation_re = re.compile("SurfaceOrientation:\s+(\d+)")
+            output = self.shell("dumpsys input")
+            m = surface_orientation_re.search(output)
+            if m:
+                display_info['orientation'] = int(m.group(1))
+
+        density = None
+        float_re = re.compile(r"[-+]?\d*\.\d+|\d+")
+        d = self.get_property('ro.sf.lcd_density')
+        if float_re.match(d):
+            density = float(d)
+        else:
+            d = self.get_property('qemu.sf.lcd_density')
+            if float_re.match(d):
+                density = float(d)
+            else:
+                physical_density_re = re.compile('Physical density: (?P<density>[\d.]+)', re.MULTILINE)
+                m = physical_density_re.search(self.shell('wm density'))
+                if m:
+                    density = float(m.group('density'))
+        if density is not None:
+            display_info['density'] = density
+
+        display_info_keys = {'width', 'height', 'orientation', 'density'}
+        if not display_info_keys.issuperset(display_info):
+            self.logger.warning("getDisplayInfo failed to get: %s" % display_info_keys)
+
+        self._display_info_cache = display_info
+        self._display_info_cache_time = current_time
+        return display_info
+
+    def get_enabled_accessibility_services(self):
+        """
+        Get enabled accessibility services
+        :return: the enabled service names, each service name is in <package_name>/<service_name> format
+        """
+        r = self.shell("settings get secure enabled_accessibility_services")
+        r = re.sub(r'(?m)^WARNING:.*\n?', '', r)
+        return r.strip().split(":") if r.strip() != '' else []
+
+    def enable_accessibility_service(self, service_name):
+        """
+        Enable an accessibility service
+        :param service_name: the service to enable, in <package_name>/<service_name> format
+        """
+        service_names = self.get_enabled_accessibility_services()
+        if service_name not in service_names:
+            service_names.append(service_name)
+            self.shell("settings put secure enabled_accessibility_services %s" % ":".join(service_names))
+        self.shell("settings put secure accessibility_enabled 1")
+
+    def enable_accessibility_service_db(self, service_name):
+        """
+        Enable an accessibility service
+        :param service_name: the service to enable, in <package_name>/<service_name> format
+        """
+        subprocess.check_call(
+            "adb shell \""
+            "sqlite3 -batch /data/data/com.android.providers.settings/databases/settings.db \\\""
+            "DELETE FROM secure WHERE name='enabled_accessibility_services' OR name='accessibility_enabled' "
+            "OR name='touch_exploration_granted_accessibility_services' OR name='touch_exploration_enabled';"
+            "INSERT INTO secure (name, value) VALUES "
+            "('enabled_accessibility_services','" + service_name + "'), "
+            "('accessibility_enabled','1'), "
+            "('touch_exploration_granted_accessibility_services','" + service_name + "'), "
+            "('touch_exploration_enabled','1')\\\";\"", shell=True)
+        self.shell("stop")
+        time.sleep(1)
+        self.shell("start")
+
+    def get_installed_apps(self):
+        """
+        Get the package names and apk paths of installed apps on the device
+        :return: a dict, each key is a package name of an app and each value is the file path to the apk
+        """
+        app_lines = self.shell("pm list packages -f").splitlines()
+        app_line_re = re.compile("package:(?P<apk_path>.+)=(?P<package>[^=]+)")
+        package_to_path = {}
+        for app_line in app_lines:
+            m = app_line_re.match(app_line)
+            if m:
+                package_to_path[m.group('package')] = m.group('apk_path')
+        return package_to_path
+
+
+
+    def __transform_point_by_orientation(self, xy, orientation_orig, orientation_dest, display_info=None):
+        (x, y) = xy
+        if orientation_orig != orientation_dest:
+            if display_info is None:
+                display_info = self.get_display_info()
+            if orientation_dest == 1:
+                _x = x
+                x = display_info['width'] - y
+                y = _x
+            elif orientation_dest == 3:
+                _x = x
+                x = y
+                y = display_info['height'] - _x
+        return x, y
+
+    def get_orientation(self):
+        display_info = self.get_display_info()
+        if 'orientation' in display_info:
+            return display_info['orientation']
+        else:
+            return -1
+
+    def unlock(self):
+        """
+        Unlock the screen of the device
+        """
+        self.shell("input keyevent MENU")
+        self.shell("input keyevent BACK")
+
+    def press(self, key_code):
+        """
+        Press a key
+        """
+        self.shell("input keyevent %s" % key_code)
+
+    def touch(self, x, y, orientation=-1, event_type=DOWN_AND_UP):
+        display_info = self.get_display_info()
+        current_orientation = display_info.get('orientation', -1)
+        if orientation == -1:
+            orientation = current_orientation
+        self.shell("input tap %d %d" %
+                   self.__transform_point_by_orientation((x, y), orientation, current_orientation, display_info))
+
+    def long_touch(self, x, y, duration=2000, orientation=-1):
+        """
+        Long touches at (x, y)
+        """
+        self.drag((x, y), (x, y), duration, orientation)
+
+    def drag(self, start_xy, end_xy, duration, orientation=-1):
+        """
+        Sends drag event n PX (actually it's using C{input swipe} command.
+        @param start_xy: starting point in pixel
+        @param end_xy: ending point in pixel
+        @param duration: duration of the event in ms
+        @param orientation: the orientation (-1: undefined)
+        """
+        (x0, y0) = start_xy
+        (x1, y1) = end_xy
+        display_info = self.get_display_info()
+        current_orientation = display_info.get('orientation', -1)
+        if orientation == -1:
+            orientation = current_orientation
+        (x0, y0) = self.__transform_point_by_orientation((x0, y0), orientation, current_orientation, display_info)
+        (x1, y1) = self.__transform_point_by_orientation((x1, y1), orientation, current_orientation, display_info)
+
+        version = self.get_sdk_version()
+        if version <= 15:
+            self.logger.error("drag: API <= 15 not supported (version=%d)" % version)
+        elif version <= 17:
+            self.shell("input swipe %d %d %d %d" % (x0, y0, x1, y1))
+        else:
+            self.shell("input touchscreen swipe %d %d %d %d %d" % (x0, y0, x1, y1, duration))
+
+    def type(self, text):
+        if isinstance(text, str):
+            escaped = text.replace("%s", "\\%s")
+            encoded = escaped.replace(" ", "%s")
+        else:
+            encoded = str(text)
+        # TODO find out which characters can be dangerous, and handle non-English characters
+        self.shell("input text %s" % encoded)

DroidBot/platforms/android/adapters/droidbot_app.py

@@ -0,0 +1,270 @@
+import logging
+import socket
+import subprocess
+import time
+import json
+import struct
+import traceback
+
+DROIDBOT_APP_REMOTE_ADDR = "tcp:7336"
+DROIDBOT_APP_PACKAGE = "io.github.ylimit.droidbotapp"
+DROIDBOT_APP_PACKET_HEAD_LEN = 6
+ACCESSIBILITY_SERVICE = DROIDBOT_APP_PACKAGE + "/io.github.privacystreams.accessibility.PSAccessibilityService"
+MAX_NUM_GET_VIEWS = 5
+GET_VIEW_WAIT_TIME = 1
+
+
+from ....exceptions import ADBException
+
+
+class EOF(Exception):
+    """
+    Exception in telnet connection
+    """
+    pass
+
+
+class DroidBotAppConn(object):
+    """
+    a connection with droidbot app.
+    """
+
+    def __init__(self, device=None):
+        """
+        initiate a droidbot app connection
+        :param device: instance of Device
+        :return:
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.host = "localhost"
+        if device is None:
+            from DroidBot.platforms.android import AndroidDevice as Device
+            device = Device()
+        self.device = device
+        self.port = self.device.get_random_port()
+        self.connected = False
+        self.__can_wait = True
+
+        self.sock = None
+        self.last_acc_event = None
+        self.enable_accessibility_hard = device.enable_accessibility_hard
+        self.ignore_ad = device.ignore_ad
+        if self.ignore_ad:
+            import re
+            self.__first_cap_re = re.compile("(.)([A-Z][a-z]+)")
+            self.__all_cap_re = re.compile("([a-z0-9])([A-Z])")
+
+    def __id_convert(self, name):
+        name = name.replace(".", "_").replace(":", "_").replace("/", "_")
+        s1 = self.__first_cap_re.sub(r"\1_\2", name)
+        return self.__all_cap_re.sub(r"\1_\2", s1).lower()
+
+    def set_up(self):
+        device = self.device
+        if DROIDBOT_APP_PACKAGE in device.adb.get_installed_apps():
+            self.logger.debug("DroidBot app was already installed.")
+        else:
+            # install droidbot app
+            import pkg_resources
+            droidbot_app_path = pkg_resources.resource_filename("DroidBot", "resources/droidbotApp.apk")
+            install_cmd = ["install", droidbot_app_path]
+            self.device.adb.run_cmd(install_cmd)
+            self.logger.debug("DroidBot app installed.")
+
+        device.adb.enable_accessibility_service(ACCESSIBILITY_SERVICE)
+        time.sleep(1)
+        if ACCESSIBILITY_SERVICE not in device.get_service_names() \
+                and self.device.get_sdk_version() < 23 and self.enable_accessibility_hard:
+            device.adb.enable_accessibility_service_db(ACCESSIBILITY_SERVICE)
+            if ACCESSIBILITY_SERVICE not in device.get_service_names():
+                raise ADBException("无障碍服务启用失败")
+
+        # device.start_app(droidbot_app)
+        if ACCESSIBILITY_SERVICE not in device.get_service_names() and self.__can_wait:
+            raise ADBException("无障碍服务启用失败")
+
+    def tear_down(self):
+        # Note: uninstall_app now works on the device's configured app,
+        # not arbitrary packages. Use adb directly for DroidBot app.
+
+        self.device.adb.run_cmd(["uninstall", DROIDBOT_APP_PACKAGE])
+
+    def connect(self):
+        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        try:
+            # forward host port to remote port
+            serial_cmd = "" if self.device is None else "-s " + self.device.serial
+            forward_cmd = "adb %s forward tcp:%d %s" % (serial_cmd, self.port, DROIDBOT_APP_REMOTE_ADDR)
+            subprocess.check_call(forward_cmd.split())
+            self.sock.connect((self.host, self.port))
+            import threading
+            listen_thread = threading.Thread(target=self.listen_messages)
+            listen_thread.start()
+        except subprocess.CalledProcessError as e:
+            self.connected = False
+            error_msg = f"ADB forward failed: device '{self.device.serial}' not found or disconnected"
+            self.logger.error(error_msg)
+            raise ADBException(error_msg)
+        except socket.error:
+            self.connected = False
+            traceback.print_exc()
+            raise ADBException("Failed to connect to DroidBot app")
+
+    def sock_read(self, rest_len):
+        buf = None
+        while rest_len:
+            pkt = self.sock.recv(rest_len)
+            if not pkt:
+                raise EOF()
+            if not buf:
+                buf = pkt
+            else:
+                buf += pkt
+            rest_len -= len(pkt)
+        return buf
+
+    def read_head(self):
+        header = self.sock_read(DROIDBOT_APP_PACKET_HEAD_LEN)
+        data = struct.unpack(">BBI", header)
+        return data
+
+    def listen_messages(self):
+        self.logger.debug("start listening messages")
+        self.connected = True
+        reconnect_attempts = 0
+        max_reconnect_attempts = 3
+        try:
+            while self.connected:
+                _, _, message_len = self.read_head()
+                message = self.sock_read(message_len)
+                if not isinstance(message, str):
+                    message = message.decode()
+                self.handle_message(message)
+                reconnect_attempts = 0  # 成功接收消息后重置重连计数
+            print("[CONNECTION] %s is disconnected" % self.__class__.__name__)
+        except ADBException:
+            # 设备断开连接，不再重试
+            self.logger.error("Device disconnected, stopping reconnection attempts")
+            self.connected = False
+        except Exception:
+            if self.check_connectivity():
+                reconnect_attempts += 1
+                if reconnect_attempts > max_reconnect_attempts:
+                    self.logger.error(f"Max reconnection attempts ({max_reconnect_attempts}) reached, giving up")
+                    self.connected = False
+                    return
+                traceback.print_exc()
+                # clear self.last_acc_event
+                self.logger.warning(f"Restarting droidbot app (attempt {reconnect_attempts}/{max_reconnect_attempts})")
+                self.last_acc_event = None
+                self.disconnect()
+                try:
+                    self.connect()
+                except ADBException:
+                    self.logger.error("Failed to reconnect, device may be disconnected")
+                    self.connected = False
+
+    def handle_message(self, message):
+        acc_event_idx = message.find("AccEvent >>> ")
+        if acc_event_idx >= 0:
+            if acc_event_idx > 0:
+                self.logger.warning("Invalid data before packet head: " + message[:acc_event_idx])
+            body = json.loads(message[acc_event_idx + len("AccEvent >>> "):])
+            self.last_acc_event = body
+            return
+
+        rotation_idx = message.find("rotation >>> ")
+        if rotation_idx >= 0:
+            if rotation_idx > 0:
+                self.logger.warning("Invalid data before packet head: " + message[:rotation_idx])
+            self.device.handle_rotation()
+            return
+
+        self.logger.warning("Unhandled message from droidbot app: " + message)
+        raise DroidBotAppConnException()
+
+    def check_connectivity(self):
+        """
+        check if droidbot app is connected
+        :return: True for connected
+        """
+        return self.connected
+
+    def disconnect(self):
+        """
+        disconnect telnet
+        """
+        self.connected = False
+        if self.sock is not None:
+            try:
+                self.sock.close()
+            except Exception as e:
+                self.logger.error(e)
+        try:
+            forward_remove_cmd = "adb -s %s forward --remove tcp:%d" % (self.device.serial, self.port)
+            p = subprocess.Popen(forward_remove_cmd.split(), stderr=subprocess.PIPE, stdout=subprocess.PIPE)
+            out, err = p.communicate()
+        except Exception as e:
+            self.logger.error(e)
+        self.__can_wait = False
+
+    def __view_tree_to_list(self, view_tree, view_list):
+        tree_id = len(view_list)
+        view_tree['temp_id'] = tree_id
+
+        bounds = [[-1, -1], [-1, -1]]
+        bounds[0][0] = view_tree['bounds'][0]
+        bounds[0][1] = view_tree['bounds'][1]
+        bounds[1][0] = view_tree['bounds'][2]
+        bounds[1][1] = view_tree['bounds'][3]
+        width = bounds[1][0] - bounds[0][0]
+        height = bounds[1][1] - bounds[0][1]
+        view_tree['size'] = "%d*%d" % (width, height)
+        view_tree['bounds'] = bounds
+
+        # 重命名 'class' 为 'class_name' (符合 ViewDict 标准)
+        if 'class' in view_tree:
+            view_tree['class_name'] = view_tree.pop('class')
+
+        view_list.append(view_tree)
+        children_ids = []
+        for child_tree in view_tree['children']:
+            if self.ignore_ad and child_tree['resource_id'] is not None:
+                id_word_list = self.__id_convert(child_tree['resource_id']).split('_')
+                if "ad" in id_word_list or \
+                   "banner" in id_word_list:
+                    continue
+            child_tree['parent'] = tree_id
+            self.__view_tree_to_list(child_tree, view_list)
+            children_ids.append(child_tree['temp_id'])
+        view_tree['children'] = children_ids
+
+    def get_views(self):
+        get_views_times = 0
+        while not self.last_acc_event:
+            self.logger.warning("last_acc_event is None, waiting")
+            get_views_times += 1
+            if get_views_times > MAX_NUM_GET_VIEWS:
+                self.logger.warning("cannot get non-None last_acc_event")
+                return None
+            time.sleep(GET_VIEW_WAIT_TIME)
+
+        if 'view_list' in self.last_acc_event:
+            return self.last_acc_event['view_list']
+
+        import copy
+        view_tree = copy.deepcopy(self.last_acc_event['root_node'])
+        # print view_tree
+        if not view_tree:
+            return None
+        view_tree['parent'] = -1
+        view_list = []
+        self.__view_tree_to_list(view_tree, view_list)
+        self.last_acc_event['view_list'] = view_list
+        return view_list
+
+
+if __name__ == "__main__":
+    droidbot_app_conn = DroidBotAppConn()
+    droidbot_app_conn.set_up()
+    droidbot_app_conn.connect()

DroidBot/platforms/android/adapters/droidbot_ime.py

@@ -0,0 +1,93 @@
+# coding=utf-8
+
+import logging
+import time
+
+
+
+DROIDBOT_APP_PACKAGE = "io.github.ylimit.droidbotapp"
+IME_SERVICE = DROIDBOT_APP_PACKAGE + "/.DroidBotIME"
+
+
+class DroidBotIme(object):
+    """
+    a connection with droidbot ime app.
+    """
+    def __init__(self, device=None):
+        """
+        initiate a emulator console via telnet
+        :param device: instance of Device
+        :return:
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        if device is None:
+            from DroidBot.platforms.android import AndroidDevice as Device
+            device = Device()
+        self.device = device
+        self.connected = False
+
+    def set_up(self):
+        device = self.device
+        if DROIDBOT_APP_PACKAGE in device.adb.get_installed_apps():
+            self.logger.debug("DroidBot app was already installed.")
+        else:
+            # install droidbot app
+
+            import pkg_resources
+            droidbot_app_path = pkg_resources.resource_filename("DroidBot", "resources/droidbotApp.apk")
+            install_cmd = ["install", droidbot_app_path]
+            self.device.adb.run_cmd(install_cmd)
+            self.logger.debug("DroidBot app installed.")
+
+    def tear_down(self):
+        self.device.uninstall_app(DROIDBOT_APP_PACKAGE)
+
+    def connect(self):
+        r_enable = self.device.adb.shell("ime enable %s" % IME_SERVICE)
+        if "now enabled" in r_enable or "already enabled" in r_enable:
+            r_set = self.device.adb.shell("ime set %s" % IME_SERVICE)
+            if f"{IME_SERVICE} selected" in r_set:
+                self.connected = True
+                return
+        self.logger.warning("Failed to connect DroidBotIME!")
+
+    def check_connectivity(self):
+        """
+        check if droidbot app is connected
+        :return: True for connected
+        """
+        return self.connected
+
+    def disconnect(self):
+        """
+        disconnect telnet
+        """
+        self.connected = False
+        r_disable = self.device.adb.shell("ime disable %s" % IME_SERVICE)
+        if "now disabled" in r_disable:
+            self.connected = False
+            print("[CONNECTION] %s is disconnected" % self.__class__.__name__)
+            return
+        self.logger.warning("Failed to disconnect DroidBotIME!")
+
+    def input_text(self, text, mode=0):
+        """
+        Input text to target device
+        :param text: text to input, can be unicode format
+        :param mode: 0 - set text; 1 - append text.
+        """
+        text_nospace = text.replace(' ', '--')
+        input_cmd = 'am broadcast -a DROIDBOT_INPUT_TEXT --es text %s --ei mode %d' % (text_nospace, mode)
+        self.device.adb.shell(str(input_cmd))
+
+
+if __name__ == "__main__":
+    droidbot_ime_conn = DroidBotIme()
+    droidbot_ime_conn.set_up()
+    droidbot_ime_conn.connect()
+    droidbot_ime_conn.input_text("hello world!", 0)
+    droidbot_ime_conn.input_text("世界你好!", 1)
+    time.sleep(2)
+    droidbot_ime_conn.input_text("再见。Bye bye.", 0)
+    droidbot_ime_conn.disconnect()
+    droidbot_ime_conn.tear_down()

DroidBot/platforms/android/adapters/logcat.py

@@ -0,0 +1,78 @@
+import subprocess
+import logging
+import copy
+
+
+class Logcat(object):
+    """
+    A connection with the target device through logcat.
+    """
+
+    def __init__(self, device=None, enable_file_output=False):
+        """
+        initialize logcat connection
+        :param device: a Device instance
+        :param enable_file_output: whether to write logs to file
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        if device is None:
+            from DroidBot.platforms.android import AndroidDevice as Device
+            device = Device()
+        self.device = device
+        self.connected = False
+        self.process = None
+        self.parsers = []
+        self.recent_lines = []
+        if device.output_dir is None or not enable_file_output:
+            self.out_file = None
+        else:
+            self.out_file = "%s/logcat.txt" % device.output_dir
+
+    def connect(self):
+        self.device.adb.run_cmd("logcat -c")
+        self.process = subprocess.Popen(["adb", "-s", self.device.serial, "logcat", "-v", "threadtime", "*:I"],
+                                        stdin=subprocess.PIPE,
+                                        stderr=subprocess.PIPE,
+                                        stdout=subprocess.PIPE)
+        import threading
+        listen_thread = threading.Thread(target=self.handle_output)
+        listen_thread.start()
+
+    def disconnect(self):
+        self.connected = False
+        if self.process is not None:
+            self.process.terminate()
+
+    def check_connectivity(self):
+        return self.connected
+
+    def get_recent_lines(self):
+        lines = self.recent_lines
+        self.recent_lines = []
+        return lines
+
+    def handle_output(self):
+        self.connected = True
+
+        f = None
+        if self.out_file is not None:
+            f = open(self.out_file, 'w', encoding='utf-8')
+
+        while self.connected:
+            if self.process is None:
+                continue
+            line = self.process.stdout.readline()
+            if not isinstance(line, str):
+                line = line.decode()
+            self.recent_lines.append(line)
+            self.parse_line(line)
+            if f is not None:
+                f.write(line)
+        if f is not None:
+            f.close()
+        print("[CONNECTION] %s is disconnected" % self.__class__.__name__)
+
+    def parse_line(self, logcat_line):
+        for parser in self.parsers:
+            parser.parse(logcat_line)
+

DroidBot/platforms/android/adapters/minicap.py

@@ -0,0 +1,327 @@
+import logging
+import socket
+import subprocess
+import time
+import os
+from datetime import datetime
+
+
+
+MINICAP_REMOTE_ADDR = "localabstract:minicap"
+ROTATION_CHECK_INTERVAL_S = 1 # Check rotation once per second
+
+
+class MinicapException(Exception):
+    """
+    Exception in minicap connection
+    """
+    pass
+
+
+class Minicap(object):
+    """
+    a connection with target device through minicap.
+    """
+    def __init__(self, device=None):
+        """
+        initiate a minicap connection
+        :param device: instance of Device
+        :return:
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.host = "localhost"
+
+        if device is None:
+            from DroidBot.platforms.android import AndroidDevice as Device
+            device = Device()
+        self.device = device
+        self.port = self.device.get_random_port()
+
+        self.remote_minicap_path = "/data/local/tmp/minicap-devel"
+
+        self.sock = None
+        self.connected = False
+        self.minicap_process = None
+        self.banner = None
+        self.width = -1
+        self.height = -1
+        self.orientation = -1
+
+        self.last_screen = None
+        self.last_screen_time = None
+        self.last_views = []
+        self.last_rotation_check_time = datetime.now()
+
+    def set_up(self):
+        device = self.device
+
+        try:
+            minicap_files = device.adb.shell("ls %s 2>/dev/null" % self.remote_minicap_path).split()
+            if "minicap.so" in minicap_files and ("minicap" in minicap_files or "minicap-nopie" in minicap_files):
+                self.logger.debug("minicap was already installed.")
+                return
+        except:
+            pass
+
+        if device is not None:
+            # install minicap
+            import pkg_resources
+            local_minicap_path = pkg_resources.resource_filename("DroidBot", "resources/minicap")
+            try:
+                device.adb.shell("mkdir %s" % self.remote_minicap_path)
+            except Exception:
+                pass
+            abi = device.adb.get_property('ro.product.cpu.abi')
+            sdk = device.get_sdk_version()
+            if sdk >= 16:
+                minicap_bin = "minicap"
+            else:
+                minicap_bin = "minicap-nopie"
+            minicap_bin_path = os.path.join(local_minicap_path, 'libs', abi, minicap_bin)
+            device.push_file(local_file=minicap_bin_path, remote_dir=self.remote_minicap_path)
+            minicap_so_path = os.path.join(local_minicap_path, 'jni', 'libs', f'android-{sdk}', abi, 'minicap.so')
+            device.push_file(local_file=minicap_so_path, remote_dir=self.remote_minicap_path)
+            self.logger.debug("minicap installed.")
+
+    def tear_down(self):
+        try:
+            delete_minicap_cmd = "adb -s %s shell rm -r %s" % (self.device.serial, self.remote_minicap_path)
+            p = subprocess.Popen(delete_minicap_cmd.split(), stderr=subprocess.PIPE, stdout=subprocess.PIPE)
+            out, err = p.communicate()
+        except Exception:
+            pass
+
+    def connect(self):
+        device = self.device
+        display = device.get_display_info(refresh=True)
+        if 'width' not in display or 'height' not in display or 'orientation' not in display:
+            self.logger.warning("Cannot get the size of current device.")
+            return
+        w = display['width']
+        h = display['height']
+        if w > h:
+            temp = w
+            w = h
+            h = temp
+        o = display['orientation'] * 90
+        self.width = w
+        self.height = h
+        self.orientation = o
+
+        size_opt = "%dx%d@%dx%d/%d" % (w, h, w, h, o)
+        grant_minicap_perm_cmd = "adb -s %s shell chmod -R a+x %s" % \
+                                 (device.serial, self.remote_minicap_path)
+        start_minicap_cmd = "adb -s %s shell LD_LIBRARY_PATH=%s %s/minicap -P %s" % \
+                            (device.serial, self.remote_minicap_path, self.remote_minicap_path, size_opt)
+        self.logger.debug("starting minicap: " + start_minicap_cmd)
+
+        p = subprocess.Popen(grant_minicap_perm_cmd.split(), stderr=subprocess.PIPE, stdout=subprocess.PIPE)
+        out, err = p.communicate()
+
+        self.minicap_process = subprocess.Popen(start_minicap_cmd.split(),
+                                                stdin=subprocess.PIPE, stdout=subprocess.PIPE)
+        # Wait 2 seconds for starting minicap
+        time.sleep(2)
+        self.logger.debug("minicap started.")
+
+        try:
+            # forward host port to remote port
+            forward_cmd = "adb -s %s forward tcp:%d %s" % (device.serial, self.port, MINICAP_REMOTE_ADDR)
+            subprocess.check_call(forward_cmd.split())
+            self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+            self.sock.connect((self.host, self.port))
+            import threading
+            listen_thread = threading.Thread(target=self.listen_messages)
+            listen_thread.start()
+        except socket.error as e:
+            self.connected = False
+            self.logger.warning(e)
+            raise MinicapException()
+
+    def listen_messages(self):
+        self.logger.debug("start listening minicap images ...")
+        CHUNK_SIZE = 4096
+
+        readBannerBytes = 0
+        bannerLength = 2
+        readFrameBytes = 0
+        frameBodyLength = 0
+        frameBody = bytearray()
+        banner = {
+            "version": 0,
+            "length": 0,
+            "pid": 0,
+            "realWidth": 0,
+            "realHeight": 0,
+            "virtualWidth": 0,
+            "virtualHeight": 0,
+            "orientation": 0,
+            "quirks": 0,
+        }
+
+        self.connected = True
+        while self.connected:
+            chunk = bytearray(self.sock.recv(CHUNK_SIZE))
+            if not chunk:
+                continue
+            chunk_len = len(chunk)
+            cursor = 0
+            while cursor < chunk_len and self.connected:
+                if readBannerBytes < bannerLength:
+                    if readBannerBytes == 0:
+                        banner['version'] = chunk[cursor]
+                    elif readBannerBytes == 1:
+                        banner['length'] = bannerLength = chunk[cursor]
+                    elif 2 <= readBannerBytes <= 5:
+                        banner['pid'] += (chunk[cursor] << ((readBannerBytes - 2) * 8))
+                    elif 6 <= readBannerBytes <= 9:
+                        banner['realWidth'] += (chunk[cursor] << ((readBannerBytes - 6) * 8))
+                    elif 10 <= readBannerBytes <= 13:
+                        banner['realHeight'] += (chunk[cursor] << ((readBannerBytes - 10) * 8))
+                    elif 14 <= readBannerBytes <= 17:
+                        banner['virtualWidth'] += (chunk[cursor] << ((readBannerBytes - 14) * 8))
+                    elif 18 <= readBannerBytes <= 21:
+                        banner['virtualHeight'] += (chunk[cursor] << ((readBannerBytes - 18) * 8))
+                    elif readBannerBytes == 22:
+                        banner['orientation'] += chunk[cursor] * 90
+                    elif readBannerBytes == 23:
+                        banner['quirks'] = chunk[cursor]
+
+                    cursor += 1
+                    readBannerBytes += 1
+                    if readBannerBytes == bannerLength:
+                        self.banner = banner
+                        self.logger.debug("minicap initialized: %s" % banner)
+
+                elif readFrameBytes < 4:
+                    frameBodyLength += (chunk[cursor] << (readFrameBytes * 8))
+                    cursor += 1
+                    readFrameBytes += 1
+                else:
+                    if chunk_len - cursor >= frameBodyLength:
+                        frameBody += chunk[cursor: cursor + frameBodyLength]
+                        self.handle_image(frameBody)
+                        cursor += frameBodyLength
+                        frameBodyLength = readFrameBytes = 0
+                        frameBody = bytearray()
+                    else:
+                        frameBody += chunk[cursor:]
+                        frameBodyLength -= chunk_len - cursor
+                        readFrameBytes += chunk_len - cursor
+                        cursor = chunk_len
+        print("[CONNECTION] %s is disconnected" % self.__class__.__name__)
+
+    def handle_image(self, frameBody):
+        # Sanity check for JPG header, only here for debugging purposes.
+        if frameBody[0] != 0xFF or frameBody[1] != 0xD8:
+            self.logger.warning("Frame body does not start with JPG header")
+        self.last_screen = frameBody
+        self.last_screen_time = datetime.now()
+        self.last_views = None
+        self.logger.debug("Received an image at %s" % self.last_screen_time)
+        self.check_rotation()
+
+    def check_rotation(self):
+        current_time = datetime.now()
+        if (current_time - self.last_rotation_check_time).total_seconds() < ROTATION_CHECK_INTERVAL_S:
+            return
+
+        display = self.device.get_display_info(refresh=True)
+        if 'orientation' in display:
+            cur_orientation = display['orientation'] * 90
+            if cur_orientation != self.orientation:
+                self.device.handle_rotation()
+        self.last_rotation_check_time = current_time
+
+    def check_connectivity(self):
+        """
+        check if droidbot app is connected
+        :return: True for connected
+        """
+        if not self.connected:
+            return False
+        if self.last_screen_time is None:
+            return False
+        return True
+
+    def disconnect(self):
+        """
+        disconnect telnet
+        """
+        self.connected = False
+        if self.sock is not None:
+            try:
+                self.sock.close()
+            except Exception as e:
+                print(e)
+        if self.minicap_process is not None:
+            try:
+                self.minicap_process.terminate()
+            except Exception as e:
+                print(e)
+        try:
+            forward_remove_cmd = "adb -s %s forward --remove tcp:%d" % (self.device.serial, self.port)
+            p = subprocess.Popen(forward_remove_cmd.split(), stderr=subprocess.PIPE, stdout=subprocess.PIPE)
+            out, err = p.communicate()
+        except Exception as e:
+            print(e)
+
+    def get_views(self):
+        """
+        get UI views using cv module - 使用统一的 ViewDict 格式
+        opencv-python need to be installed for this function
+        :return: a list of views (List[ViewDict])
+        """
+        if not self.last_screen:
+            self.logger.warning("last_screen is None")
+            return None
+        if self.last_views:
+            return self.last_views
+
+        from . import cv
+        img = cv.load_image_from_buf(self.last_screen)
+        
+        # find_views 现在直接返回 List[ViewDict] 格式
+        cv_views = cv.find_views(img)
+        
+        # 构建根视图
+        root_view = {
+            "class_name": "CVViewRoot",
+            "bounds": [[0, 0], [self.width, self.height]],
+            "enabled": True,
+            "visible": True,
+            "clickable": False,
+            "scrollable": False,
+            "editable": False,
+            "temp_id": 0,
+            "children": [],
+            "text": "",
+            "source": "cv",
+            "resource_id": "",
+            "view_str": "cv_root",
+        }
+        
+        # 重新分配 temp_id 并设置父子关系
+        views = [root_view]
+        for idx, view in enumerate(cv_views):
+            view["temp_id"] = idx + 1
+            view["parent"] = 0
+            views.append(view)
+        
+        root_view["children"] = list(range(1, len(views)))
+
+        self.last_views = views
+        return views
+
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.DEBUG)
+    minicap = Minicap()
+    try:
+        minicap.set_up()
+        minicap.connect()
+    except:
+        minicap.disconnect()
+        minicap.tear_down()
+        minicap.device.disconnect()

DroidBot/platforms/android/adapters/process_monitor.py

@@ -0,0 +1,84 @@
+import threading
+import logging
+import time
+import subprocess
+class ProcessMonitor(object):
+    """
+    monitoring the state of process on the device
+    """
+
+    def __init__(self, device=None, app=None):
+        """
+        initiate a process monitor
+        :param device: Device instance
+        :param app: App instance
+        :return:
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.enabled = True
+        self.device = device
+        self.app = app
+        self.pid2user = {}
+        self.pid2ppid = {}
+        self.pid2name = {}
+        self.lock = threading.Lock()
+
+
+    def connect(self):
+        """
+        start the monitor in a another thread.
+        From now on, the on_state_updated method in listeners will be continuously called
+        :return:
+        """
+        self.enabled = True
+        gps_thread = threading.Thread(target=self.maintain_process_mapping)
+        gps_thread.start()
+        return True
+
+    def disconnect(self):
+        self.enabled = False
+
+    def check_connectivity(self):
+        return self.enabled
+
+    def maintain_process_mapping(self):
+        """
+        maintain pid2user mapping, pid2ppid mapping and pid2name mapping by continuously calling ps command
+        """
+        while self.enabled:
+            if self.device is not None:
+                ps_cmd = ["adb", "-s", self.device.serial, "shell", "ps"]
+            else:
+                ps_cmd = ["adb", "shell", "ps"]
+
+            try:
+                ps_out = subprocess.check_output(ps_cmd)
+                if not isinstance(ps_out, str):
+                    ps_out = ps_out.decode()
+            except subprocess.CalledProcessError:
+                continue
+
+            # parse ps_out to update self.pid2uid mapping and self.pid2name mapping
+            ps_out_lines = ps_out.splitlines()
+            ps_out_head = ps_out_lines[0].split()
+            if ps_out_head[0] != "USER" or ps_out_head[1] != "PID" \
+                    or ps_out_head[2] != "PPID" or ps_out_head[-1] != "NAME":
+                self.device.logger.warning("ps command output format error: %s" % ps_out_head)
+
+            for ps_out_line in ps_out_lines[1:]:
+                segs = ps_out_line.split()
+                if len(segs) < 4:
+                    continue
+                user = segs[0]
+                pid = segs[1]
+                ppid = segs[2]
+                name = segs[-1]
+                self.lock.acquire()
+                self.pid2name[pid] = name
+                self.pid2ppid[pid] = ppid
+                self.pid2user[pid] = user
+                self.lock.release()
+
+            time.sleep(1)
+        print("[CONNECTION] %s is disconnected" % self.__class__.__name__)
+

DroidBot/platforms/android/adapters/telnet.py

@@ -0,0 +1,101 @@
+import logging
+import threading
+import logging
+import threading
+
+
+class TelnetException(Exception):
+    """
+    Exception in telnet connection
+    """
+    pass
+
+
+class TelnetConsole(object):
+    """
+    interface of telnet console, see:
+    http://developer.android.com/tools/devices/emulator.html
+    """
+    def __init__(self, device=None, auth_token=None):
+        """
+        Initiate a emulator console via telnet.
+        On some devices, an authentication token is required to use telnet
+        :param device: instance of Device
+        :return:
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+
+        if device is None:
+            from DroidBot.platforms.android import AndroidDevice as Device
+            device = Device()
+        self.device = device
+        self.auth_token = auth_token
+        self.console = None
+        self.__lock__ = threading.Lock()
+
+    def connect(self):
+        if self.device.serial and self.device.serial.startswith("emulator-"):
+            host = "localhost"
+            port = int(self.device.serial[9:])
+            from telnetlib import Telnet
+            self.console = Telnet(host, port)
+            if self.auth_token is not None:
+                self.run_cmd("auth %s" % self.auth_token)
+            if self.check_connectivity():
+                self.logger.debug("telnet successfully initiated, the port is %d" % port)
+                return
+        raise TelnetException()
+
+    def run_cmd(self, args):
+        """
+        run a command in emulator console
+        :param args: arguments to be executed in telnet console
+        :return:
+        """
+        if self.console is None:
+            self.logger.warning("telnet is not connected!")
+            return None
+        if isinstance(args, list):
+            cmd_line = " ".join(args)
+        elif isinstance(args, str):
+            cmd_line = args
+        else:
+            self.logger.warning("unsupported command format:" + args)
+            return None
+
+        self.logger.debug('command:')
+        self.logger.debug(cmd_line)
+
+        cmd_line += '\n'
+
+        self.__lock__.acquire()
+        self.console.write(cmd_line)
+        r = self.console.read_until('OK', 5)
+        # eat the rest outputs
+        self.console.read_until('NEVER MATCH', 1)
+        self.__lock__.release()
+
+        self.logger.debug('return:')
+        self.logger.debug(r)
+        return r
+
+    def check_connectivity(self):
+        """
+        check if console is connected
+        :return: True for connected
+        """
+        if self.console is None:
+            return False
+        try:
+            self.run_cmd("help")
+        except:
+            return False
+        return True
+
+    def disconnect(self):
+        """
+        disconnect telnet
+        """
+        if self.console is not None:
+            self.console.close()
+        print("[CONNECTION] %s is disconnected" % self.__class__.__name__)

DroidBot/platforms/android/adapters/user_input_monitor.py

@@ -0,0 +1,70 @@
+import subprocess
+import logging
+import subprocess
+import logging
+
+
+class UserInputMonitor(object):
+    """
+    A connection with the target device through `getevent`.
+    `getevent` is able to get raw user input from device.
+    """
+
+    def __init__(self, device=None):
+        """
+        initialize connection
+        :param device: a Device instance
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+
+        if device is None:
+            from DroidBot.platforms.android import AndroidDevice as Device
+            device = Device()
+        self.device = device
+        self.connected = False
+        self.process = None
+        if device.output_dir is None:
+            self.out_file = None
+        else:
+            self.out_file = "%s/user_input.txt" % device.output_dir
+
+    def connect(self):
+        self.process = subprocess.Popen(["adb", "-s", self.device.serial, "shell", "getevent", "-lt"],
+                                        stdin=subprocess.PIPE,
+                                        stderr=subprocess.PIPE,
+                                        stdout=subprocess.PIPE)
+        import threading
+        listen_thread = threading.Thread(target=self.handle_output)
+        listen_thread.start()
+
+    def disconnect(self):
+        self.connected = False
+        if self.process is not None:
+            self.process.terminate()
+
+    def check_connectivity(self):
+        return self.connected
+
+    def handle_output(self):
+        self.connected = True
+
+        f = None
+        if self.out_file is not None:
+            f = open(self.out_file, 'w')
+
+        while self.connected:
+            if self.process is None:
+                continue
+            line = self.process.stdout.readline()
+            if not isinstance(line, str):
+                line = line.decode()
+            self.parse_line(line)
+            if f is not None:
+                f.write(line)
+
+        if f is not None:
+            f.close()
+        print("[CONNECTION] %s is disconnected" % self.__class__.__name__)
+
+    def parse_line(self, _getevent_line):
+        pass

DroidBot/platforms/android/android_app.py

@@ -0,0 +1,113 @@
+"""
+Android App Module
+Android-specific application model.
+"""
+import logging
+import os
+from .android_intent import AndroidIntent
+from ...core.abstract_app import AbstractApp
+from ...exceptions import FATAL_EXCEPTIONS
+
+
+class AndroidApp(AbstractApp):
+    """
+    Android 应用类 - 用于解析和管理 APK 文件信息
+    
+    继承自 AbstractApp，实现 Android 特定的应用管理功能。
+    """
+
+    def __init__(self, package_name, output_dir=None):
+        """
+        创建 AndroidApp 实例
+        
+        :param package_name: 应用包名
+        :param output_dir: 输出目录路径
+        """
+        super().__init__(output_dir=output_dir)
+        
+        self.app_path = None
+        self.output_dir = output_dir
+        
+        if output_dir is not None:
+            if not os.path.isdir(output_dir):
+                os.makedirs(output_dir)
+
+        self.package_name = package_name
+        self._main_activity = None
+        self._activities = []
+        self.dumpsys_main_activity = None
+
+    @property
+    def identifier(self) -> str:
+        """获取应用唯一标识符（package_name）"""
+        return self.package_name
+
+    @property
+    def main_activity(self) -> str:
+        """获取主 Activity 名称"""
+        if self._main_activity is not None:
+            return self._main_activity
+        else:
+            if self.dumpsys_main_activity:
+                self.logger.warning("Cannot get main activity from manifest. Using dumpsys result instead.")
+                return self.dumpsys_main_activity
+            return None
+    
+    @main_activity.setter
+    def main_activity(self, value):
+        """设置主 Activity"""
+        self._main_activity = value
+    
+    @property
+    def activities(self) -> list:
+        """获取入口点列表（activities）"""
+        return self._activities
+
+    @activities.setter
+    def activities(self, value):
+        """设置 Activity 列表"""
+        self._activities = value
+
+    def get_start_with_profiling_intent(self, trace_file, sampling=None) -> 'AndroidIntent':
+        """获取带 profiling 的启动 Intent"""
+        package_name = self.package_name
+        if self.main_activity:
+            package_name += "/%s" % self.main_activity
+        if sampling is not None:
+            return AndroidIntent(prefix="start --start-profiler %s --sampling %d" % (trace_file, sampling), suffix=package_name)
+        else:
+            return AndroidIntent(prefix="start --start-profiler %s" % trace_file, suffix=package_name)
+
+
+    def populate_activities(self, device):
+        """
+        使用 adb dumpsys package 获取所有 Activity
+        """
+        import re
+        self.logger.info(f"Populating activities for {self.package_name} via dumpsys...")
+        try:
+            cmd = f"dumpsys package {self.package_name}"
+            output = device.adb.shell(cmd)
+            
+            # 匹配形如: [hash] package/activity
+            pattern = re.compile(r'([0-9a-f]+)\s+' + re.escape(self.package_name) + r'/([^ \s]+)')
+            matches = pattern.findall(output)
+            
+            new_activities = []
+            for m in matches:
+                activity_full_name = f"{self.package_name}/{m[1]}"
+                if activity_full_name not in new_activities:
+                    new_activities.append(activity_full_name)
+            
+            if new_activities:
+                self.activities = sorted(new_activities)
+                self.logger.info(f"Found {len(self.activities)} activities via dumpsys.")
+            else:
+                self.logger.warning("No activities found via dumpsys.")
+                
+        except FATAL_EXCEPTIONS:
+            raise
+        except Exception as e:
+            self.logger.error(f"Failed to populate activities via dumpsys: {e}")
+            # 非致命异常，activities保持为空
+

DroidBot/platforms/android/android_device_state.py

@@ -0,0 +1,360 @@
+"""
+Android Device State Implementation
+Concrete implementation of AbstractDeviceState for Android devices.
+"""
+import copy
+import os
+from typing import Optional, Dict, Any, List, Set
+
+from ...core.abstract_device_state import AbstractDeviceState
+from ...core.abstract_input_event import EventType
+from ...exceptions import FATAL_EXCEPTIONS
+
+
+class AndroidDeviceState(AbstractDeviceState):
+    """
+    Android 设备状态的具体实现
+    """
+
+    def __init__(self, device, views, foreground_activity, activity_stack, 
+                 background_services, tag=None, screenshot_path=None):
+        """
+        初始化 Android 设备状态
+        
+        :param device: AndroidDevice 实例
+        :param views: 视图列表
+        :param foreground_activity: 前台 Activity
+        :param activity_stack: Activity 栈
+        :param background_services: 后台服务
+        :param tag: 状态标签
+        :param screenshot_path: 截图路径
+        """
+        super().__init__(device, tag, screenshot_path)
+        
+        self._foreground_activity = foreground_activity
+        self.activity_stack = activity_stack if isinstance(activity_stack, list) else []
+        self.background_services = background_services
+        
+        # 解析视图
+        self._views = self._parse_views(views)
+        self._view_tree = {}
+        self._assemble_view_tree(self._view_tree, self._views)
+        self._generate_view_strs()
+        
+        # 计算状态标识
+        self._state_str = self._get_state_str()
+        self._structure_str = self._get_content_free_state_str()
+        self._search_content = self._get_search_content()
+    
+    # ==================== 属性实现 ====================
+    
+    @property
+    def search_content(self) -> str:
+        return self._search_content
+    
+    @property
+    def views(self) -> List[Dict[str, Any]]:
+        return self._views
+    
+    @property
+    def view_tree(self) -> Dict[str, Any]:
+        return self._view_tree
+    
+    @property
+    def state_str(self) -> str:
+        return self._state_str
+    
+    @property
+    def structure_str(self) -> str:
+        return self._structure_str
+    
+    @property
+    def foreground_page(self) -> Optional[str]:
+        """实现抽象接口 - 返回当前页面标识符（Android 上就是 foreground_activity）"""
+        return self._foreground_activity
+    
+    
+    def _parse_views(self, raw_views) -> List[Dict[str, Any]]:
+        """解析原始视图数据，确保符合 ViewDict 格式"""
+        views = []
+        if not raw_views or len(raw_views) == 0:
+            return views
+        for view_dict in raw_views:
+            # 添加来源标识
+            view_dict['source'] = 'accessibility'
+            views.append(view_dict)
+        return views
+
+    
+    def _assemble_view_tree(self, root_view, views) -> None:
+        """组装视图树"""
+        if not len(self._view_tree):
+            if not len(views):
+                return
+            self._view_tree = copy.deepcopy(views[0])
+            self._assemble_view_tree(self._view_tree, views)
+        else:
+            children = list(enumerate(root_view.get("children", [])))
+            if not len(children):
+                return
+            for i, j in children:
+                if j < len(self._views):
+                    root_view["children"][i] = copy.deepcopy(self._views[j])
+                    self._assemble_view_tree(root_view["children"][i], views)
+    
+    def _generate_view_strs(self) -> None:
+        """生成视图字符串标识"""
+        for view_dict in self._views:
+            self._get_view_str(view_dict)
+    
+    # ==================== 状态标识 ====================
+    
+    def _get_state_str(self) -> str:
+        """获取状态唯一标识"""
+        state_str_raw = self._get_state_str_raw()
+        from .utils import md5
+        return md5(state_str_raw)
+    
+    def _get_state_str_raw(self) -> str:
+        """获取原始状态字符串"""
+        if self.device.humanoid is not None:
+            import json
+            from xmlrpc.client import ServerProxy
+            proxy = ServerProxy("http://%s/" % self.device.humanoid)
+            return proxy.render_view_tree(json.dumps({
+                "view_tree": self._view_tree,
+                "screen_res": [self.device.display_info["width"],
+                               self.device.display_info["height"]]
+            }))
+        else:
+            view_signatures = set()
+            for view in self._views:
+                view_signature = self._get_view_signature(view)
+                if view_signature:
+                    view_signatures.add(view_signature)
+            return "%s{%s}" % (self._foreground_activity, ",".join(sorted(view_signatures)))
+    
+    def _get_content_free_state_str(self) -> str:
+        """获取内容无关的状态标识"""
+        view_signatures = set()
+        for view in self._views:
+            view_signature = self._get_content_free_view_signature(view)
+            if view_signature:
+                view_signatures.add(view_signature)
+        state_str = "%s{%s}" % (self._foreground_activity, ",".join(sorted(view_signatures)))
+        import hashlib
+        return hashlib.md5(state_str.encode('utf-8')).hexdigest()
+    
+    def _get_search_content(self) -> str:
+        """获取搜索内容"""
+        words = [
+            ",".join(self._get_property_from_all_views("resource_id")),
+            ",".join(self._get_property_from_all_views("text"))
+        ]
+        return "\n".join(words)
+    
+    def _get_property_from_all_views(self, property_name: str) -> Set[str]:
+        """从所有视图获取属性值"""
+        property_values = set()
+        for view in self._views:
+            property_value = self._safe_dict_get(view, property_name, None)
+            if property_value:
+                property_values.add(property_value)
+        return property_values
+    
+    # ==================== 视图签名 ====================
+    
+    @staticmethod
+    def _get_view_signature(view_dict: Dict[str, Any]) -> Optional[str]:
+        """获取视图签名"""
+        if 'signature' in view_dict:
+            return view_dict['signature']
+        
+        view_text = AndroidDeviceState._safe_dict_get(view_dict, 'text', "None")
+        if view_text is None or len(view_text) > 50:
+            view_text = "None"
+        
+        signature = "[class]%s[resource_id]%s[text]%s[%s,%s,%s]" % (
+            AndroidDeviceState._safe_dict_get(view_dict, 'class_name', "None"),
+            AndroidDeviceState._safe_dict_get(view_dict, 'resource_id', "None"),
+            view_text,
+            AndroidDeviceState._key_if_true(view_dict, 'enabled'),
+            AndroidDeviceState._key_if_true(view_dict, 'checked'),
+            AndroidDeviceState._key_if_true(view_dict, 'selected')
+        )
+        view_dict['signature'] = signature
+        return signature
+    
+    @staticmethod
+    def _get_content_free_view_signature(view_dict: Dict[str, Any]) -> Optional[str]:
+        """获取内容无关的视图签名"""
+        if 'content_free_signature' in view_dict:
+            return view_dict['content_free_signature']
+        
+        content_free_signature = "[class]%s[resource_id]%s" % (
+            AndroidDeviceState._safe_dict_get(view_dict, 'class_name', "None"),
+            AndroidDeviceState._safe_dict_get(view_dict, 'resource_id', "None")
+        )
+        view_dict['content_free_signature'] = content_free_signature
+        return content_free_signature
+    
+    def _get_view_str(self, view_dict: Dict[str, Any]) -> str:
+        """获取视图字符串"""
+        if 'view_str' in view_dict:
+            return view_dict['view_str']
+        
+        view_signature = self._get_view_signature(view_dict)
+        parent_strs = []
+        for parent_id in self.get_all_ancestors(view_dict):
+            parent_strs.append(self._get_view_signature(self._views[parent_id]))
+        parent_strs.reverse()
+        
+        child_strs = []
+        for child_id in self.get_all_children(view_dict):
+            child_strs.append(self._get_view_signature(self._views[child_id]))
+        child_strs.sort()
+        
+        view_str = "Activity:%s\nSelf:%s\nParents:%s\nChildren:%s" % (
+            self._foreground_activity, view_signature, 
+            "//".join(parent_strs), "||".join(child_strs)
+        )
+        import hashlib
+        view_str = hashlib.md5(view_str.encode('utf-8')).hexdigest()
+        view_dict['view_str'] = view_str
+        return view_str
+    
+    # ==================== 辅助方法 ====================
+    
+    @staticmethod
+    def _key_if_true(view_dict: Dict[str, Any], key: str) -> str:
+        return key if (key in view_dict and view_dict[key]) else ""
+    
+    @staticmethod
+    def _safe_dict_get(view_dict: Dict[str, Any], key: str, default=None):
+        value = view_dict.get(key, None)
+        return value if value is not None else default
+    
+    def get_all_ancestors(self, view_dict: Dict[str, Any]) -> List[int]:
+        """获取所有祖先节点 ID"""
+        result = []
+        parent_id = self._safe_dict_get(view_dict, 'parent', -1)
+        if 0 <= parent_id < len(self._views):
+            result.append(parent_id)
+            result += self.get_all_ancestors(self._views[parent_id])
+        return result
+    
+    def get_all_children(self, view_dict: Dict[str, Any]) -> Set[int]:
+        """获取所有子节点 ID"""
+        children = self._safe_dict_get(view_dict, 'children')
+        if not children:
+            return set()
+        children = set(children)
+        for child in list(children):
+            if child < len(self._views):
+                children_of_child = self.get_all_children(self._views[child])
+                children = children.union(children_of_child)
+        return children
+    
+    # ==================== 抽象方法实现 ====================
+    
+    def get_possible_input(self) -> List:
+        """获取可能的输入事件"""
+        if self._possible_events:
+            return [] + self._possible_events
+        
+        from .android_input_event import (
+            AndroidTouchEvent, AndroidLongTouchEvent, 
+            AndroidScrollEvent, AndroidSetTextEvent
+        )
+        
+        possible_events = []
+        enabled_view_ids = []
+        touch_exclude_view_ids = set()
+        
+        for view_dict in self._views:
+            if (self._safe_dict_get(view_dict, 'enabled') and
+                self._safe_dict_get(view_dict, 'visible') and
+                self._safe_dict_get(view_dict, 'resource_id') not in
+                ['android:id/navigationBarBackground', 'android:id/statusBarBackground']):
+                enabled_view_ids.append(view_dict['temp_id'])
+        
+        for view_id in enabled_view_ids:
+            if self._safe_dict_get(self._views[view_id], 'clickable'):
+                possible_events.append(AndroidTouchEvent(view=self._views[view_id]))
+                touch_exclude_view_ids.add(view_id)
+                touch_exclude_view_ids = touch_exclude_view_ids.union(
+                    self.get_all_children(self._views[view_id])
+                )
+        
+        # 添加滚动事件
+        possible_events.append(AndroidScrollEvent(direction="up"))
+        possible_events.append(AndroidScrollEvent(direction="up"))
+        possible_events.append(AndroidScrollEvent(direction="up"))
+        possible_events.append(AndroidScrollEvent(direction="down"))
+        
+        for view_id in enabled_view_ids:
+            if self._safe_dict_get(self._views[view_id], 'checkable'):
+                possible_events.append(AndroidTouchEvent(view=self._views[view_id]))
+                touch_exclude_view_ids.add(view_id)
+        
+        for view_id in enabled_view_ids:
+            if self._safe_dict_get(self._views[view_id], 'long_clickable'):
+                possible_events.append(AndroidLongTouchEvent(view=self._views[view_id]))
+        
+        for view_id in enabled_view_ids:
+            if self._safe_dict_get(self._views[view_id], 'editable'):
+                possible_events.append(AndroidSetTextEvent(
+                    view=self._views[view_id], text="cat"
+                ))
+                touch_exclude_view_ids.add(view_id)
+        
+        for view_id in enabled_view_ids:
+            if view_id in touch_exclude_view_ids:
+                continue
+            children = self._safe_dict_get(self._views[view_id], 'children')
+            if children and len(children) > 0:
+                continue
+            possible_events.append(AndroidTouchEvent(view=self._views[view_id]))
+        
+        self._possible_events = possible_events
+        return [] + possible_events
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """序列化为字典"""
+        return {
+            'tag': self.tag,
+            'state_str': self._state_str,
+            'state_str_content_free': self._structure_str,
+            'foreground_activity': self._foreground_activity,
+            'activity_stack': self.activity_stack,
+            'background_services': self.background_services,
+            'width': self.width,
+            'height': self.height,
+            'views': self._views
+        }
+    
+    def get_app_page_depth(self) -> int:
+        """获取应用页面深度 - 实现抽象接口"""
+        if not self.device._app:
+            return 0  # 没有指定应用时，返回0（在前台）
+        
+        package_name = self.device.app_identifier
+        if self._foreground_activity and package_name in self._foreground_activity:
+            return 0
+        
+        try:
+            app_pid = self.device.get_app_pid(package_name)
+            if app_pid:
+                return 0
+        except FATAL_EXCEPTIONS:
+            raise
+        except Exception as e:
+            self.device.logger.error(f"Exception checking app PID: {e}")
+            return -1  # 非致命异常，返回默认深度
+        
+        depth = 0
+        for activity_str in self.activity_stack:
+            if package_name in activity_str:
+                return depth
+            depth += 1
+        return -1

DroidBot/platforms/android/android_input_event.py

@@ -0,0 +1,171 @@
+"""
+Android Input Event Implementations
+Concrete implementations of input events for Android devices.
+"""
+from typing import Optional, Dict, Any, List
+
+from ...core.abstract_input_event import (
+    AbstractInputEvent, EventType,
+    BaseTouchEvent, BaseLongTouchEvent, BaseSwipeEvent,
+    BaseScrollEvent, BaseSetTextEvent, BaseKeyEvent, BaseKillAppEvent
+)
+
+
+class AndroidTouchEvent(BaseTouchEvent):
+    """Android 触摸事件"""
+    
+    def send(self, device) -> bool:
+        """发送触摸事件到 Android 设备"""
+        x, y = self.x, self.y
+        if self.view is not None:
+            from .android_device_state import AndroidDeviceState
+            x, y = AndroidDeviceState.get_view_center(self.view)
+        device.view_touch(int(x), int(y))
+        return True
+
+
+class AndroidLongTouchEvent(BaseLongTouchEvent):
+    """Android 长按事件"""
+    
+    def send(self, device) -> bool:
+        """发送长按事件到 Android 设备"""
+        x, y = self.x, self.y
+        if self.view is not None:
+            from .android_device_state import AndroidDeviceState
+            x, y = AndroidDeviceState.get_view_center(self.view)
+        device.view_long_touch(int(x), int(y), self.duration)
+        return True
+
+
+class AndroidSwipeEvent(BaseSwipeEvent):
+    """Android 滑动事件"""
+    
+    def send(self, device) -> bool:
+        """发送滑动事件到 Android 设备"""
+        device.view_drag(
+            (self.start_x, self.start_y),
+            (self.end_x, self.end_y),
+            self.duration
+        )
+        return True
+
+
+class AndroidScrollEvent(BaseScrollEvent):
+    """Android 滚动事件"""
+    
+    def send(self, device) -> bool:
+        """发送滚动事件到 Android 设备"""
+        display_info = device.get_display_info()
+        width = display_info.get('width', 1080)
+        height = display_info.get('height', 1920)
+        
+        # 计算滚动的起点和终点
+        center_x = width // 2
+        center_y = height // 2
+        
+        if self.direction == self.DIRECTION_UP:
+            start_y = center_y + height // 4
+            end_y = center_y - height // 4
+            device.view_drag((center_x, start_y), (center_x, end_y), 300)
+        elif self.direction == self.DIRECTION_DOWN:
+            start_y = center_y - height // 4
+            end_y = center_y + height // 4
+            device.view_drag((center_x, start_y), (center_x, end_y), 300)
+        elif self.direction == self.DIRECTION_LEFT:
+            start_x = center_x + width // 4
+            end_x = center_x - width // 4
+            device.view_drag((start_x, center_y), (end_x, center_y), 300)
+        elif self.direction == self.DIRECTION_RIGHT:
+            start_x = center_x - width // 4
+            end_x = center_x + width // 4
+            device.view_drag((start_x, center_y), (end_x, center_y), 300)
+        
+        return True
+
+
+class AndroidSetTextEvent(BaseSetTextEvent):
+    """Android 文本输入事件"""
+    
+    def send(self, device) -> bool:
+        """发送文本输入事件到 Android 设备"""
+        # 先点击目标视图
+        if self.view is not None:
+            from .android_device_state import AndroidDeviceState
+            x, y = AndroidDeviceState.get_view_center(self.view)
+            device.view_touch(int(x), int(y))
+            import time
+            time.sleep(0.5)
+        
+        # 输入文本
+        device.view_set_text(self.text)
+        
+        # 默认按回车键确认
+        device.key_press("66") # ENTER key code
+
+        return True
+
+
+class AndroidKeyEvent(BaseKeyEvent):
+    """Android 按键事件"""
+    
+    # Android 特定按键映射
+    KEY_MAP = {
+        'BACK': 'BACK',
+        'HOME': 'HOME',
+        'MENU': 'MENU',
+        'ENTER': '66',
+        'ESCAPE': '111',
+        'POWER': 'POWER',
+        'VOLUME_UP': 'VOLUME_UP',
+        'VOLUME_DOWN': 'VOLUME_DOWN',
+    }
+    
+    def send(self, device) -> bool:
+        """发送按键事件到 Android 设备"""
+        key_code = self.KEY_MAP.get(self.key_name, self.key_name)
+        device.key_press(key_code)
+        return True
+
+
+class AndroidIntentEvent(AbstractInputEvent):
+    """Android Intent 事件"""
+    
+    def __init__(self, intent=None):
+        """
+        初始化 Intent 事件
+        
+        :param intent: Intent 对象或命令字符串
+        """
+        super().__init__(EventType.INTENT)
+        self.intent = intent
+    
+    def send(self, device) -> bool:
+        """发送 Intent 到 Android 设备"""
+        if self.intent is not None:
+            device.send_intent(self.intent)
+        return True
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "event_type": self.event_type.value,
+            "intent": str(self.intent) if self.intent else None
+        }
+    
+    def get_event_str(self, _state=None) -> str:
+        return f"Intent({self.intent})"
+
+
+class AndroidKillAppEvent(BaseKillAppEvent):
+    """Android 终止应用事件"""
+    
+    def send(self, device) -> bool:
+        """终止 Android 应用"""
+        if self.app is None:
+            return False
+        
+        package_name = (self.app.get_package_name() 
+                       if hasattr(self.app, 'get_package_name') 
+                       else self.app)
+        
+        device.adb.shell(f"am force-stop {package_name}")
+        return True

DroidBot/platforms/android/android_intent.py

@@ -0,0 +1,106 @@
+"""
+Android Intent Module
+Android-specific Intent class for constructing am (Activity Manager) commands.
+"""
+
+
+class AndroidIntent:
+    """
+    Android Intent 类 - 用于构建 am 命令
+    
+    此类仅在 Android 平台内部使用。
+    """
+
+    def __init__(self, prefix="start", action=None, data_uri=None, mime_type=None, category=None,
+                 component=None, flag=None, extra_keys=None, extra_string=None, extra_boolean=None,
+                 extra_int=None, extra_long=None, extra_float=None, extra_uri=None, extra_component=None,
+                 extra_array_int=None, extra_array_long=None, extra_array_float=None, flags=None, suffix=""):
+        self.event_type = 'intent'
+        self.prefix = prefix
+        self.action = action
+        self.data_uri = data_uri
+        self.mime_type = mime_type
+        self.category = category
+        self.component = component
+        self.flag = flag
+        self.extra_keys = extra_keys
+        self.extra_string = extra_string
+        self.extra_boolean = extra_boolean
+        self.extra_int = extra_int
+        self.extra_long = extra_long
+        self.extra_float = extra_float
+        self.extra_uri = extra_uri
+        self.extra_component = extra_component
+        self.extra_array_int = extra_array_int
+        self.extra_array_long = extra_array_long
+        self.extra_array_float = extra_array_float
+        self.flags = flags
+        self.suffix = suffix
+        self.cmd = None
+        self.get_cmd()
+
+    def get_cmd(self) -> str:
+        """
+        将 Intent 转换为 am 命令字符串
+        
+        :return: am 命令字符串
+        """
+        if self.cmd is not None:
+            return self.cmd
+        cmd = "am "
+        if self.prefix:
+            cmd += self.prefix
+        if self.action is not None:
+            cmd += " -a " + self.action
+        if self.data_uri is not None:
+            cmd += " -d " + self.data_uri
+        if self.mime_type is not None:
+            cmd += " -t " + self.mime_type
+        if self.category is not None:
+            cmd += " -c " + self.category
+        if self.component is not None:
+            cmd += " -n " + self.component
+        if self.flag is not None:
+            cmd += " -f " + self.flag
+        if self.extra_keys:
+            for key in self.extra_keys:
+                cmd += " --esn '%s'" % key
+        if self.extra_string:
+            for key in list(self.extra_string.keys()):
+                cmd += " -e '%s' '%s'" % (key, self.extra_string[key])
+        if self.extra_boolean:
+            for key in list(self.extra_boolean.keys()):
+                cmd += " -ez '%s' %s" % (key, self.extra_boolean[key])
+        if self.extra_int:
+            for key in list(self.extra_int.keys()):
+                cmd += " -ei '%s' %s" % (key, self.extra_int[key])
+        if self.extra_long:
+            for key in list(self.extra_long.keys()):
+                cmd += " -el '%s' %s" % (key, self.extra_long[key])
+        if self.extra_float:
+            for key in list(self.extra_float.keys()):
+                cmd += " -ef '%s' %s" % (key, self.extra_float[key])
+        if self.extra_uri:
+            for key in list(self.extra_uri.keys()):
+                cmd += " -eu '%s' '%s'" % (key, self.extra_uri[key])
+        if self.extra_component:
+            for key in list(self.extra_component.keys()):
+                cmd += " -ecn '%s' %s" % (key, self.extra_component[key])
+        if self.extra_array_int:
+            for key in list(self.extra_array_int.keys()):
+                cmd += " -eia '%s' %s" % (key, ",".join(self.extra_array_int[key]))
+        if self.extra_array_long:
+            for key in list(self.extra_array_long.keys()):
+                cmd += " -ela '%s' %s" % (key, ",".join(self.extra_array_long[key]))
+        if self.extra_array_float:
+            for key in list(self.extra_array_float.keys()):
+                cmd += " -efa '%s' %s" % (key, ",".join(self.extra_array_float[key]))
+        if self.flags:
+            cmd += " " + " ".join(self.flags)
+        if self.suffix:
+            cmd += " " + self.suffix
+        self.cmd = cmd
+        return self.cmd
+
+    def __str__(self) -> str:
+        return self.get_cmd()

DroidBot/platforms/android/utils.py

@@ -0,0 +1,32 @@
+import re
+import subprocess
+import hashlib
+
+def get_available_devices():
+    """
+    Get a list of device serials connected via adb
+    :return: list of str, each str is a device serial number
+    """
+    try:
+        r = subprocess.check_output(["adb", "devices"])
+    except Exception:
+        return []
+        
+    if not isinstance(r, str):
+        r = r.decode()
+    devices = []
+    for line in r.splitlines():
+        segs = line.strip().split()
+        if len(segs) == 2 and segs[1] == "device":
+            devices.append(segs[0])
+    return devices
+
+
+def md5(input_str):
+    """
+    Calculate MD5 hash of a string
+    
+    :param input_str: input string
+    :return: MD5 hex digest
+    """
+    return hashlib.md5(input_str.encode('utf-8')).hexdigest()

DroidBot/platforms/ios/__init__.py

@@ -0,0 +1,75 @@
+# iOS platform implementation
+"""
+iOS 平台模块
+
+提供 iOS 设备的完整支持，通过 WebDriverAgent (WDA) 进行设备控制。
+"""
+from .ios_device import IOSDevice
+from .ios_device_state import IOSDeviceState
+from .ios_input_event import (
+    IOSTouchEvent,
+    IOSLongTouchEvent,
+    IOSSwipeEvent,
+    IOSScrollEvent,
+    IOSSetTextEvent,
+    IOSKeyEvent,
+    IOSKillAppEvent,
+    IOSIntentEvent,
+)
+
+__all__ = [
+    'IOSDevice',
+    'IOSDeviceState',
+    'IOSTouchEvent',
+    'IOSLongTouchEvent',
+    'IOSSwipeEvent',
+    'IOSScrollEvent',
+    'IOSSetTextEvent',
+    'IOSKeyEvent',
+    'IOSKillAppEvent',
+    'IOSIntentEvent',
+]
+
+
+# Register iOS platform with the factory
+def register_ios_platform():
+    """Register iOS platform with PlatformFactory"""
+    from ...core.platform_factory import PlatformFactory, Platform
+    from .ios_device import IOSDevice
+    from .ios_device_state import IOSDeviceState
+    from .ios_input_event import (
+        IOSTouchEvent,
+        IOSLongTouchEvent,
+        IOSSwipeEvent,
+        IOSScrollEvent,
+        IOSSetTextEvent,
+        IOSKeyEvent,
+        IOSKillAppEvent,
+        IOSIntentEvent,
+    )
+    
+    event_classes = {
+        'touch': IOSTouchEvent,
+        'long_touch': IOSLongTouchEvent,
+        'swipe': IOSSwipeEvent,
+        'scroll': IOSScrollEvent,
+        'set_text': IOSSetTextEvent,
+        'key': IOSKeyEvent,
+        'kill_app': IOSKillAppEvent,
+        'intent': IOSIntentEvent,
+    }
+    
+    PlatformFactory.register_platform(
+        Platform.IOS,
+        IOSDevice,
+        IOSDeviceState,
+        event_classes
+    )
+
+
+# Auto-register on import
+try:
+    register_ios_platform()
+except ImportError:
+    # Platform classes may not be fully initialized yet
+    pass

DroidBot/platforms/ios/ios_app.py

@@ -0,0 +1,45 @@
+"""
+iOS App Module
+iOS-specific application model.
+"""
+import logging
+from typing import Optional
+
+
+class IOSApp:
+    """
+    iOS 应用类 - 用于管理 iOS 应用信息
+    
+    此类仅供 IOSDevice 内部使用，不作为公共接口暴露。
+    """
+
+    def __init__(self, bundle_id: str, output_dir: Optional[str] = None):
+        """
+        创建 IOSApp 实例
+        
+        :param bundle_id: 应用的 Bundle ID
+        :param output_dir: 输出目录路径
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        
+        self.bundle_id = bundle_id
+        self.output_dir = output_dir
+        
+        # App metadata (populated when connected to device)
+        self.app_name: Optional[str] = None
+        self.version: Optional[str] = None
+        
+    @property
+    def identifier(self) -> str:
+        """获取应用唯一标识符（bundle_id）"""
+        return self.bundle_id
+    
+    def get_bundle_id(self) -> str:
+        """获取应用的 Bundle ID"""
+        return self.bundle_id
+    
+    def __str__(self) -> str:
+        return f"IOSApp(bundle_id={self.bundle_id})"
+    
+    def __repr__(self) -> str:
+        return self.__str__()

DroidBot/platforms/ios/ios_device_state.py

@@ -0,0 +1,544 @@
+"""
+iOS Device State Implementation
+Concrete implementation of AbstractDeviceState for iOS devices.
+"""
+import copy
+import hashlib
+import os
+from typing import Optional, Dict, Any, List, Set
+
+from ...core.abstract_device_state import AbstractDeviceState
+from ...core.abstract_input_event import EventType
+
+
+class IOSDeviceState(AbstractDeviceState):
+    """
+    iOS 设备状态的具体实现
+    
+    通过 WDA 获取的界面信息构建设备状态。
+    """
+    
+    def __init__(self, device, views: List[Dict[str, Any]], 
+                 foreground_page: str = None, screenshot_path: str = None, tag: str = None):
+        """
+        初始化 iOS 设备状态
+        
+        :param device: IOSDevice 实例
+        :param views: 视图列表（ViewDict 格式）
+        :param foreground_page: 前台应用/页面标识
+        :param screenshot_path: 截图路径
+        :param tag: 状态标签
+        """
+        super().__init__(device, tag=tag, screenshot_path=screenshot_path)
+        
+        self._views = views or []
+        self._foreground_page = foreground_page
+        self._view_tree = {}
+        
+        # 缓存
+        self._state_str_cache = None
+        self._structure_str_cache = None
+        
+        # 需要过滤的视图 ID 集合
+        self._status_bar_ids: Set[int] = set()
+        self._off_screen_ids: Set[int] = set()  # 屏幕外视图（ScrollView 中不可见部分）
+        
+        # 解析视图并生成标识
+        self._parse_views()
+    
+    # ==================== 视图信息 ====================
+    
+    @property
+    def views(self) -> List[Dict[str, Any]]:
+        """获取视图列表"""
+        return self._views
+    
+    @property
+    def view_tree(self) -> Dict[str, Any]:
+        """获取视图树"""
+        return self._view_tree
+    
+    @property
+    def foreground_page(self) -> Optional[str]:
+        """获取前台页面标识"""
+        return self._foreground_page
+    
+    @property
+    def foreground_activity(self) -> Optional[str]:
+        """
+        获取前台活动标识（兼容 Android 接口）
+        
+        iOS 没有 Activity 概念，使用 foreground_page (bundle_id) 代替
+        """
+        return self._foreground_page
+    
+    @property
+    def search_content(self) -> str:
+        """
+        获取用于搜索的内容（兼容 UTG 接口）
+        
+        收集所有视图的文本内容用于搜索，过滤状态栏视图
+        """
+        texts = []
+        for view in self._views:
+            # 跳过状态栏视图及其子元素、屏幕外视图
+            if view.get("temp_id") in self._status_bar_ids:
+                continue
+            if view.get("temp_id") in self._off_screen_ids:
+                continue
+            text = view.get("text", "")
+            if text:
+                texts.append(text)
+            content_desc = view.get("content_description", "")
+            if content_desc and content_desc != text:
+                texts.append(content_desc)
+        return " ".join(texts)
+    
+    # ==================== 状态标识 ====================
+    
+    @property
+    def state_str(self) -> str:
+        """获取状态唯一标识"""
+        if self._state_str_cache is None:
+            self._state_str_cache = self._generate_state_str()
+        return self._state_str_cache
+    
+    @property
+    def structure_str(self) -> str:
+        """获取结构标识（忽略内容）"""
+        if self._structure_str_cache is None:
+            self._structure_str_cache = self._generate_structure_str()
+        return self._structure_str_cache
+    
+    # ==================== 输入事件 ====================
+    
+    def get_possible_input(self) -> List['AbstractInputEvent']:
+        """获取可能的输入事件列表"""
+        # 缓存机制：如果已经计算过，直接返回缓存
+        # 可能导致一直不更新，先禁用
+        # if self._possible_events:
+        #     return [] + self._possible_events
+        
+        from .ios_input_event import (
+            IOSTouchEvent, IOSLongTouchEvent, IOSSwipeEvent, 
+            IOSScrollEvent, IOSSetTextEvent, IOSKeyEvent
+        )
+        
+        possible_events = []
+        enabled_view_ids = []
+        touch_exclude_view_ids = set()
+        
+        # 预筛选：收集所有 enabled 且 visible 的视图 ID，排除屏幕外视图
+        for view_dict in self._views:
+            if view_dict.get('temp_id') in self._off_screen_ids:
+                continue
+            if (self._safe_dict_get(view_dict, 'enabled') and
+                self._safe_dict_get(view_dict, 'visible')):
+                # 检查边界是否有效
+                bounds = view_dict.get("bounds", [[0, 0], [0, 0]])
+                if bounds[0][0] < bounds[1][0] and bounds[0][1] < bounds[1][1]:
+                    enabled_view_ids.append(view_dict['temp_id'])
+        
+        # 第一轮：clickable 元素 -> 点击事件
+        # 并排除其所有子元素
+        for view_id in enabled_view_ids:
+            if self._safe_dict_get(self._views[view_id], 'clickable'):
+                view = self._views[view_id]
+                bounds = view.get("bounds", [[0, 0], [0, 0]])
+                center_x = (bounds[0][0] + bounds[1][0]) // 2
+                center_y = (bounds[0][1] + bounds[1][1]) // 2
+                possible_events.append(IOSTouchEvent(x=center_x, y=center_y, view=view))
+                touch_exclude_view_ids.add(view_id)
+                # 排除所有子元素
+                touch_exclude_view_ids = touch_exclude_view_ids.union(
+                    set(self.get_all_children(self._views[view_id]))
+                )
+        
+        # 添加通用滚动事件（类似 Android）
+        # 但排除系统界面，避免在系统弹窗上执行无意义的滚动
+        if not self._is_system_ui():
+            possible_events.append(IOSScrollEvent(
+                start_x=self.width // 2, start_y=self.height // 2 + 100,
+                end_x=self.width // 2, end_y=self.height // 2 - 100,
+                direction="up"
+            ))
+            # possible_events.append(IOSScrollEvent(
+            #     start_x=self.width // 2, start_y=self.height // 2 + 100,
+            #     end_x=self.width // 2, end_y=self.height // 2 - 100,
+            #     direction="up"
+            # ))
+            # possible_events.append(IOSScrollEvent(
+            #     start_x=self.width // 2, start_y=self.height // 2 - 100,
+            #     end_x=self.width // 2, end_y=self.height // 2 + 100,
+            #     direction="down"
+            # ))
+        
+        # 第二轮：iOS 特定的 scrollable 元素 -> 滚动事件
+        for view_id in enabled_view_ids:
+            if self._safe_dict_get(self._views[view_id], 'scrollable'):
+                view = self._views[view_id]
+                bounds = view.get("bounds", [[0, 0], [0, 0]])
+                center_x = (bounds[0][0] + bounds[1][0]) // 2
+                center_y = (bounds[0][1] + bounds[1][1]) // 2
+                
+                possible_events.append(IOSScrollEvent(
+                    start_x=center_x, start_y=center_y + 100,
+                    end_x=center_x, end_y=center_y - 100,
+                    direction="up", view=view
+                ))
+                possible_events.append(IOSScrollEvent(
+                    start_x=center_x, start_y=center_y - 100,
+                    end_x=center_x, end_y=center_y + 100,
+                    direction="down", view=view
+                ))
+        
+        # 第三轮：长按事件（iOS 特定类型）
+        for view_id in enabled_view_ids:
+            view = self._views[view_id]
+            class_name = view.get("class_name", "")
+            supports_long_press = class_name in ["Cell", "CollectionView", "TableView", "Link"]
+            
+            if self._safe_dict_get(view, 'clickable') and supports_long_press:
+                bounds = view.get("bounds", [[0, 0], [0, 0]])
+                center_x = (bounds[0][0] + bounds[1][0]) // 2
+                center_y = (bounds[0][1] + bounds[1][1]) // 2
+                possible_events.append(IOSLongTouchEvent(x=center_x, y=center_y, view=view))
+        
+        # 第四轮：editable 元素 -> 输入事件
+        for view_id in enabled_view_ids:
+            if self._safe_dict_get(self._views[view_id], 'editable'):
+                possible_events.append(IOSSetTextEvent(
+                    text="cat", view=self._views[view_id]
+                ))
+                touch_exclude_view_ids.add(view_id)
+        
+        # 第五轮：叶子节点兜底（没有子元素且未被处理）
+        # 过滤明显不可交互的类型，避免生成无效点击事件
+        NON_INTERACTIVE_TYPES = ['StaticText', 'Image', 'Icon', 'Other', 'Indicator', 'PageIndicator']
+        if len(possible_events) == 1:   # 仅有1个手动添加的滚动事件
+            for view_id in enabled_view_ids:
+                if view_id in touch_exclude_view_ids:
+                    continue
+                children = self._safe_dict_get(self._views[view_id], 'children')
+                if children and len(children) > 0:
+                    continue
+                
+                view = self._views[view_id]
+                class_name = view.get("class_name", "")
+                
+                # 跳过明显不可交互的元素类型
+                if any(non_interactive in class_name for non_interactive in NON_INTERACTIVE_TYPES):
+                    continue
+                
+                bounds = view.get("bounds", [[0, 0], [0, 0]])
+                center_x = (bounds[0][0] + bounds[1][0]) // 2
+                center_y = (bounds[0][1] + bounds[1][1]) // 2
+                possible_events.append(IOSTouchEvent(x=center_x, y=center_y, view=view))
+        
+        # 缓存结果
+        # self._possible_events = possible_events
+        return [] + possible_events
+    
+    @staticmethod
+    def _safe_dict_get(view_dict: Dict[str, Any], key: str, default=None):
+        """安全获取字典值"""
+        value = view_dict.get(key, None)
+        return value if value is not None else default
+    
+    # ==================== 序列化 ====================
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """序列化为字典"""
+        return {
+            "state_str": self.state_str,
+            "foreground_page": self._foreground_page,
+            "views": self._views,
+            "tag": self.tag,
+            "screenshot_path": self.screenshot_path,
+            "width": self.width,
+            "height": self.height,
+        }
+    
+    # ==================== 内部方法 ====================
+    
+    def _parse_views(self) -> None:
+        """解析视图，构建视图树"""
+        if not self._views:
+            return
+        
+        # 收集需要过滤的视图 ID
+        self._collect_status_bar_ids()
+        self._collect_off_screen_ids()
+        
+        # 找到根视图（parent == -1 的视图）
+        root_views = [v for v in self._views if v.get("parent", -1) == -1]
+        if root_views:
+            self._view_tree = self._build_tree(root_views[0])
+    
+    def _collect_status_bar_ids(self) -> None:
+        """收集 StatusBar 类型视图及其所有子元素的 temp_id"""
+        for view in self._views:
+            # 原始 WDA 的 type 字段在 _parse_wda_source 中被映射为 class_name
+            if view.get("class_name") == "StatusBar":
+                status_bar_id = view.get("temp_id")
+                if status_bar_id is not None:
+                    self._status_bar_ids.add(status_bar_id)
+                    # 递归收集所有子元素 ID
+                    self._status_bar_ids.update(self.get_all_children(view))
+    
+    def _collect_off_screen_ids(self) -> None:
+        """
+        收集屏幕外视图的 temp_id
+        
+        WDA source() 返回的 rect 坐标是可滚动内容的绝对位置，
+        超出屏幕可见范围的视图不应参与事件生成和状态标识计算。
+        判定标准：视图 bounds 完全位于屏幕可见区域之外。
+        """
+        screen_w = self.width
+        screen_h = self.height
+        
+        for view in self._views:
+            bounds = view.get("bounds", [[0, 0], [0, 0]])
+            x1, y1 = bounds[0]  # 左上角
+            x2, y2 = bounds[1]  # 右下角
+            
+            # 视图完全在屏幕外（不与屏幕可见区域有任何交集）
+            if x2 <= 0 or x1 >= screen_w or y2 <= 0 or y1 >= screen_h:
+                view_id = view.get("temp_id")
+                if view_id is not None:
+                    self._off_screen_ids.add(view_id)
+    
+    def _build_tree(self, view: Dict) -> Dict:
+        """递归构建视图树"""
+        tree = copy.copy(view)
+        children_ids = view.get("children", [])
+        tree["children"] = []
+        
+        for child_id in children_ids:
+            child_view = self._get_view_by_id(child_id)
+            if child_view:
+                tree["children"].append(self._build_tree(child_view))
+        
+        return tree
+    
+    def _get_view_by_id(self, temp_id: int) -> Optional[Dict]:
+        """根据 temp_id 获取视图"""
+        for view in self._views:
+            if view.get("temp_id") == temp_id:
+                return view
+        return None
+    
+    def _generate_state_str(self) -> str:
+        """生成状态唯一标识（过滤状态栏视图）"""
+        # 收集所有视图的签名，跳过状态栏
+        view_signatures = []
+        for view in self._views:
+            if view.get("temp_id") in self._status_bar_ids:
+                continue
+            if view.get("temp_id") in self._off_screen_ids:
+                continue
+            sig = self._get_view_signature(view)
+            view_signatures.append(sig)
+        
+        # 组合前台页面和视图签名
+        state_content = f"{self._foreground_page or ''}_{'_'.join(sorted(view_signatures))}"
+        
+        return hashlib.md5(state_content.encode()).hexdigest()
+    
+    def _generate_structure_str(self) -> str:
+        """生成结构标识（忽略文本内容，过滤状态栏视图）"""
+        view_signatures = []
+        for view in self._views:
+            if view.get("temp_id") in self._status_bar_ids:
+                continue
+            if view.get("temp_id") in self._off_screen_ids:
+                continue
+            sig = self._get_structure_signature(view)
+            view_signatures.append(sig)
+        
+        state_content = f"{self._foreground_page or ''}_{'_'.join(sorted(view_signatures))}"
+        
+        return hashlib.md5(state_content.encode()).hexdigest()
+    
+    @staticmethod
+    def _get_view_signature(view: Dict) -> str:
+        """获取视图签名（包含内容）"""
+        parts = [
+            view.get("class_name", ""),
+            view.get("resource_id", ""),
+            view.get("text", "")[:50] if view.get("text") else "",
+            str(view.get("bounds", [])),
+        ]
+        content = "_".join(filter(None, parts))
+        return hashlib.md5(content.encode()).hexdigest()[:8]
+    
+    @staticmethod
+    def _get_structure_signature(view: Dict) -> str:
+        """获取结构签名（忽略文本内容）"""
+        parts = [
+            view.get("class_name", ""),
+            view.get("resource_id", ""),
+            str(view.get("bounds", [])),
+        ]
+        content = "_".join(filter(None, parts))
+        return hashlib.md5(content.encode()).hexdigest()[:8]
+    
+    # ==================== 辅助方法 ====================
+    
+    def get_all_ancestors(self, view_dict: Dict[str, Any]) -> List[int]:
+        """获取所有祖先节点 ID"""
+        ancestors = []
+        parent_id = view_dict.get("parent", -1)
+        while parent_id >= 0:
+            ancestors.append(parent_id)
+            parent_view = self._get_view_by_id(parent_id)
+            if parent_view:
+                parent_id = parent_view.get("parent", -1)
+            else:
+                break
+        return ancestors
+    
+    def get_all_children(self, view_dict: Dict[str, Any]) -> List[int]:
+        """获取所有子节点 ID（递归）"""
+        all_children = []
+        children_ids = view_dict.get("children", [])
+        
+        for child_id in children_ids:
+            all_children.append(child_id)
+            child_view = self._get_view_by_id(child_id)
+            if child_view:
+                all_children.extend(self.get_all_children(child_view))
+        
+        return all_children
+    
+    def _is_system_ui(self) -> bool:
+        """
+        检查当前是否是系统界面
+        
+        Returns:
+            bool: True 表示系统界面（如 springboard），False 表示应用界面
+        """
+        return self._foreground_page == "com.apple.springboard"
+    
+    def get_app_page_depth(self) -> int:
+        """获取应用页面深度（iOS 不支持，返回 -1）"""
+        # iOS 没有 Android 那样的 Activity 栈概念
+        return -1
+
+
+if __name__ == "__main__":
+    """
+    测试屏幕外视图过滤效果
+    用法: cd autool && python -m DroidBot.platforms.ios.ios_device_state
+    """
+    import json
+    import sys
+    import os
+
+    # 测试用的两个状态文件
+    TEST_FILES = [
+        ("State1（欢迎页）", "output/ios_test_20260316_103839/co_vulcanlabs_moodtracker_iOS_20260316_103846/droidbot/states/state_2026-03-16_103924.json"),
+        ("State2（评论列表页）", "output/ios_test_20260316_103839/co_vulcanlabs_moodtracker_iOS_20260316_103846/droidbot/states/state_2026-03-16_104804.json"),
+    ]
+
+    # Mock device 对象，提供 width/height
+    class MockDevice:
+        def __init__(self, width, height):
+            self._width = width
+            self._height = height
+            self.output_dir = None
+        def get_width(self):
+            return self._width
+        def get_height(self):
+            return self._height
+        def get_display_info(self):
+            return {"width": self._width, "height": self._height}
+
+    # 项目根目录
+    project_root = os.path.dirname(os.path.abspath(__file__))
+    while not os.path.exists(os.path.join(project_root, "output")) and project_root != "/":
+        project_root = os.path.dirname(project_root)
+
+    for label, rel_path in TEST_FILES:
+        json_path = os.path.join(project_root, rel_path)
+        if not os.path.exists(json_path):
+            print(f"⚠️  {label}: 文件不存在 {json_path}")
+            continue
+
+        data = json.load(open(json_path))
+        views = data["views"]
+        width = data.get("width", 390)
+        height = data.get("height", 844)
+        foreground_page = data.get("foreground_page", "")
+
+        # 创建 Mock device 并构建 IOSDeviceState
+        device = MockDevice(width, height)
+        state = IOSDeviceState(
+            device=device,
+            views=views,
+            foreground_page=foreground_page,
+        )
+
+        # 统计信息
+        total_views = len(views)
+        off_screen_count = len(state._off_screen_ids)
+        status_bar_count = len(state._status_bar_ids)
+        on_screen_count = total_views - off_screen_count - status_bar_count
+
+        print(f"\n{'='*60}")
+        print(f"📱 {label}")
+        print(f"{'='*60}")
+        print(f"屏幕尺寸: {width}x{height}")
+        print(f"前台页面: {foreground_page}")
+        print(f"总视图数: {total_views}")
+        print(f"  状态栏视图: {status_bar_count}")
+        print(f"  屏幕外视图: {off_screen_count}")
+        print(f"  有效视图:   {on_screen_count}")
+
+        # 获取 possible input 事件
+        events = state.get_possible_input()
+        touch_events = [e for e in events if e.__class__.__name__ == "IOSTouchEvent"]
+        long_touch_events = [e for e in events if e.__class__.__name__ == "IOSLongTouchEvent"]
+        scroll_events = [e for e in events if e.__class__.__name__ == "IOSScrollEvent"]
+        set_text_events = [e for e in events if e.__class__.__name__ == "IOSSetTextEvent"]
+
+        print(f"\n📋 可执行事件 (共 {len(events)} 个):")
+        print(f"  Touch:     {len(touch_events)}")
+        print(f"  LongTouch: {len(long_touch_events)}")
+        print(f"  Scroll:    {len(scroll_events)}")
+        print(f"  SetText:   {len(set_text_events)}")
+
+        # 验证所有 touch 事件坐标在屏幕范围内
+        out_of_bounds = []
+        for e in touch_events + long_touch_events:
+            if e.x < 0 or e.x > width or e.y < 0 or e.y > height:
+                out_of_bounds.append(e)
+
+        if out_of_bounds:
+            print(f"\n❌ 发现 {len(out_of_bounds)} 个坐标越界事件:")
+            for e in out_of_bounds[:5]:
+                print(f"  {e.get_event_str()}")
+        else:
+            print(f"\n✅ 所有触摸/长按事件坐标均在屏幕范围内")
+
+        # 打印 touch 事件详情
+        print(f"\n📍 Touch 事件列表:")
+        for e in touch_events:
+            view = e.view
+            class_name = view.get("class_name", "?") if view else "?"
+            text = (view.get("text", "") or "")[:30] if view else ""
+            text_str = f' text="{text}"' if text else ""
+            print(f"  ({e.x:4d}, {e.y:4d}) [{class_name}]{text_str}")
+
+        # 打印 long touch 事件详情
+        if long_touch_events:
+            print(f"\n📍 LongTouch 事件列表:")
+            for e in long_touch_events:
+                view = e.view
+                class_name = view.get("class_name", "?") if view else "?"
+                text = (view.get("text", "") or "")[:30] if view else ""
+                text_str = f' text="{text}"' if text else ""
+                print(f"  ({e.x:4d}, {e.y:4d}) [{class_name}]{text_str}")
+
+        print(f"\n🔑 state_str: {state.state_str}")
+        print(f"🔑 structure_str: {state.structure_str}")

DroidBot/platforms/ios/ios_input_event.py

@@ -0,0 +1,303 @@
+"""
+iOS Input Event Implementation
+iOS-specific input event classes that implement AbstractInputEvent.
+"""
+from typing import Optional, Dict, Any
+
+from ...core.abstract_input_event import (
+    AbstractInputEvent, 
+    BaseTouchEvent,
+    BaseLongTouchEvent,
+    BaseSwipeEvent,
+    BaseScrollEvent,
+    BaseSetTextEvent,
+    BaseKeyEvent,
+    BaseKillAppEvent,
+    EventType
+)
+
+
+class IOSTouchEvent(BaseTouchEvent):
+
+    """iOS 触摸/点击事件"""
+    
+    def __init__(self, x: int, y: int, view: Dict[str, Any] = None):
+        """
+        初始化触摸事件
+        
+        :param x: 触摸 X 坐标
+        :param y: 触摸 Y 坐标
+        :param view: 关联的视图字典（可选）
+        """
+        super().__init__(x=x, y=y, view=view)
+    
+    def send(self, device) -> bool:
+        """发送事件到设备"""
+        try:
+            device.view_touch(self.x, self.y)
+            return True
+        except Exception as e:
+            device.logger.warning(f"Failed to send touch event: {e}")
+            return False
+    
+    def get_event_str(self, state=None) -> str:
+        return f"IOSTouchEvent(x={self.x}, y={self.y})"
+
+
+class IOSLongTouchEvent(BaseLongTouchEvent):
+    """iOS 长按事件"""
+    
+    def __init__(self, x: int, y: int, duration: float = 2.0, view: Dict[str, Any] = None):
+        """
+        初始化长按事件
+        
+        :param x: 长按 X 坐标
+        :param y: 长按 Y 坐标
+        :param duration: 长按持续时间（秒）
+        :param view: 关联的视图字典（可选）
+        """
+        super().__init__(x=x, y=y, duration=duration, view=view)
+    
+    def send(self, device) -> bool:
+        """发送事件到设备"""
+        try:
+            device.view_long_touch(self.x, self.y, self.duration)
+            return True
+        except Exception as e:
+            device.logger.warning(f"Failed to send long touch event: {e}")
+            return False
+    
+    def get_event_str(self, state=None) -> str:
+        return f"IOSLongTouchEvent(x={self.x}, y={self.y}, duration={self.duration})"
+
+
+class IOSSwipeEvent(BaseSwipeEvent):
+    """iOS 滑动事件"""
+    
+    def __init__(self, start_x: int, start_y: int, end_x: int, end_y: int,
+                 duration: float = 0.5, view: Dict[str, Any] = None):
+        """
+        初始化滑动事件
+        
+        :param start_x: 起始 X 坐标
+        :param start_y: 起始 Y 坐标
+        :param end_x: 结束 X 坐标
+        :param end_y: 结束 Y 坐标
+        :param duration: 滑动持续时间（秒）
+        :param view: 关联的视图字典（可选）
+        """
+        super().__init__(
+            start_x=start_x, start_y=start_y,
+            end_x=end_x, end_y=end_y,
+            duration=duration, view=view
+        )
+    
+    def send(self, device) -> bool:
+        """发送事件到设备"""
+        try:
+            device.view_drag(
+                (self.start_x, self.start_y),
+                (self.end_x, self.end_y),
+                self.duration
+            )
+            return True
+        except Exception as e:
+            device.logger.warning(f"Failed to send swipe event: {e}")
+            return False
+    
+    def get_event_str(self, state=None) -> str:
+        return f"IOSSwipeEvent(({self.start_x},{self.start_y})->({self.end_x},{self.end_y}))"
+
+
+class IOSScrollEvent(BaseScrollEvent):
+    """iOS 滚动事件"""
+    
+    def __init__(self, start_x: int, start_y: int, end_x: int, end_y: int,
+                 direction: str = "down", view: Dict[str, Any] = None):
+        """
+        初始化滚动事件
+        
+        :param start_x: 起始 X 坐标
+        :param start_y: 起始 Y 坐标
+        :param end_x: 结束 X 坐标
+        :param end_y: 结束 Y 坐标
+        :param direction: 滚动方向 (up/down/left/right)
+        :param view: 关联的视图字典（可选）
+        """
+        super().__init__(direction=direction, view=view)
+        self.start_x = start_x
+        self.start_y = start_y
+        self.end_x = end_x
+        self.end_y = end_y
+    
+    def send(self, device) -> bool:
+        """发送事件到设备"""
+        try:
+            device.view_drag(
+                (self.start_x, self.start_y),
+                (self.end_x, self.end_y),
+                0.3  # 滚动通常比较快
+            )
+            return True
+        except Exception as e:
+            device.logger.warning(f"Failed to send scroll event: {e}")
+            return False
+    
+    def get_event_str(self, state=None) -> str:
+        return f"IOSScrollEvent(direction={self.direction})"
+
+
+class IOSSetTextEvent(BaseSetTextEvent):
+    """iOS 文本输入事件"""
+    
+    def __init__(self, text: str, view: Dict[str, Any] = None):
+        """
+        初始化文本输入事件
+        
+        :param text: 要输入的文本
+        :param view: 关联的视图字典（可选）
+        """
+        super().__init__(text=text, view=view)
+    
+    def send(self, device) -> bool:
+        """发送事件到设备"""
+        try:
+            # 如果有关联视图，先点击激活
+            if self.view:
+                bounds = self.view.get("bounds", [[0, 0], [0, 0]])
+                center_x = (bounds[0][0] + bounds[1][0]) // 2
+                center_y = (bounds[0][1] + bounds[1][1]) // 2
+                device.view_touch(center_x, center_y)
+                import time
+                time.sleep(0.5)
+            
+            device.view_set_text(self.text)
+            return True
+        except Exception as e:
+            device.logger.warning(f"Failed to send set text event: {e}")
+            return False
+    
+    def get_event_str(self, state=None) -> str:
+        text_preview = self.text[:20] + "..." if len(self.text) > 20 else self.text
+        return f"IOSSetTextEvent(text='{text_preview}')"
+
+
+class IOSKeyEvent(BaseKeyEvent):
+    """iOS 按键事件"""
+    
+    # iOS 支持的按键
+    SUPPORTED_KEYS = {
+        "HOME": "home",
+        "BACK": "home",  # iOS 无返回键，用 Home 代替
+        "VOLUME_UP": "volumeUp",
+        "VOLUME_DOWN": "volumeDown",
+    }
+    
+    def __init__(self, key_name: str = None, key_code: str = None):
+        """
+        初始化按键事件
+        
+        :param key_name: 按键名称 (兼容参数，与基类一致)
+        :param key_code: 按键代码 (HOME, VOLUME_UP, VOLUME_DOWN)
+        """
+        # 兼容两种参数名
+        key = key_name or key_code or "HOME"
+        super().__init__(key_name=key)
+        self.key_code = key  # 保留 key_code 供 send 方法使用
+    
+    def send(self, device) -> bool:
+        """发送事件到设备"""
+        try:
+            device.key_press(self.key_name)
+            return True
+        except Exception as e:
+            device.logger.warning(f"Failed to send key event: {e}")
+            return False
+    
+    def get_event_str(self, state=None) -> str:
+        return f"IOSKeyEvent(key={self.key_name})"
+
+
+class IOSKillAppEvent(BaseKillAppEvent):
+    """iOS 终止应用事件"""
+    
+    def __init__(self, app: str = None, bundle_id: str = None):
+        """
+        初始化终止应用事件
+        
+        :param app: 要终止的应用（兼容参数，与基类一致）
+        :param bundle_id: 要终止的应用 Bundle ID（None 表示当前应用）
+        """
+        # 兼容两种参数名
+        app_id = app or bundle_id
+        super().__init__(app=app_id)
+        self.bundle_id = app_id  # 保留 bundle_id 供 send 方法使用
+    
+    def send(self, device) -> bool:
+        """发送事件到设备"""
+        try:
+            # 等待 WDA 就绪
+            if hasattr(device, '_wait_wda_ready'):
+                device._wait_wda_ready()
+            
+            bundle_id = self.bundle_id or device.bundle_id
+            if bundle_id and device._wda_client:
+                device._wda_client.app_terminate(bundle_id)
+                return True
+            return False
+        except Exception as e:
+            device.logger.warning(f"Failed to kill app: {e}")
+            return False
+    
+    def get_event_str(self, state=None) -> str:
+        return f"IOSKillAppEvent(bundle_id={self.bundle_id})"
+
+
+class IOSIntentEvent(AbstractInputEvent):
+    """
+    iOS Intent 事件（适配 Android Intent 概念）
+    
+    iOS 没有真正的 Intent，此类用于兼容 input_policy 中的应用启动逻辑。
+    """
+    
+    def __init__(self, intent=None, bundle_id: str = None):
+        """
+        初始化 Intent 事件
+        
+        :param intent: 兼容参数（iOS 中忽略）
+        :param bundle_id: 要启动的应用 Bundle ID
+        """
+        super().__init__(event_type=EventType.INTENT)
+        self.intent = intent
+        self.bundle_id = bundle_id
+    
+    def send(self, device) -> bool:
+        """发送事件到设备"""
+        try:
+            # 等待 WDA 就绪
+            if hasattr(device, '_wait_wda_ready'):
+                device._wait_wda_ready()
+            
+            # 从 intent 或 bundle_id 确定目标应用
+            target_bundle_id = self.bundle_id or device.bundle_id
+            
+            if target_bundle_id and device._wda_client:
+                # 使用 WDA 启动应用
+                device._wda_client.app_launch(target_bundle_id)
+                import time
+                time.sleep(1)
+                return True
+            return False
+        except Exception as e:
+            device.logger.warning(f"Failed to launch app via intent: {e}")
+            return False
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "event_type": self.event_type.value,
+            "bundle_id": self.bundle_id,
+            "intent": str(self.intent) if self.intent else None,
+        }
+    
+    def get_event_str(self, state=None) -> str:
+        return f"IOSIntentEvent(bundle_id={self.bundle_id})"

DroidBot/platforms/ios/wda/_proto.py

@@ -0,0 +1,42 @@
+#
+
+__all__ = ['AppiumSettings', 'AlertAction']
+
+
+import enum
+
+class AppiumSettings(str, enum.Enum):
+    """
+    {'boundElementsByIndex': False,
+    'shouldUseCompactResponses': True,
+    'mjpegServerFramerate': 10,
+    'snapshotMaxDepth': 50,
+    'screenshotOrientation': 'auto',
+    'activeAppDetectionPoint': '64.00,64.00',
+    'acceptAlertButtonSelector': '',
+    'snapshotTimeout': 15,
+    'elementResponseAttributes': 'type,label',
+    'keyboardPrediction': 0,
+    'screenshotQuality': 2,
+    'keyboardAutocorrection': 0,
+    'useFirstMatch': False,
+    'reduceMotion': False,
+    'defaultActiveApplication': 'auto',
+    'mjpegScalingFactor': 100,
+    'mjpegServerScreenshotQuality': 25,
+    'dismissAlertButtonSelector': '',
+    'includeNonModalElements': False}
+    """
+    AcceptAlertButtonSelector = "acceptAlertButtonSelector"
+    DismissAlertButtonSelector = "dismissAlertButtonSelector"
+
+
+
+# default_alert_accept_selector = "**/XCUIElementTypeButton[`label IN {'允许','好','仅在使用应用期间','暂不'}`]"
+# default_alert_dismiss_selector = "**/XCUIElementTypeButton[`label IN {'不允许','暂不'}`]"
+
+
+class AlertAction(str, enum.Enum):
+    ACCEPT = "accept"
+    DISMISS = "dismiss"
+

DroidBot/platforms/ios/wda/exceptions.py

@@ -0,0 +1,105 @@
+# coding: utf-8
+# author: codeskyblue
+
+import json
+
+
+JSONDecodeError = json.decoder.JSONDecodeError if hasattr(
+    json.decoder, "JSONDecodeError") else ValueError
+
+
+class MuxError(Exception):
+    """ Mutex error """
+
+
+class MuxConnectError(MuxError, ConnectionError):
+    """ Error when MessageType: Connect """
+
+
+class WDAError(Exception):
+    """ base wda error """
+
+
+class WDABadGateway(WDAError):
+    """ bad gateway """
+
+
+class WDAEmptyResponseError(WDAError):
+    """ response body is empty """
+
+
+class WDAElementNotFoundError(WDAError):
+    """ element not found """
+
+
+class WDAElementNotDisappearError(WDAError):
+    """ element not disappera """
+
+
+class WDARequestError(WDAError):
+    def __init__(self, status, value):
+        self.status = status
+        self.value = value
+
+    def __str__(self):
+        return 'WDARequestError(status=%d, value=%s)' % (self.status,
+                                                         self.value)
+
+
+class WDAKeyboardNotPresentError(WDARequestError):
+    # {'error': 'invalid element state', 
+    #  'message': 'Error Domain=com.facebook.WebDriverAgent Code=1 
+    #     "The on-screen keyboard must be present to send keys" 
+    #     UserInfo={NSLocalizedDescription=The on-screen keyboard must be present to send keys}',
+    #  'traceback': ''})
+
+    @staticmethod
+    def check(v: dict):
+        if v.get('error') == 'invalid element state' and \
+                'keyboard must be present to send keys' in v.get('message', ''):
+            return True
+        return False
+
+
+class WDAInvalidSessionIdError(WDARequestError):
+    """
+    "value" : {
+        "error" : "invalid session id",
+        "message" : "Session does not exist",
+    """
+    @staticmethod
+    def check(v: dict):
+        if v.get('error') == 'invalid session id':
+            return True
+        return False
+
+
+class WDAPossiblyCrashedError(WDARequestError):
+    @staticmethod
+    def check(v: dict):
+        if "possibly crashed" in v.get('message', ''):
+            return True
+        return False
+
+
+class WDAUnknownError(WDARequestError):
+    """ error: unknown error, message: *** - """
+    @staticmethod
+    def check(v: dict):
+        return v.get("error") == "unknown error"
+
+
+class WDAStaleElementReferenceError(WDARequestError):
+    """ error: 'stale element reference' """
+    @staticmethod
+    def check(v: dict):
+        return v.get("error") == 'stale element reference'
+
+
+class WDAStuckError(WDAError):
+    """WDA 卡死且多次恢复失败
+    
+    当 WDA 连续多次异步恢复都失败时抛出此异常，
+    表示需要终止当前测试任务。
+    """
+    pass

DroidBot/platforms/ios/wda/usbmux/__init__.py

@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+"""Created on Thu Dec 09 2021 09:56:30 by codeskyblue
+"""
+
+import json
+from http.client import HTTPConnection, HTTPSConnection, HTTPResponse
+from urllib.parse import urlparse
+
+from .exceptions import HTTPError, MuxConnectError, MuxError
+from .pyusbmux import select_device
+
+_DEFAULT_CHUNK_SIZE = 4096
+
+def http_create(url: str) -> HTTPConnection:
+    u = urlparse(url)
+    if u.scheme == "http+usbmux":
+        udid, device_wda_port = u.netloc.split(":")
+        device = select_device(udid)
+        return device.make_http_connection(int(device_wda_port))
+    elif u.scheme == "http":
+        return HTTPConnection(u.netloc)
+    elif u.scheme == "https":
+        return HTTPSConnection(u.netloc)
+    else:
+        raise ValueError(f"unknown scheme: {u.scheme}")
+
+
+class HTTPResponseWrapper:
+    def __init__(self, content: bytes, status_code: int):
+        self.content = content
+        self.status_code = status_code
+    
+    def json(self):
+        return json.loads(self.content)
+
+    @property
+    def text(self) -> str:
+        return self.content.decode("utf-8")
+
+    def getcode(self) -> int:
+        return self.status_code
+    
+
+def fetch(url: str, method="GET", data=None, timeout=None, chunk_size: int = _DEFAULT_CHUNK_SIZE) -> HTTPResponseWrapper:
+    """
+    thread safe http request
+
+    Raises:
+        HTTPError
+    """
+    try:
+        method = method.upper()
+        conn = http_create(url)
+        conn.timeout = timeout
+        u = urlparse(url)
+        urlpath = url[len(u.scheme) + len(u.netloc) + 3:]
+
+        if not data:
+            conn.request(method, urlpath)
+        else:
+            conn.request(method, urlpath, json.dumps(data), headers={"Content-Type": "application/json"})
+        response = conn.getresponse()
+        content = _read_response(response, chunk_size)
+        resp = HTTPResponseWrapper(content, response.status)
+        return resp
+    except Exception as e:
+        raise HTTPError(e)
+
+
+def _read_response(response:HTTPResponse, chunk_size: int = _DEFAULT_CHUNK_SIZE) -> bytearray:
+    content = bytearray()
+    while True:
+        chunk = response.read(chunk_size)
+        if len(chunk) == 0:
+            break
+        content.extend(chunk)
+    return content

DroidBot/platforms/ios/wda/usbmux/exceptions.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+"""Created on Tue Mar 05 2024 10:18:09 by codeskyblue
+
+Copy from https://github.com/doronz88/pymobiledevice3
+"""
+
+class NotPairedError(Exception):
+    pass
+
+
+class MuxError(Exception):
+    pass
+
+
+class MuxVersionError(MuxError):
+    pass
+
+
+class BadCommandError(MuxError):
+    pass
+
+
+class BadDevError(MuxError):
+    pass
+
+
+class MuxConnectError(MuxError):
+    pass
+
+
+class MuxConnectToUsbmuxdError(MuxConnectError):
+    pass
+
+
+class ArgumentError(Exception):
+    pass
+
+
+class HTTPError(Exception):
+    pass
+    

DroidBot/platforms/ios/wda/usbmux/pyusbmux.py

@@ -0,0 +1,485 @@
+"""
+Copy from https://github.com/doronz88/pymobiledevice3
+
+Add http.client.HTTPConnection
+"""
+import abc
+import plistlib
+import socket
+import sys
+import time
+from dataclasses import dataclass
+from http.client import HTTPConnection
+from typing import List, Mapping, Optional
+
+from construct import Const, CString, Enum, FixedSized, GreedyBytes, Int16ul, Int32ul, Padding, Prefixed, StreamError, \
+    Struct, Switch, this
+
+from .exceptions import BadCommandError, BadDevError, MuxConnectError, \
+    MuxConnectToUsbmuxdError, MuxError, MuxVersionError, NotPairedError
+
+usbmuxd_version = Enum(Int32ul,
+                       BINARY=0,
+                       PLIST=1,
+                       )
+
+usbmuxd_result = Enum(Int32ul,
+                      OK=0,
+                      BADCOMMAND=1,
+                      BADDEV=2,
+                      CONNREFUSED=3,
+                      BADVERSION=6,
+                      )
+
+usbmuxd_msgtype = Enum(Int32ul,
+                       RESULT=1,
+                       CONNECT=2,
+                       LISTEN=3,
+                       ADD=4,
+                       REMOVE=5,
+                       PAIRED=6,
+                       PLIST=8,
+                       )
+
+usbmuxd_header = Struct(
+    'version' / usbmuxd_version,  # protocol version
+    'message' / usbmuxd_msgtype,  # message type
+    'tag' / Int32ul,  # responses to this query will echo back this tag
+)
+
+usbmuxd_request = Prefixed(Int32ul, Struct(
+    'header' / usbmuxd_header,
+    'data' / Switch(this.header.message, {
+        usbmuxd_msgtype.CONNECT: Struct(
+            'device_id' / Int32ul,
+            'port' / Int16ul,  # TCP port number
+            'reserved' / Const(0, Int16ul),
+        ),
+        usbmuxd_msgtype.PLIST: GreedyBytes,
+    }),
+), includelength=True)
+
+usbmuxd_device_record = Struct(
+    'device_id' / Int32ul,
+    'product_id' / Int16ul,
+    'serial_number' / FixedSized(256, CString('ascii')),
+    Padding(2),
+    'location' / Int32ul
+)
+
+usbmuxd_response = Prefixed(Int32ul, Struct(
+    'header' / usbmuxd_header,
+    'data' / Switch(this.header.message, {
+        usbmuxd_msgtype.RESULT: Struct(
+            'result' / usbmuxd_result,
+        ),
+        usbmuxd_msgtype.ADD: usbmuxd_device_record,
+        usbmuxd_msgtype.REMOVE: Struct(
+            'device_id' / Int32ul,
+        ),
+        usbmuxd_msgtype.PLIST: GreedyBytes,
+    }),
+), includelength=True)
+
+
+        
+
+@dataclass
+class MuxDevice:
+    devid: int
+    serial: str
+    connection_type: str
+
+    def connect(self, port: int, usbmux_address: Optional[str] = None) -> socket.socket:
+        mux = create_mux(usbmux_address=usbmux_address)
+        try:
+            return mux.connect(self, port)
+        except:  # noqa: E722
+            mux.close()
+            raise
+
+    @property
+    def is_usb(self) -> bool:
+        return self.connection_type == 'USB'
+
+    @property
+    def is_network(self) -> bool:
+        return self.connection_type == 'Network'
+
+    def matches_udid(self, udid: str) -> bool:
+        return self.serial.replace('-', '') == udid.replace('-', '')
+
+    def make_http_connection(self, port: int) -> HTTPConnection:
+        return USBMuxHTTPConnection(self, port)
+
+
+class SafeStreamSocket:
+    """ wrapper to native python socket object to be used with construct as a stream """
+
+    def __init__(self, address, family):
+        self._offset = 0
+        self.sock = socket.socket(family, socket.SOCK_STREAM)
+        self.sock.connect(address)
+
+    def send(self, msg: bytes) -> int:
+        self._offset += len(msg)
+        self.sock.sendall(msg)
+        return len(msg)
+
+    def recv(self, size: int) -> bytes:
+        msg = b''
+        while len(msg) < size:
+            chunk = self.sock.recv(size - len(msg))
+            self._offset += len(chunk)
+            if not chunk:
+                raise MuxError('socket connection broken')
+            msg += chunk
+        return msg
+
+    def close(self) -> None:
+        self.sock.close()
+
+    def settimeout(self, interval: float) -> None:
+        self.sock.settimeout(interval)
+
+    def setblocking(self, blocking: bool) -> None:
+        self.sock.setblocking(blocking)
+
+    def tell(self) -> int:
+        return self._offset
+
+    read = recv
+    write = send
+
+
+class MuxConnection:
+    # used on Windows
+    ITUNES_HOST = ('127.0.0.1', 27015)
+
+    # used for macOS and Linux
+    USBMUXD_PIPE = '/var/run/usbmuxd'
+
+    @staticmethod
+    def create_usbmux_socket(usbmux_address: Optional[str] = None) -> SafeStreamSocket:
+        try:
+            if usbmux_address is not None:
+                if ':' in usbmux_address:
+                    # assume tcp address
+                    hostname, port = usbmux_address.split(':')
+                    port = int(port)
+                    address = (hostname, port)
+                    family = socket.AF_INET
+                else:
+                    # assume unix domain address
+                    address = usbmux_address
+                    family = socket.AF_UNIX
+            else:
+                if sys.platform in ['win32', 'cygwin']:
+                    address = MuxConnection.ITUNES_HOST
+                    family = socket.AF_INET
+                else:
+                    address = MuxConnection.USBMUXD_PIPE
+                    family = socket.AF_UNIX
+            return SafeStreamSocket(address, family)
+        except ConnectionRefusedError:
+            raise MuxConnectToUsbmuxdError()
+
+    @staticmethod
+    def create(usbmux_address: Optional[str] = None):
+        # first attempt to connect with possibly the wrong version header (plist protocol)
+        sock = MuxConnection.create_usbmux_socket(usbmux_address=usbmux_address)
+
+        message = usbmuxd_request.build({
+            'header': {'version': usbmuxd_version.PLIST, 'message': usbmuxd_msgtype.PLIST, 'tag': 1},
+            'data': plistlib.dumps({'MessageType': 'ReadBUID'})
+        })
+        sock.send(message)
+        response = usbmuxd_response.parse_stream(sock)
+
+        # if we sent a bad request, we should re-create the socket in the correct version this time
+        sock.close()
+        sock = MuxConnection.create_usbmux_socket(usbmux_address=usbmux_address)
+
+        if response.header.version == usbmuxd_version.BINARY:
+            return BinaryMuxConnection(sock)
+        elif response.header.version == usbmuxd_version.PLIST:
+            return PlistMuxConnection(sock)
+
+        raise MuxVersionError(f'usbmuxd returned unsupported version: {response.version}')
+
+    def __init__(self, sock: SafeStreamSocket):
+        self._sock = sock
+
+        # after initiating the "Connect" packet, this same socket will be used to transfer data into the service
+        # residing inside the target device. when this happens, we can no longer send/receive control commands to
+        # usbmux on same socket
+        self._connected = False
+
+        # message sequence number. used when verifying the response matched the request
+        self._tag = 1
+
+        self.devices = []
+
+    @abc.abstractmethod
+    def _connect(self, device_id: int, port: int):
+        """ initiate a "Connect" request to target port """
+        pass
+
+    @abc.abstractmethod
+    def get_device_list(self, timeout: float = None):
+        """
+        request an update to current device list
+        """
+        pass
+
+    def connect(self, device: MuxDevice, port: int) -> socket.socket:
+        """ connect to a relay port on target machine and get a raw python socket object for the connection """
+        self._connect(device.devid, socket.htons(port))
+        self._connected = True
+        return self._sock.sock
+
+    def close(self):
+        """ close current socket """
+        self._sock.close()
+
+    def _assert_not_connected(self):
+        """ verify active state is in state for control messages """
+        if self._connected:
+            raise MuxError('Mux is connected, cannot issue control packets')
+
+    def _raise_mux_exception(self, result: int, message: str = None):
+        exceptions = {
+            int(usbmuxd_result.BADCOMMAND): BadCommandError,
+            int(usbmuxd_result.BADDEV): BadDevError,
+            int(usbmuxd_result.CONNREFUSED): MuxConnectError,
+            int(usbmuxd_result.BADVERSION): MuxVersionError,
+        }
+        exception = exceptions.get(result, MuxError)
+        raise exception(message)
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.close()
+
+
+class BinaryMuxConnection(MuxConnection):
+    """ old binary protocol """
+
+    def __init__(self, sock: SafeStreamSocket):
+        super().__init__(sock)
+        self._version = usbmuxd_version.BINARY
+
+    def get_device_list(self, timeout: float = None):
+        """ use timeout to wait for the device list to be fully populated """
+        self._assert_not_connected()
+        end = time.time() + timeout
+        self.listen()
+        while time.time() < end:
+            self._sock.settimeout(end - time.time())
+            try:
+                self._receive_device_state_update()
+            except (BlockingIOError, StreamError):
+                continue
+            except IOError:
+                try:
+                    self._sock.setblocking(True)
+                    self.close()
+                except OSError:
+                    pass
+                raise MuxError('Exception in listener socket')
+
+    def listen(self):
+        """ start listening for events of attached and detached devices """
+        self._send_receive(usbmuxd_msgtype.LISTEN)
+
+    def _connect(self, device_id: int, port: int):
+        self._send({'header': {'version': self._version,
+                               'message': usbmuxd_msgtype.CONNECT,
+                               'tag': self._tag},
+                    'data': {'device_id': device_id, 'port': port},
+                    })
+        response = self._receive()
+        if response.header.message != usbmuxd_msgtype.RESULT:
+            raise MuxError(f'unexpected message type received: {response}')
+
+        if response.data.result != usbmuxd_result.OK:
+            raise self._raise_mux_exception(int(response.data.result),
+                                            f'failed to connect to device: {device_id} at port: {port}. reason: '
+                                            f'{response.data.result}')
+
+    def _send(self, data: Mapping):
+        self._assert_not_connected()
+        self._sock.send(usbmuxd_request.build(data))
+        self._tag += 1
+
+    def _receive(self, expected_tag: int = None):
+        self._assert_not_connected()
+        response = usbmuxd_response.parse_stream(self._sock)
+        if expected_tag and response.header.tag != expected_tag:
+            raise MuxError(f'Reply tag mismatch: expected {expected_tag}, got {response.header.tag}')
+        return response
+
+    def _send_receive(self, message_type: int):
+        self._send({'header': {'version': self._version, 'message': message_type, 'tag': self._tag},
+                    'data': b''})
+        response = self._receive(self._tag - 1)
+        if response.header.message != usbmuxd_msgtype.RESULT:
+            raise MuxError(f'unexpected message type received: {response}')
+
+        result = response.data.result
+        if result != usbmuxd_result.OK:
+            raise self._raise_mux_exception(int(result), f'{message_type} failed: error {result}')
+
+    def _add_device(self, device: MuxDevice):
+        self.devices.append(device)
+
+    def _remove_device(self, device_id: int):
+        self.devices = [device for device in self.devices if device.devid != device_id]
+
+    def _receive_device_state_update(self):
+        response = self._receive()
+        if response.header.message == usbmuxd_msgtype.ADD:
+            # old protocol only supported USB devices
+            self._add_device(MuxDevice(response.data.device_id, response.data.serial_number, 'USB'))
+        elif response.header.message == usbmuxd_msgtype.REMOVE:
+            self._remove_device(response.data.device_id)
+        else:
+            raise MuxError(f'Invalid packet type received: {response}')
+
+
+class PlistMuxConnection(BinaryMuxConnection):
+    def __init__(self, sock: SafeStreamSocket):
+        super().__init__(sock)
+        self._version = usbmuxd_version.PLIST
+
+    def listen(self) -> None:
+        self._send_receive({'MessageType': 'Listen'})
+
+    def get_pair_record(self, serial: str) -> Mapping:
+        # serials are saved inside usbmuxd without '-'
+        self._send({'MessageType': 'ReadPairRecord', 'PairRecordID': serial})
+        response = self._receive(self._tag - 1)
+        pair_record = response.get('PairRecordData')
+        if pair_record is None:
+            raise NotPairedError('device should be paired first')
+        return plistlib.loads(pair_record)
+
+    def get_device_list(self, timeout: float = None) -> None:
+        """ get device list synchronously without waiting the timeout """
+        self.devices = []
+        self._send({'MessageType': 'ListDevices'})
+        for response in self._receive(self._tag - 1)['DeviceList']:
+            if response['MessageType'] == 'Attached':
+                super()._add_device(MuxDevice(response['DeviceID'], response['Properties']['SerialNumber'],
+                                              response['Properties']['ConnectionType']))
+            elif response['MessageType'] == 'Detached':
+                super()._remove_device(response['DeviceID'])
+            else:
+                raise MuxError(f'Invalid packet type received: {response}')
+
+    def get_buid(self) -> str:
+        """ get SystemBUID """
+        self._send({'MessageType': 'ReadBUID'})
+        return self._receive(self._tag - 1)['BUID']
+
+    def save_pair_record(self, serial: str, device_id: int, record_data: bytes):
+        # serials are saved inside usbmuxd without '-'
+        self._send_receive({'MessageType': 'SavePairRecord',
+                            'PairRecordID': serial,
+                            'PairRecordData': record_data,
+                            'DeviceID': device_id})
+
+    def _connect(self, device_id: int, port: int):
+        self._send_receive({'MessageType': 'Connect', 'DeviceID': device_id, 'PortNumber': port})
+
+    def _send(self, data: Mapping):
+        request = {'ClientVersionString': 'qt4i-usbmuxd', 'ProgName': 'pymobiledevice3', 'kLibUSBMuxVersion': 3}
+        request.update(data)
+        super()._send({'header': {'version': self._version,
+                                  'message': usbmuxd_msgtype.PLIST,
+                                  'tag': self._tag},
+                       'data': plistlib.dumps(request),
+                       })
+
+    def _receive(self, expected_tag: int = None) -> Mapping:
+        response = super()._receive(expected_tag=expected_tag)
+        if response.header.message != usbmuxd_msgtype.PLIST:
+            raise MuxError(f'Received non-plist type {response}')
+        return plistlib.loads(response.data)
+
+    def _send_receive(self, data: Mapping):
+        self._send(data)
+        response = self._receive(self._tag - 1)
+        if response['MessageType'] != 'Result':
+            raise MuxError(f'got an invalid message: {response}')
+        if response['Number'] != 0:
+            raise self._raise_mux_exception(response['Number'], f'got an error message: {response}')
+
+
+def create_mux(usbmux_address: Optional[str] = None) -> MuxConnection:
+    return MuxConnection.create(usbmux_address=usbmux_address)
+
+
+def list_devices(usbmux_address: Optional[str] = None) -> List[MuxDevice]:
+    mux = create_mux(usbmux_address=usbmux_address)
+    mux.get_device_list(0.1)
+    devices = mux.devices
+    mux.close()
+    return devices
+
+
+def select_device(udid: str = None, connection_type: str = None, usbmux_address: Optional[str] = None) \
+        -> Optional[MuxDevice]:
+    """
+    select a UsbMux device according to given arguments.
+    if more than one device could be selected, always prefer the usb one.
+    """
+    tmp = None
+    for device in list_devices(usbmux_address=usbmux_address):
+        if connection_type is not None and device.connection_type != connection_type:
+            # if a specific connection_type was desired and not of this one then skip
+            continue
+
+        if udid is not None and not device.matches_udid(udid):
+            # if a specific udid was desired and not of this one then skip
+            continue
+
+        # save best result as a temporary
+        tmp = device
+
+        if device.is_usb:
+            # always prefer usb connection
+            return device
+
+    return tmp
+
+
+def select_devices_by_connection_type(connection_type: str, usbmux_address: Optional[str] = None) -> List[MuxDevice]:
+    """
+    select all UsbMux devices by connection type
+    """
+    tmp = []
+    for device in list_devices(usbmux_address=usbmux_address):
+        if device.connection_type == connection_type:
+            tmp.append(device)
+
+    return tmp
+
+
+
+class USBMuxHTTPConnection(HTTPConnection):
+    def __init__(self, device: MuxDevice, port=8100):
+        super().__init__("localhost", port)
+        self.__device = device
+        self.__port = port
+
+    def connect(self):
+        self.sock = self.__device.connect(self.__port)
+
+    def __enter__(self) -> HTTPConnection:
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.close()

DroidBot/platforms/ios/wda/utils.py

@@ -0,0 +1,69 @@
+# coding: utf-8
+
+import functools
+import threading
+import typing
+import inspect
+
+
+def inject_call(fn, *args, **kwargs):
+    """
+    Call function without known all the arguments
+
+    Args:
+        fn: function
+        args: arguments
+        kwargs: key-values
+    
+    Returns:
+        as the fn returns
+    """
+    assert callable(fn), "first argument must be callable"
+
+    st = inspect.signature(fn)
+    fn_kwargs = {
+        key: kwargs[key]
+        for key in st.parameters.keys() if key in kwargs
+    }
+    ba = st.bind(*args, **fn_kwargs)
+    ba.apply_defaults()
+    return fn(*ba.args, **ba.kwargs)
+
+
+def limit_call_depth(n: int):
+    """
+    n = 0 means not allowed recursive call
+    """
+    def wrapper(fn: typing.Callable):
+        local = threading.local()
+        
+        @functools.wraps(fn)
+        def _inner(*args, **kwargs):
+            if not hasattr(local, 'depth'):
+                local.depth = 0
+            if local.depth > n:
+                raise RuntimeError("call depth exceed %d" % n)
+
+            local.depth += 1
+            try:
+                return fn(*args, **kwargs)
+            finally:
+                local.depth -= 1
+        
+        return _inner
+    
+    return wrapper
+
+
+class AttrDict(dict):
+    def __getattr__(self, key):
+        if isinstance(key, str) and key in self:
+            return self[key]
+        raise AttributeError("Attribute key not found", key)
+
+
+def convert(dictionary):
+    """
+    Convert dict to namedtuple
+    """
+    return AttrDict(dictionary)

DroidBot/platforms/ios/wda/xcui_element_types.py

@@ -0,0 +1,89 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+from __future__ import unicode_literals
+
+
+ELEMENTS = [
+    'Any',
+    'Other',
+    'Application',
+    'Group',
+    'Window',
+    'Sheet',
+    'Drawer',
+    'Alert',
+    'Dialog',
+    'Button',
+    'RadioButton',
+    'RadioGroup',
+    'CheckBox',
+    'DisclosureTriangle',
+    'PopUpButton',
+    'ComboBox',
+    'MenuButton',
+    'ToolbarButton',
+    'Popover',
+    'Keyboard',
+    'Key',
+    'NavigationBar',
+    'TabBar',
+    'TabGroup',
+    'Toolbar',
+    'StatusBar',
+    'Table',
+    'TableRow',
+    'TableColumn',
+    'Outline',
+    'OutlineRow',
+    'Browser',
+    'CollectionView',
+    'Slider',
+    'PageIndicator',
+    'ProgressIndicator',
+    'ActivityIndicator',
+    'SegmentedControl',
+    'Picker',
+    'PickerWheel',
+    'Switch',
+    'Toggle',
+    'Link',
+    'Image',
+    'Icon',
+    'SearchField',
+    'ScrollView',
+    'ScrollBar',
+    'StaticText',
+    'TextField',
+    'SecureTextField',
+    'DatePicker',
+    'TextView',
+    'Menu',
+    'MenuItem',
+    'MenuBar',
+    'MenuBarItem',
+    'Map',
+    'WebView',
+    'IncrementArrow',
+    'DecrementArrow',
+    'Timeline',
+    'RatingIndicator',
+    'ValueIndicator',
+    'SplitGroup',
+    'Splitter',
+    'RelevanceIndicator',
+    'ColorWell',
+    'HelpTag',
+    'Matte',
+    'DockItem',
+    'Ruler',
+    'RulerMarker',
+    'Grid',
+    'LevelIndicator',
+    'Cell',
+    'LayoutArea',
+    'LayoutItem',
+    'Handle',
+    'Stepper',
+    'Tab'
+]

DroidBot/platforms/web/__init__.py

@@ -0,0 +1,47 @@
+"""
+Web Platform Module
+Platform-specific implementations for Web testing.
+"""
+
+from .web_device import WebDevice
+from .web_device_state import WebDeviceState
+from .web_app import WebApp
+from .web_input_event import (
+    WebTouchEvent,
+    WebLongTouchEvent,
+    WebSwipeEvent,
+    WebScrollEvent,
+    WebSetTextEvent,
+    WebKeyEvent,
+    WebKillAppEvent,
+)
+
+from ...core.platform_factory import PlatformFactory, Platform
+
+PlatformFactory.register_platform(
+    Platform.WEB,
+    WebDevice,
+    WebDeviceState,
+    {
+        'touch': WebTouchEvent,
+        'long_touch': WebLongTouchEvent,
+        'swipe': WebSwipeEvent,
+        'scroll': WebScrollEvent,
+        'set_text': WebSetTextEvent,
+        'key': WebKeyEvent,
+        'kill_app': WebKillAppEvent,
+    }
+)
+
+__all__ = [
+    'WebDevice',
+    'WebDeviceState',
+    'WebApp',
+    'WebTouchEvent',
+    'WebLongTouchEvent',
+    'WebSwipeEvent',
+    'WebScrollEvent',
+    'WebSetTextEvent',
+    'WebKeyEvent',
+    'WebKillAppEvent',
+]

DroidBot/platforms/web/web_app.py

@@ -0,0 +1,73 @@
+"""
+Web App Module
+Web-specific application model.
+"""
+import logging
+import os
+from ...core.abstract_app import AbstractApp
+
+
+class WebApp(AbstractApp):
+    """
+    Web 应用类 - 用于管理 Web 应用的信息
+    
+    继承自 AbstractApp，实现 Web 特定的应用管理功能。
+    """
+
+    def __init__(self, url: str, output_dir: str = None):
+        """
+        创建 WebApp 实例
+        
+        :param url: Web 应用的 URL
+        :param output_dir: 输出目录路径
+        """
+        super().__init__(output_dir=output_dir)
+        
+        self.app_url = url  # Web应用的URL
+        self._main_page = url  # 主页面URL
+        self._pages = []  # 已访问页面列表
+        
+    @property
+    def identifier(self) -> str:
+        """获取应用唯一标识符（URL）"""
+        return self.app_url
+
+    @property
+    def main_activity(self) -> str:
+        """获取主入口点（主页面URL）"""
+        return self._main_page
+    
+    @main_activity.setter
+    def main_activity(self, value):
+        """设置主入口点"""
+        self._main_page = value
+    
+    @property
+    def activities(self) -> list:
+        """获取入口点列表（已访问页面）"""
+        return self._pages
+
+    @activities.setter
+    def activities(self, value):
+        """设置入口点列表"""
+        self._pages = value
+    
+    def add_page(self, url: str):
+        """
+        添加已访问页面
+        
+        :param url: 已访问的页面URL
+        """
+        if url not in self._pages:
+            self._pages.append(url)
+    
+    def get_current_page(self, device) -> str:
+        """
+        获取当前页面URL
+        
+        :param device: WebDevice实例
+        :return: 当前页面URL
+        """
+        if device and device._current_url:
+            return device._current_url
+        return self.app_url

DroidBot/platforms/web/web_device.py

@@ -0,0 +1,814 @@
+"""
+Web Device Implementation
+Concrete implementation of AbstractDevice for Web applications.
+"""
+import logging
+import os
+import time
+from typing import Optional, Dict, Any, List
+
+from selenium import webdriver
+from selenium.webdriver.chrome.options import Options
+from webdriver_manager.chrome import ChromeDriverManager
+from selenium.webdriver.chrome.service import Service as ChromeService
+
+from ...core.abstract_device import AbstractDevice
+
+
+class WebDevice(AbstractDevice):
+    """
+    Web 设备的具体实现
+    
+    继承自 AbstractDevice，使用 Selenium WebDriver 实现所有 Web 特定的设备操作。
+    """
+
+    def __init__(self, app_path: Optional[str] = None, output_dir: Optional[str] = None,
+                 browser: str = "chrome", headless: bool = False, **kwargs):
+        """
+        初始化 Web 设备连接
+        
+        :param app_path: Web 应用的 URL 或本地 HTML 文件路径
+        :param output_dir: 输出目录
+        :param browser: 浏览器类型，支持 "chrome" 等
+        :param headless: 是否使用无头模式
+        :param kwargs: 其他参数
+        """
+        super().__init__(output_dir=output_dir)
+        
+        self.browser = browser.lower()
+        self.headless = headless
+        self.engine = kwargs.get('engine', 'playwright')
+        self.driver = None  # Selenium WebDriver实例 或 Playwright Page实例
+        self.playwright = None
+        self.playwright_browser = None
+        self.playwright_context = None
+        self._current_url = None  # 当前页面URL缓存
+        self._window_handles = set()  # 记录已知的窗口句柄
+        self._last_sync_time = 0  # P2: 窗口句柄同步时间戳缓存
+        self._last_state = None
+        self._view_limit = int(kwargs.get('view_limit', 250))
+        
+        # 设备信息
+        self.display_info = None
+        
+        # 初始化WebApp实例（与AndroidDevice保持一致的命名方式）
+        self.app = None
+        self._app = None
+        if app_path:
+            from .web_app import WebApp
+            self.app = WebApp(app_path, output_dir=output_dir)
+            self._app = self.app  # 与AndroidDevice保持一致，使用_app属性
+            self.logger.info(f"初始化WebApp: {app_path}")
+    
+    @property
+    def app_url(self) -> Optional[str]:
+        """获取 Web 应用的 URL"""
+        if self._app:
+            return self._app.app_url
+        return None
+
+
+    # ==================== 平台信息 ====================
+    
+    def get_platform_name(self) -> str:
+        return "web"
+    
+    
+    # ==================== 连接管理 ====================
+    
+    def set_up(self) -> None:
+        """设置WebDriver或Playwright"""
+        self.logger.info(f"Setting up Web engine ({self.engine})...")
+        
+        if self.engine == "playwright":
+            from playwright.sync_api import sync_playwright
+            self.playwright = sync_playwright().start()
+            
+            if self.browser == "lightpanda":
+                self.logger.info("Connecting to local Lightpanda via CDP (ws://127.0.0.1:9222)...")
+                try:
+                    self.playwright_browser = self.playwright.chromium.connect_over_cdp("ws://127.0.0.1:9222")
+                    self.playwright_context = self.playwright_browser.contexts[0] if self.playwright_browser.contexts else self.playwright_browser.new_context()
+                    self.driver = self.playwright_context.pages[0] if self.playwright_context.pages else self.playwright_context.new_page()
+                except Exception as e:
+                    self.logger.error(f"Failed to connect to Lightpanda: {e}")
+                    raise
+            else:
+                self.logger.info("Launching Playwright Chrome...")
+                self.playwright_browser = self.playwright.chromium.launch(headless=self.headless)
+                self.playwright_context = self.playwright_browser.new_context(
+                    viewport={'width': 1920, 'height': 1080},
+                    ignore_https_errors=True
+                )
+                self.driver = self.playwright_context.new_page()
+        else:
+            # 配置浏览器选项 (Selenium logic)
+            if self.browser == "chrome":
+                options = Options()
+                if self.headless:
+                    options.add_argument("--headless")
+                # 窗口配置
+                options.add_argument("--window-size=1920,1080")
+                # 性能和兼容性配置
+                options.add_argument("--no-sandbox")
+                options.add_argument("--disable-dev-shm-usage")
+                options.add_argument("--disable-gpu")  # 禁用GPU加速
+                # 用户体验配置
+                options.add_argument("--disable-infobars")  # 禁用信息栏
+                options.add_argument("--disable-extensions")  # 禁用扩展
+                # SSL/证书配置
+                options.add_argument("--ignore-certificate-errors")  # 忽略证书错误
+                options.add_argument("--ignore-ssl-errors")  # 忽略SSL错误
+                # 自动化检测规避
+                options.add_experimental_option("excludeSwitches", ["enable-automation"])
+                options.add_experimental_option("useAutomationExtension", False)
+                
+                # 初始化WebDriver - 优先使用本地chromedriver，避免每次检查更新
+                try:
+                    # 尝试直接使用本地chromedriver（系统PATH中或默认位置）
+                    self.driver = webdriver.Chrome(options=options)
+                    self.logger.info("使用本地chromedriver")
+                except Exception as e:
+                    # 如果本地没有找到，再使用webdriver_manager下载
+                    self.logger.warning(f"本地chromedriver未找到，尝试下载: {e}")
+                    self.driver = webdriver.Chrome(
+                        service=ChromeService(ChromeDriverManager(cache_valid_range=365).install()),
+                        options=options
+                    )
+            else:
+                raise NotImplementedError(f"Browser {self.browser} is not supported yet")
+
+    def _make_state_tag(self) -> str:
+        """生成高精度状态标签，避免同秒内截图文件名冲突。"""
+        from datetime import datetime
+        return datetime.now().strftime("%Y-%m-%d_%H%M%S_%f")
+
+    def _wait_for_page_ready(self, timeout_ms: int = 5000) -> None:
+        """在截图和控件提取前等待页面达到可操作状态。"""
+        if not self.driver:
+            return
+
+        try:
+            if self.engine == "playwright":
+                self.driver.wait_for_load_state("domcontentloaded", timeout=timeout_ms)
+                try:
+                    self.driver.wait_for_load_state("load", timeout=timeout_ms)
+                except Exception:
+                    # 某些站点会持续请求资源，load 超时不应阻塞探索。
+                    pass
+                self.driver.wait_for_timeout(200)
+            else:
+                deadline = time.time() + timeout_ms / 1000.0
+                while time.time() < deadline:
+                    try:
+                        ready_state = self.driver.execute_script("return document.readyState")
+                        if ready_state in ("interactive", "complete"):
+                            break
+                    except Exception:
+                        break
+                    time.sleep(0.1)
+        except Exception as e:
+            self.logger.debug(f"wait_for_page_ready skipped: {e}")
+
+    def get_views(self) -> List[Dict[str, Any]]:
+        """从当前 DOM 提取可交互控件，统一转换为 ViewDict。"""
+        if not self.driver:
+            return []
+
+        self._wait_for_page_ready()
+
+        script = f"""
+        () => {{
+            const MAX_VIEWS = {self._view_limit};
+            const viewportW = window.innerWidth || document.documentElement.clientWidth || 1920;
+            const viewportH = window.innerHeight || document.documentElement.clientHeight || 1080;
+            const allElements = Array.from(document.querySelectorAll('body *'));
+
+            const normalizeText = (value) => (value || '')
+                .replace(/\\s+/g, ' ')
+                .trim()
+                .slice(0, 200);
+
+            const isVisible = (el) => {{
+                if (!el || !el.isConnected) return false;
+                const style = window.getComputedStyle(el);
+                if (!style) return false;
+                if (style.display === 'none' || style.visibility === 'hidden' || style.pointerEvents === 'none') return false;
+                if (Number(style.opacity || '1') === 0) return false;
+                const rect = el.getBoundingClientRect();
+                if (!rect || rect.width < 4 || rect.height < 4) return false;
+                if (rect.bottom <= 0 || rect.right <= 0 || rect.top >= viewportH || rect.left >= viewportW) return false;
+                return true;
+            }};
+
+            const isEnabled = (el) => !el.disabled && el.getAttribute('aria-disabled') !== 'true';
+
+            const isEditable = (el) => {{
+                if (!el) return false;
+                if (el.isContentEditable) return true;
+                const tag = el.tagName;
+                if (tag === 'TEXTAREA' || tag === 'SELECT') return true;
+                if (tag === 'INPUT') {{
+                    const type = (el.getAttribute('type') || 'text').toLowerCase();
+                    return !['button', 'checkbox', 'color', 'file', 'hidden', 'image', 'radio', 'range', 'reset', 'submit'].includes(type);
+                }}
+                return false;
+            }};
+
+            const isScrollable = (el) => {{
+                if (!el) return false;
+                const style = window.getComputedStyle(el);
+                if (!style) return false;
+                const overflowY = style.overflowY || '';
+                const overflowX = style.overflowX || '';
+                const scrollY = el.scrollHeight - el.clientHeight > 24 && ['auto', 'scroll', 'overlay'].includes(overflowY);
+                const scrollX = el.scrollWidth - el.clientWidth > 24 && ['auto', 'scroll', 'overlay'].includes(overflowX);
+                return scrollY || scrollX;
+            }};
+
+            const isClickable = (el, editable) => {{
+                if (!el) return false;
+                if (editable) return true;
+                const tag = el.tagName;
+                const role = (el.getAttribute('role') || '').toLowerCase();
+                const type = (el.getAttribute('type') || '').toLowerCase();
+                const style = window.getComputedStyle(el);
+                return Boolean(
+                    typeof el.onclick === 'function' ||
+                    el.hasAttribute('onclick') ||
+                    el.hasAttribute('ng-click') ||
+                    el.hasAttribute('v-on:click') ||
+                    el.hasAttribute('@click') ||
+                    (tag === 'A' && el.getAttribute('href')) ||
+                    ['BUTTON', 'SUMMARY', 'LABEL'].includes(tag) ||
+                    ['button', 'link', 'tab', 'menuitem', 'checkbox', 'radio', 'switch', 'option'].includes(role) ||
+                    (tag === 'INPUT' && ['button', 'checkbox', 'radio', 'submit'].includes(type)) ||
+                    (style && style.cursor === 'pointer') ||
+                    el.tabIndex >= 0
+                );
+            }};
+
+            const getText = (el) => {{
+                if (!el) return '';
+                const tag = el.tagName;
+                if (tag === 'INPUT' || tag === 'TEXTAREA' || tag === 'SELECT') {{
+                    return normalizeText(el.value || el.getAttribute('placeholder') || el.getAttribute('aria-label'));
+                }}
+                return normalizeText(el.innerText || el.textContent || el.getAttribute('aria-label') || el.getAttribute('title'));
+            }};
+
+            const getDescription = (el) => normalizeText(
+                el.getAttribute('aria-label') ||
+                el.getAttribute('title') ||
+                el.getAttribute('placeholder') ||
+                ''
+            );
+
+            const decorativeTags = new Set(['path', 'svg', 'g', 'use', 'circle', 'rect', 'polygon', 'line', 'polyline', 'ellipse']);
+            const semanticTags = new Set(['a', 'button', 'input', 'textarea', 'select', 'summary', 'label', 'option']);
+
+            const toBounds = (rect) => {{
+                const left = Math.max(0, Math.floor(rect.left));
+                const top = Math.max(0, Math.floor(rect.top));
+                const right = Math.min(viewportW, Math.ceil(rect.right));
+                const bottom = Math.min(viewportH, Math.ceil(rect.bottom));
+                return [[left, top], [right, bottom]];
+            }};
+
+            const candidates = [];
+            for (const el of allElements) {{
+                if (!isVisible(el)) continue;
+
+                const editable = isEditable(el);
+                const scrollable = isScrollable(el);
+                const clickable = isClickable(el, editable);
+                if (!clickable && !editable && !scrollable) continue;
+
+                const rect = el.getBoundingClientRect();
+                const area = Math.max(1, rect.width * rect.height);
+                if (area >= viewportW * viewportH * 0.98 && !editable && !scrollable) continue;
+                const className = (el.tagName || 'div').toLowerCase();
+                const text = getText(el);
+                const contentDescription = getDescription(el);
+                if (decorativeTags.has(className) && !text && !contentDescription && area < 2000) continue;
+                const semantic = semanticTags.has(className) ? 1 : 0;
+                const hasText = text || contentDescription ? 1 : 0;
+
+                candidates.push({{
+                    element: el,
+                    bounds: toBounds(rect),
+                    text,
+                    content_description: contentDescription,
+                    visible: true,
+                    enabled: isEnabled(el),
+                    clickable,
+                    editable,
+                    scrollable,
+                    checkable: ['checkbox', 'radio', 'switch'].includes((el.getAttribute('role') || '').toLowerCase()) ||
+                               (el.tagName === 'INPUT' && ['checkbox', 'radio'].includes((el.getAttribute('type') || '').toLowerCase())),
+                    checked: Boolean(el.checked),
+                    selected: Boolean(el.selected || el.getAttribute('aria-selected') === 'true'),
+                    long_clickable: false,
+                    children: [],
+                    parent: -1,
+                    resource_id: el.id || '',
+                    class_name: className,
+                    source: 'dom',
+                    area,
+                    priority: editable ? 5 : (semantic ? 4 : (hasText ? 3 : (clickable ? 2 : 1)))
+                }});
+            }}
+
+            candidates.sort((a, b) => {{
+                if (b.priority !== a.priority) return b.priority - a.priority;
+                return a.area - b.area;
+            }});
+
+            const selected = [];
+            const seen = new Set();
+            for (const candidate of candidates) {{
+                const key = `${{candidate.class_name}}|${{candidate.bounds[0][0]}},${{candidate.bounds[0][1]}},${{candidate.bounds[1][0]}},${{candidate.bounds[1][1]}}|${{candidate.text}}`;
+                if (seen.has(key)) continue;
+                seen.add(key);
+                selected.push(candidate);
+                if (selected.length >= MAX_VIEWS) break;
+            }}
+
+            const indexByElement = new Map();
+            selected.forEach((candidate, index) => {{
+                indexByElement.set(candidate.element, index);
+            }});
+
+            selected.forEach((candidate, index) => {{
+                let parent = candidate.element.parentElement;
+                while (parent) {{
+                    if (indexByElement.has(parent)) {{
+                        const parentIndex = indexByElement.get(parent);
+                        candidate.parent = parentIndex;
+                        selected[parentIndex].children.push(index);
+                        break;
+                    }}
+                    parent = parent.parentElement;
+                }}
+            }});
+
+            return selected.map((candidate, index) => {{
+                delete candidate.element;
+                delete candidate.area;
+                delete candidate.priority;
+                candidate.temp_id = index;
+                return candidate;
+            }});
+        }}
+        """
+
+        try:
+            if self.engine == "playwright":
+                views = self.driver.evaluate(script)
+            else:
+                views = self.driver.execute_script(f"return ({script})();")
+        except Exception as e:
+            self.logger.error(f"Failed to extract Web views: {e}")
+            return []
+
+        if not isinstance(views, list):
+            return []
+        self.logger.info(f"Extracted {len(views)} Web views")
+        return views
+        
+    def connect(self) -> bool:
+        """连接到Web应用"""
+        try:
+            if self._app:
+                self.logger.info(f"Connecting to Web application: {self._app.app_url}")
+                if self.driver is None:
+                    self.set_up()
+                
+                if self.engine == "playwright":
+                    self.driver.goto(self._app.app_url)
+                    self._wait_for_page_ready()
+                    self._current_url = self.driver.url
+                else:
+                    self.driver.get(self._app.app_url)
+                    self._wait_for_page_ready()
+                    self._current_url = self.driver.current_url
+                    self._window_handles = set(self.driver.window_handles)
+                self.connected = True
+                self.logger.info(f"Connected to Web application: {self._current_url}")
+                return True
+            else:
+                self.logger.error("No Web application URL specified")
+                return False
+        except Exception as e:
+            self.logger.error(f"Failed to connect to Web application: {e}")
+            return False
+    
+    def disconnect(self) -> None:
+        """断开连接"""
+        self.logger.info("Disconnecting Web engine...")
+        self.connected = False
+        if self.engine == "playwright":
+            if self.driver:
+                try:
+                    if not self.driver.is_closed():
+                        self.driver.close()
+                except Exception as e:
+                    self.logger.debug(f"Close Playwright page skipped: {e}")
+            if self.playwright_context:
+                try:
+                    self.playwright_context.close()
+                except Exception as e:
+                    self.logger.debug(f"Close Playwright context skipped: {e}")
+            if self.playwright_browser:
+                try:
+                    self.playwright_browser.close()
+                except Exception as e:
+                    self.logger.debug(f"Close Playwright browser skipped: {e}")
+            if self.playwright:
+                try:
+                    self.playwright.stop()
+                except Exception as e:
+                    self.logger.debug(f"Stop Playwright skipped: {e}")
+            self.driver = None
+            self.playwright_context = None
+            self.playwright_browser = None
+            self.playwright = None
+        else:
+            if self.driver:
+                try:
+                    self.driver.quit()
+                except Exception as e:
+                    self.logger.debug(f"Quit Selenium driver skipped: {e}")
+                self.driver = None
+    
+    def tear_down(self) -> None:
+        """清理WebDriver资源"""
+        self.disconnect()
+    
+    def check_connectivity(self) -> bool:
+        """检查WebDriver连接状态"""
+        try:
+            if self.driver:
+                if self.engine == "playwright":
+                    if self.driver.is_closed():
+                        raise Exception("Playwright page is closed")
+                    self.driver.title()
+                else:
+                    self.driver.title  # 尝试获取页面标题，如果失败则表示连接已断开
+                self.connected = True
+                return True
+            else:
+                self.connected = False
+                return False
+        except Exception as e:
+            self.logger.error(f"WebDriver connectivity check failed: {e}")
+            self.connected = False
+            return False
+    
+    # ==================== 状态获取 ====================
+    
+    def get_current_state(self) -> 'WebDeviceState':
+        """获取当前Web页面状态"""
+        from .web_device_state import WebDeviceState
+        
+        self.logger.debug("Getting current Web page state...")
+        try:
+            tag = self._make_state_tag()
+            views = self.get_views()
+            screenshot_path = self.take_screenshot(tag=tag)
+            current_state = WebDeviceState(
+                self,
+                views=views,
+                tag=tag,
+                screenshot_path=screenshot_path
+            )
+            return current_state
+        except Exception as e:
+            self.logger.error(f"Failed to get current Web page state: {e}")
+            return None
+    
+    def get_display_info(self, refresh: bool = False) -> Dict[str, Any]:
+        """获取显示信息"""
+        if self.display_info is None or refresh:
+            if self.driver:
+                if self.engine == "playwright":
+                    viewport = self.driver.viewport_size
+                    if viewport:
+                        width = viewport['width']
+                        height = viewport['height']
+                    else:
+                        width, height = 1920, 1080
+                else:
+                    window_size = self.driver.get_window_size()
+                    width = window_size["width"]
+                    height = window_size["height"]
+                self.display_info = {
+                    "width": width,
+                    "height": height,
+                    "density": 1.0  # Web没有物理密度概念，使用1.0
+                }
+            else:
+                self.display_info = {
+                    "width": 1920,
+                    "height": 1080,
+                    "density": 1.0
+                }
+        return self.display_info
+    
+    # ==================== 屏幕操作 ====================
+    
+    def take_screenshot(self, path: str = None, tag: str = None) -> str:
+        """截取Web页面截图
+        
+        P4: 支持 tag 参数，与state关联便于去重
+        """
+        if not self.driver:
+            self.logger.error("WebDriver not initialized, cannot take screenshot")
+            return None
+        
+        if self.output_dir is None and path is None:
+            return None
+        
+        if tag is None:
+            from datetime import datetime
+            tag = datetime.now().strftime("%Y-%m-%d_%H%M%S")
+        local_image_dir = os.path.join(self.output_dir, "temp") if self.output_dir else "/tmp"
+        if not os.path.exists(local_image_dir):
+            os.makedirs(local_image_dir)
+        
+        if path is None:
+            local_image_path = os.path.join(local_image_dir, f"web_screen_{tag}.png")
+        else:
+            local_image_path = path
+
+        parent_dir = os.path.dirname(local_image_path)
+        if parent_dir and not os.path.exists(parent_dir):
+            os.makedirs(parent_dir)
+        
+        last_error = None
+        for attempt in range(2):
+            try:
+                self._wait_for_page_ready(timeout_ms=2000)
+                if self.engine == "playwright":
+                    self.driver.screenshot(path=local_image_path)
+                else:
+                    success = self.driver.save_screenshot(local_image_path)
+                    if not success:
+                        raise RuntimeError("save_screenshot returned False")
+                if os.path.exists(local_image_path) and os.path.getsize(local_image_path) > 0:
+                    self.logger.info(f"Screenshot saved to: {local_image_path}")
+                    return local_image_path
+                raise RuntimeError("screenshot file not created")
+            except Exception as e:
+                last_error = e
+                self.logger.warning(f"Failed to take screenshot (attempt {attempt + 1}/2): {e}")
+                time.sleep(0.2)
+
+        self.logger.error(f"Failed to take screenshot: {last_error}")
+        return None
+    
+    def unlock(self) -> None:
+        """Web平台无需解锁屏幕，实现空方法"""
+        pass
+    
+    def get_current_url(self) -> str:
+        """
+        获取当前页面URL
+        
+        :return: 当前页面URL
+        """
+        if self.driver:
+            try:
+                if self.engine == "playwright":
+                    self._current_url = self.driver.url
+                else:
+                    self._current_url = self.driver.current_url
+                return self._current_url
+            except Exception as e:
+                self.logger.error(f"Failed to get current URL: {e}")
+        return self._current_url
+    
+    def get_page_title(self) -> str:
+        """
+        获取当前页面标题
+        
+        :return: 当前页面标题
+        """
+        if self.driver:
+            try:
+                if self.engine == "playwright":
+                    return self.driver.title()
+                else:
+                    return self.driver.title
+            except Exception as e:
+                self.logger.error(f"Failed to get page title: {e}")
+        return ""
+    
+    # ==================== 事件发送 ====================
+    
+    def send_event(self, event) -> bool:
+        """发送Web事件"""
+        try:
+            self.logger.debug(f"Sending Web event: {event}")
+            success = event.send(self)
+            if success and self.engine != "playwright":
+                self._sync_window_handles()
+            return bool(success)
+        except Exception as e:
+            self.logger.error(f"Failed to send Web event: {e}")
+            return False
+    
+    # ==================== 应用管理 ====================
+    
+    @property
+    def app_identifier(self) -> str:
+        """获取Web应用的唯一标识符（URL）"""
+        if self._app:
+            return self._app.identifier
+        return ""
+    
+    def _get_domain(self, url: str) -> str:
+        """从 URL 中提取二级域名"""
+        if not url:
+            return ""
+        from urllib.parse import urlparse
+        try:
+            netloc = urlparse(url).netloc
+            if not netloc:
+                return ""
+            parts = netloc.split('.')
+            if len(parts) >= 2:
+                # 取最后两部分，例如 www.baidu.com -> baidu.com
+                return ".".join(parts[-2:])
+            return netloc
+        except Exception:
+            return ""
+
+    def is_foreground(self) -> bool:
+        """检查Web应用是否在前台（基于域名匹配）"""
+        if not self.driver:
+            return False
+        
+        try:
+            current_url = self.get_current_url()
+            # 检查当前URL是否与目标应用二级域名匹配
+            if self._app:
+                target_domain = self._get_domain(self._app.identifier)
+                current_domain = self._get_domain(current_url)
+                return target_domain == current_domain
+            return False
+        except Exception as e:
+            self.logger.error(f"Failed to check if Web app is in foreground: {e}")
+            return False
+    
+    def get_redirect_target_info(self):
+        """
+        Web平台跳转检测：基于域名变化判断是否离开目标站点
+        
+        - 当前域名匹配目标 → 返回 None（在前台）
+        - 当前域名不匹配 → 返回 {"target": domain, "type": "other"}
+        - 无法获取URL → 返回 {"target": None, "type": "unknown"}
+        
+        :return: 跳转信息字典或 None
+        """
+        if not self.driver:
+            return {"target": None, "type": "unknown"}
+        try:
+            current_url = self.get_current_url()
+            if self._app:
+                target_domain = self._get_domain(self._app.identifier)
+                current_domain = self._get_domain(current_url)
+                if target_domain == current_domain:
+                    return None  # 仍在目标站点
+                return {"target": current_domain, "type": "other"}
+            return None
+        except Exception:
+            return {"target": None, "type": "unknown"}
+    
+    def check_network(self, host: str = "8.8.8.8") -> bool:
+        """
+        通过浏览器内 JS 检测网络连通性
+        
+        优先使用 XMLHttpRequest 同步请求 Google generate_204，
+        失败时 fallback 到 navigator.onLine。
+        
+        :param host: 未使用（保持接口兼容）
+        :return: 网络是否可用
+        """
+        if not self.driver:
+            return False
+        try:
+            script = """
+                try {
+                    var xhr = new XMLHttpRequest();
+                    xhr.open('HEAD', 'https://www.google.com/generate_204', false);
+                    xhr.timeout = 5000;
+                    xhr.send();
+                    return xhr.status === 204 || xhr.status === 200;
+                } catch(e) {
+                    return navigator.onLine;
+                }
+            """
+            if self.engine == "playwright":
+                result = self.driver.evaluate(f"() => {{ {script} }}")
+            else:
+                result = self.driver.execute_script(script)
+            return bool(result)
+        except Exception as e:
+            self.logger.warning(f"check_network 异常: {e}")
+            return False
+    
+    def pull_back_to_app(self) -> bool:
+        """将Web应用拉回前台"""
+        app_url = self.app_url
+        if not self.driver or not app_url:
+            return False
+        
+        try:
+            if self.engine == "playwright":
+                self.driver.goto(app_url)
+                self._wait_for_page_ready()
+            else:
+                self.driver.get(app_url)
+                self._wait_for_page_ready()
+            self.logger.info(f"Pulled back to Web app: {app_url}")
+            return True
+        except Exception as e:
+            self.logger.error(f"Failed to pull back Web app to foreground: {e}")
+            return False
+    
+    def start_app(self) -> bool:
+        """启动Web应用"""
+        app_url = self.app_url
+        if not app_url:
+            self.logger.warning("No Web app URL specified, cannot start")
+            return False
+        
+        try:
+            if not self.driver:
+                self.set_up()
+            
+            if self.engine == "playwright":
+                self.driver.goto(app_url)
+                self._wait_for_page_ready()
+                self._current_url = self.driver.url
+            else:
+                self.driver.get(app_url)
+                self._wait_for_page_ready()
+                self._current_url = self.driver.current_url
+            self.logger.info(f"Started Web app: {self._current_url}")
+            return True
+        except Exception as e:
+            self.logger.error(f"Failed to start Web app: {e}")
+            return False
+    
+    # ==================== Web 特定方法 ====================
+    
+    def _sync_window_handles(self) -> None:
+        """
+        P2: 带时间戳缓存的窗口句柄同步
+        
+        500ms内不重复调用window_handles，减少IPC开销。
+        """
+        if not self.driver or getattr(self, "engine", "selenium") == "playwright":
+            return
+        
+        import time
+        now = time.time()
+        if now - self._last_sync_time < 0.5:
+            return  # 距上次同步不到500ms，跳过
+        self._last_sync_time = now
+            
+        try:
+            current_handles = self.driver.window_handles
+            # 检查是否有新窗口
+            new_handles = [h for h in current_handles if h not in self._window_handles]
+            
+            if new_handles:
+                # 切换到最新的窗口
+                target_window = new_handles[-1]
+                self.driver.switch_to.window(target_window)
+                self._window_handles.update(current_handles)
+                self.logger.info(f"Detected new window, switched to: {target_window}")
+            else:
+                # 检查当前窗口是否仍然有效，如果无效则切换回一个有效的窗口
+                try:
+                    _ = self.driver.current_window_handle
+                except Exception:
+                    self.logger.warning("Current window closed, switching to last valid handle")
+                    if current_handles:
+                        self.driver.switch_to.window(current_handles[-1])
+                        self._window_handles = set(current_handles)
+        except Exception as e:
+            self.logger.error(f"Failed to sync window handles: {e}")
+    
+    def refresh_page(self) -> None:
+        """刷新当前页面"""
+        if self.driver:
+            try:
+                if getattr(self, "engine", "selenium") == "playwright":
+                    self.driver.reload()
+                else:
+                    self.driver.refresh()
+            except Exception as e:
+                self.logger.error(f"Failed to refresh page: {e}")

DroidBot/platforms/web/web_device_state.py

@@ -0,0 +1,199 @@
+"""
+Web Device State Implementation
+Concrete implementation of AbstractDeviceState for Web applications.
+"""
+import hashlib
+from typing import Optional, Dict, Any, List
+
+from ...core.abstract_device_state import AbstractDeviceState
+
+
+class WebDeviceState(AbstractDeviceState):
+    """
+    Web 设备状态的具体实现
+    """
+    
+    def __init__(self, device, views: List[Dict[str, Any]] = None,
+                 tag: str = None, screenshot_path: str = None):
+        """
+        初始化 Web 设备状态
+        """
+        super().__init__(device, tag=tag, screenshot_path=screenshot_path)
+        
+        self._views = views or []
+        self._window_title = device.get_page_title() if device else ""
+        self._url = device.get_current_url() if device else ""
+
+        self._generate_view_strs()
+        
+        # 缓存
+        self._state_str = None
+        self._structure_str = None
+    
+    # ==================== 视图信息 ====================
+    
+    @property
+    def views(self) -> List[Dict[str, Any]]:
+        """获取视图列表"""
+        return self._views
+        
+    # ==================== 状态标识 ====================
+
+    @property
+    def state_str(self) -> str:
+        """获取状态的唯一标识字符串"""
+        if self._state_str is None:
+            view_signatures = sorted(self._get_view_signature(view) for view in self._views)
+            state_raw = f"url={self._url}&title={self._window_title}&views=" + ",".join(view_signatures)
+            self._state_str = hashlib.md5(state_raw.encode('utf-8')).hexdigest()
+        return self._state_str
+    
+    @property
+    def structure_str(self) -> str:
+        """获取状态的结构标识（忽略控件文本内容）"""
+        if self._structure_str is None:
+            view_signatures = sorted(self._get_content_free_view_signature(view) for view in self._views)
+            structure_raw = f"url={self._url}&title={self._window_title}&structure=" + ",".join(view_signatures)
+            self._structure_str = hashlib.md5(structure_raw.encode('utf-8')).hexdigest()
+        return self._structure_str
+    
+    @property
+    def foreground_page(self) -> Optional[str]:
+        """返回当前页面 URL 作为页面标识"""
+        return self._url
+    
+    @property
+    def search_content(self) -> str:
+        """获取用于搜索的文本内容"""
+        texts = []
+        for view in self._views:
+            text = view.get('text', '')
+            if text:
+                texts.append(text)
+        if texts:
+            return ' '.join(texts)
+        return self._window_title
+
+    def _generate_view_strs(self):
+        """为每个 DOM 控件生成稳定标识。"""
+        for idx, view in enumerate(self._views):
+            bounds = view.get('bounds', [[0, 0], [0, 0]])
+            x1, y1 = bounds[0]
+            x2, y2 = bounds[1]
+            class_name = view.get('class_name', 'dom')
+            resource_id = view.get('resource_id', '')
+            text = (view.get('text') or view.get('content_description') or '')[:30]
+            view['temp_id'] = idx
+            if 'children' not in view or not isinstance(view['children'], list):
+                view['children'] = []
+            if 'parent' not in view:
+                view['parent'] = -1
+            if 'view_str' not in view or not view['view_str']:
+                view['view_str'] = f"web_{idx}_{class_name}_{resource_id}_{x1}_{y1}_{x2}_{y2}_{text}"
+
+    @staticmethod
+    def _get_view_signature(view_dict: Dict[str, Any]) -> str:
+        """获取带文本内容的视图签名。"""
+        if 'signature' in view_dict:
+            return view_dict['signature']
+
+        bounds = view_dict.get('bounds', [[0, 0], [0, 0]])
+        text = view_dict.get('text', '')
+        class_name = view_dict.get('class_name', 'unknown')
+        clickable = view_dict.get('clickable', False)
+        editable = view_dict.get('editable', False)
+        scrollable = view_dict.get('scrollable', False)
+
+        signature = f"{class_name}:{bounds}:{text}:{clickable}:{editable}:{scrollable}"
+        view_dict['signature'] = signature
+        return signature
+
+    @staticmethod
+    def _get_content_free_view_signature(view_dict: Dict[str, Any]) -> str:
+        """获取忽略文本内容的视图签名。"""
+        bounds = view_dict.get('bounds', [[0, 0], [0, 0]])
+        class_name = view_dict.get('class_name', 'unknown')
+        clickable = view_dict.get('clickable', False)
+        editable = view_dict.get('editable', False)
+        scrollable = view_dict.get('scrollable', False)
+        return f"{class_name}:{bounds}:{clickable}:{editable}:{scrollable}"
+    
+    # ==================== 输入事件 ====================
+    
+    def get_possible_input(self) -> List:
+        """获取当前状态可能的输入事件列表"""
+        from .web_input_event import (
+            WebTouchEvent, WebScrollEvent, WebKeyEvent, WebSetTextEvent
+        )
+        
+        if self._possible_events:
+            return [] + self._possible_events
+
+        possible_events = []
+        enabled_view_ids = []
+        touch_exclude_view_ids = set()
+
+        for view in self._views:
+            if view.get('enabled', True) and view.get('visible', True):
+                enabled_view_ids.append(view['temp_id'])
+
+        for view_id in enabled_view_ids:
+            view = self._views[view_id]
+            if view.get('clickable', False):
+                possible_events.append(WebTouchEvent(view=view))
+                touch_exclude_view_ids.add(view_id)
+                touch_exclude_view_ids.update(view.get('children', []))
+        
+        # 添加滚动事件
+        possible_events.append(WebScrollEvent(direction='up'))
+        possible_events.append(WebScrollEvent(direction='down'))
+
+        for view_id in enabled_view_ids:
+            view = self._views[view_id]
+            if view.get('editable', False):
+                possible_events.append(WebSetTextEvent(view=view, text="test"))
+                touch_exclude_view_ids.add(view_id)
+
+        for view_id in enabled_view_ids:
+            if view_id in touch_exclude_view_ids:
+                continue
+            view = self._views[view_id]
+            children = view.get('children', [])
+            if children:
+                continue
+            possible_events.append(WebTouchEvent(view=view))
+        
+        # 添加常用按键事件
+        possible_events.append(WebKeyEvent('ESCAPE'))
+        possible_events.append(WebKeyEvent('ENTER'))
+        
+        self._possible_events = possible_events
+        return [] + possible_events
+    
+    # ==================== 序列化 ====================
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """序列化为字典"""
+        return {
+            'tag': self.tag,
+            'url': self._url,
+            'window_title': self._window_title,
+            'state_str': self.state_str,
+            'structure_str': self.structure_str,
+            'foreground_page': self.foreground_page,
+            'views': self._views,
+            'screenshot_path': self.screenshot_path,
+            'width': self.width,
+            'height': self.height,
+        }
+    
+    # ==================== 应用信息 ====================
+    
+    def get_app_page_depth(self) -> int:
+        """
+        获取应用页面深度
+        
+        对于 Web，只要没有超出域名都视为正常应用内。
+        返回 0。
+        """
+        return 0

DroidBot/platforms/web/web_input_event.py

@@ -0,0 +1,453 @@
+"""
+Web Input Event Implementation
+Concrete implementation of AbstractInputEvent for Web applications.
+"""
+from typing import Optional, Dict, Any, List
+from selenium.webdriver.common.action_chains import ActionChains
+from selenium.webdriver.common.keys import Keys
+from selenium.webdriver.common.by import By
+
+from ...core.abstract_input_event import (
+    AbstractInputEvent, EventType, BaseTouchEvent, BaseLongTouchEvent, 
+    BaseSwipeEvent, BaseScrollEvent, BaseSetTextEvent, BaseKeyEvent,
+    BaseKillAppEvent
+)
+
+
+class WebTouchEvent(BaseTouchEvent):
+    """
+    Web 点击事件实现
+    """
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """
+        发送点击事件到 Web 设备
+        
+        :param device: Web 设备对象
+        :return: 是否发送成功
+        """
+        try:
+            # 如果有视图信息，通过坐标查找元素并点击
+            if (self.view or (self.x is not None and self.y is not None)) and device.driver:
+                if self.view:
+                    bounds = self.view.get('bounds', [[0, 0], [0, 0]])
+                    x = (bounds[0][0] + bounds[1][0]) / 2
+                    y = (bounds[0][1] + bounds[1][1]) / 2
+                else: # self.x and self.y
+                    x = self.x
+                    y = self.y
+                
+                if getattr(device, "engine", "selenium") == "playwright":
+                    device.driver.mouse.click(x, y)
+                    return True
+                
+                # 使用JavaScript执行点击，避免坐标越界问题
+                js_script = f"""
+                var el = document.elementFromPoint({x}, {y});
+                if (el) {{
+                    el.click();
+                    return true;
+                }}
+                return false;
+                """
+                success = device.driver.execute_script(js_script)
+                if not success:
+                    device.logger.warning(f"Could not find element at ({x}, {y}) to click.")
+                
+                return success
+            return False
+        except Exception as e:
+            device.logger.error(f"Failed to send Web touch event: {e}")
+            return False
+
+
+class WebLongTouchEvent(BaseLongTouchEvent):
+    """
+    Web 长按事件实现
+    """
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """
+        发送长按事件到 Web 设备
+        
+        :param device: Web 设备对象
+        :return: 是否发送成功
+        """
+        try:
+            x, y = None, None
+            if self.view:
+                bounds = self.view.get('bounds', [[0, 0], [0, 0]])
+                x = (bounds[0][0] + bounds[1][0]) / 2
+                y = (bounds[0][1] + bounds[1][1]) / 2
+            elif self.x is not None and self.y is not None:
+                x = self.x
+                y = self.y
+                
+            if x is not None and y is not None and device.driver:
+                if getattr(device, "engine", "selenium") == "playwright":
+                    device.driver.mouse.move(x, y)
+                    device.driver.mouse.down()
+                    import time
+                    time.sleep(self.duration / 1000.0)
+                    device.driver.mouse.up()
+                    return True
+                
+                # 使用JavaScript模拟长按事件
+                element = device.driver.execute_script(f"return document.elementFromPoint({x}, {y});")
+                if element:
+                    js_script = """
+                    var element = arguments[0];
+                    var duration = arguments[1];
+                    
+                    var touchStart = new TouchEvent('touchstart', {
+                        bubbles: true, cancelable: true,
+                        touches: [{ clientX: arguments[2], clientY: arguments[3] }]
+                    });
+                    
+                    var touchEnd = new TouchEvent('touchend', {
+                        bubbles: true, cancelable: true
+                    });
+                    
+                    element.dispatchEvent(touchStart);
+                    
+                    setTimeout(function() {
+                        element.dispatchEvent(touchEnd);
+                    }, duration);
+                    """
+                    
+                    device.driver.execute_script(js_script, element, self.duration, x, y)
+                    return True
+            return False
+        except Exception as e:
+            device.logger.error(f"Failed to send Web long touch event: {e}")
+            return False
+
+
+class WebSwipeEvent(BaseSwipeEvent):
+    """
+    Web 滑动事件实现
+    """
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """
+        发送滑动事件到 Web 设备
+        
+        :param device: Web 设备对象
+        :return: 是否发送成功
+        """
+        try:
+            if device.driver:
+                if getattr(device, "engine", "selenium") == "playwright":
+                    device.driver.mouse.move(self.start_x, self.start_y)
+                    device.driver.mouse.down()
+                    import time
+                    time.sleep(0.05)
+                    # move in steps for swipe gesture
+                    device.driver.mouse.move(self.end_x, self.end_y, steps=10)
+                    time.sleep(0.1)
+                    device.driver.mouse.up()
+                    return True
+                
+                # 使用JavaScript模拟滑动事件
+                js_script = f"""
+                // 创建触摸开始事件
+                var touchStart = new TouchEvent('touchstart', {{
+                    bubbles: true,
+                    cancelable: true,
+                    touches: [{{
+                        clientX: {self.start_x},
+                        clientY: {self.start_y}
+                    }}]
+                }});
+                
+                // 创建触摸移动事件
+                var touchMove = new TouchEvent('touchmove', {{
+                    bubbles: true,
+                    cancelable: true,
+                    touches: [{{
+                        clientX: {self.end_x},
+                        clientY: {self.end_y}
+                    }}]
+                }});
+                
+                // 创建触摸结束事件
+                var touchEnd = new TouchEvent('touchend', {{
+                    bubbles: true,
+                    cancelable: true
+                }});
+                
+                // 在body元素上分发事件
+                document.body.dispatchEvent(touchStart);
+                
+                // 短暂延迟后分发移动事件
+                setTimeout(function() {{
+                    document.body.dispatchEvent(touchMove);
+                    
+                    // 再次延迟后分发结束事件
+                    setTimeout(function() {{
+                        document.body.dispatchEvent(touchEnd);
+                    }}, 100);
+                }}, 50);
+                """
+                
+                device.driver.execute_script(js_script)
+                return True
+            return False
+        except Exception as e:
+            device.logger.error(f"Failed to send Web swipe event: {e}")
+            return False
+
+
+class WebScrollEvent(BaseScrollEvent):
+    """
+    Web 滚动事件实现
+    """
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """
+        发送滚动事件到 Web 设备
+        
+        :param device: Web 设备对象
+        :return: 是否发送成功
+        """
+        try:
+            if device.driver:
+                if getattr(device, "engine", "selenium") == "playwright":
+                    if self.direction == self.DIRECTION_UP:
+                        device.driver.keyboard.press("PageUp")
+                    elif self.direction == self.DIRECTION_DOWN:
+                        device.driver.keyboard.press("PageDown")
+                    elif self.direction == self.DIRECTION_LEFT:
+                        device.driver.keyboard.press("ArrowLeft")
+                    elif self.direction == self.DIRECTION_RIGHT:
+                        device.driver.keyboard.press("ArrowRight")
+                    return True
+
+                # 使用 ActionChains 执行滚动操作
+                actions = ActionChains(device.driver)
+                
+                # 根据方向执行滚动
+                if self.direction == self.DIRECTION_UP:
+                    actions.send_keys(Keys.PAGE_UP)
+                elif self.direction == self.DIRECTION_DOWN:
+                    actions.send_keys(Keys.PAGE_DOWN)
+                elif self.direction == self.DIRECTION_LEFT:
+                    actions.send_keys(Keys.ARROW_LEFT)
+                elif self.direction == self.DIRECTION_RIGHT:
+                    actions.send_keys(Keys.ARROW_RIGHT)
+                
+                actions.perform()
+                return True
+            return False
+        except Exception as e:
+            device.logger.error(f"Failed to send Web scroll event: {e}")
+            return False
+
+
+class WebSetTextEvent(BaseSetTextEvent):
+    """
+    Web 文本输入事件实现
+    """
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """
+        发送文本输入事件到 Web 设备
+        
+        :param device: Web 设备对象
+        :return: 是否发送成功
+        """
+        try:
+            if self.view and device.driver:
+                bounds = self.view.get('bounds', [[0, 0], [0, 0]])
+                x = (bounds[0][0] + bounds[1][0]) / 2
+                y = (bounds[0][1] + bounds[1][1]) / 2
+                
+                if getattr(device, "engine", "selenium") == "playwright":
+                    escaped_text = self.text.replace('`', '\\`').replace('$', '\\$')
+                    js_script = f"""() => {{
+                        var element = document.elementFromPoint({x}, {y});
+                        if (element) {{
+                            element.focus();
+                            element.value = '';
+                        }}
+                    }}"""
+                    device.driver.evaluate(js_script)
+                    device.driver.keyboard.type(self.text)
+                    device.driver.evaluate(f"""() => {{
+                        var element = document.elementFromPoint({x}, {y});
+                        if (element) {{
+                            element.dispatchEvent(new Event('input', {{ bubbles: true }}));
+                            element.dispatchEvent(new Event('change', {{ bubbles: true }}));
+                        }}
+                    }}""")
+                    return True
+                
+                # 使用JavaScript找到元素并设置文本
+                element = device.driver.execute_script(f"return document.elementFromPoint({x}, {y});")
+                if element:
+                    # 先点击元素以获取焦点
+                    try:
+                        device.driver.execute_script("arguments[0].click();", element)
+                    except Exception as e:
+                        device.logger.warning(f"Failed to focus element before setting text: {e}")
+                    
+                    # 清空现有文本并输入新文本
+                    device.driver.execute_script("""
+                        var element = arguments[0];
+                        var text = arguments[1];
+                        
+                        // 清空现有文本
+                        element.value = '';
+                        
+                        // 设置新文本
+                        element.value = text;
+                        
+                        // 触发input和change事件
+                        var event = new Event('input', { bubbles: true });
+                        element.dispatchEvent(event);
+                        
+                        var changeEvent = new Event('change', { bubbles: true });
+                        element.dispatchEvent(changeEvent);
+                    """, element, self.text)
+                    return True
+            return False
+        except Exception as e:
+            device.logger.error(f"Failed to send Web set text event: {e}")
+            return False
+
+
+class WebKeyEvent(BaseKeyEvent):
+    """
+    Web 按键事件实现
+    """
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """
+        发送按键事件到 Web 设备
+        
+        :param device: Web 设备对象
+        :return: 是否发送成功
+        """
+        try:
+            if device.driver:
+                if getattr(device, "engine", "selenium") == "playwright":
+                    if self.key_name == self.KEY_BACK:
+                        device.driver.go_back()
+                        return True
+                    
+                    key_map = {
+                        self.KEY_HOME: "Home",
+                        self.KEY_ENTER: "Enter",
+                        self.KEY_ESCAPE: "Escape"
+                    }
+                    key = key_map.get(self.key_name, self.key_name)
+                    device.driver.keyboard.press(key)
+                    return True
+                
+                # 使用 ActionChains 执行按键操作
+                actions = ActionChains(device.driver)
+                
+                # 映射按键名称到 Keys 常量
+                key_map = {
+                    self.KEY_HOME: Keys.HOME,
+                    self.KEY_ENTER: Keys.ENTER,
+                    self.KEY_ESCAPE: Keys.ESCAPE
+                }
+                
+                if self.key_name == self.KEY_BACK:
+                    # 对于 Web 平台，BACK 通常意味着浏览器后退
+                    device.back()
+                    return True
+                
+                key = key_map.get(self.key_name, self.key_name)
+                actions.send_keys(key)
+                actions.perform()
+                return True
+            return False
+        except Exception as e:
+            device.logger.error(f"Failed to send Web key event: {e}")
+            return False
+
+
+class WebKillAppEvent(BaseKillAppEvent):
+    """
+    Web 应用重置事件
+    
+    清除浏览器状态（Cookies/Storage）、关闭多余标签页、导航回初始页面。
+    对应 DroidBot 框架中的 KillAppEvent，在探索循环启动和停滞恢复时调用。
+    """
+    
+    def send(self, device: 'AbstractDevice') -> bool:
+        """
+        重置 Web 应用状态
+        
+        :param device: Web 设备对象
+        :return: 是否发送成功
+        """
+        try:
+            if not device.driver:
+                device.logger.warning("WebKillAppEvent: driver not available")
+                return False
+            
+            # 1. 清除浏览器存储状态
+            try:
+                if getattr(device, "engine", "selenium") == "playwright":
+                    device.driver.evaluate("try { localStorage.clear(); } catch(e) {}")
+                    device.driver.evaluate("try { sessionStorage.clear(); } catch(e) {}")
+                    if hasattr(device, 'playwright_context') and device.playwright_context:
+                        device.playwright_context.clear_cookies()
+                else:
+                    device.driver.execute_script(
+                        "try { localStorage.clear(); } catch(e) {}"
+                        "try { sessionStorage.clear(); } catch(e) {}"
+                    )
+                    device.driver.delete_all_cookies()
+            except Exception as e:
+                device.logger.warning(f"清除浏览器存储失败（可忽略）: {e}")
+            
+            # 2. 关闭所有多余标签页，只保留第一个
+            try:
+                if getattr(device, "engine", "selenium") == "playwright":
+                    if hasattr(device, 'playwright_context') and device.playwright_context:
+                        pages = device.playwright_context.pages
+                        if len(pages) > 1:
+                            for p in pages[1:]:
+                                p.close()
+                            device.driver = pages[0]
+                            device.driver.bring_to_front()
+                            device.logger.info(f"关闭了 {len(pages) - 1} 个多余标签页")
+                else:
+                    handles = device.driver.window_handles
+                    if len(handles) > 1:
+                        first_handle = handles[0]
+                        for h in handles[1:]:
+                            device.driver.switch_to.window(h)
+                            device.driver.close()
+                        device.driver.switch_to.window(first_handle)
+                        device.logger.info(f"关闭了 {len(handles) - 1} 个多余标签页")
+            except Exception as e:
+                device.logger.warning(f"关闭多余标签页失败: {e}")
+            
+            # 3. 导航回初始页面
+            app_url = device.app_url
+            if app_url:
+                if getattr(device, "engine", "selenium") == "playwright":
+                    device.driver.goto(app_url)
+                    if hasattr(device, "_wait_for_page_ready"):
+                        device._wait_for_page_ready()
+                    device._current_url = device.driver.url
+                else:
+                    device.driver.get(app_url)
+                    if hasattr(device, "_wait_for_page_ready"):
+                        device._wait_for_page_ready()
+                    device._current_url = device.driver.current_url
+                device.logger.info(f"已导航回初始页面: {app_url}")
+            
+            # 4. 更新窗口句柄缓存
+            if getattr(device, "engine", "selenium") != "playwright":
+                device._window_handles = set(device.driver.window_handles)
+            
+            return True
+        except Exception as e:
+            device.logger.error(f"WebKillAppEvent 执行失败: {e}")
+            return False

DroidBot/platforms/windows/__init__.py

@@ -0,0 +1,46 @@
+"""
+Windows Platform Module
+Platform-specific implementations for Windows desktop applications.
+"""
+
+from .windows_device import WindowsDevice
+from .windows_device_state import WindowsDeviceState
+from .windows_input_event import (
+    WindowsTouchEvent,
+    WindowsLongTouchEvent,
+    WindowsSwipeEvent,
+    WindowsScrollEvent,
+    WindowsSetTextEvent,
+    WindowsKeyEvent,
+    WindowsKillAppEvent,
+)
+
+# 注册 Windows 平台到工厂
+from ...core.platform_factory import PlatformFactory, Platform
+
+PlatformFactory.register_platform(
+    Platform.WINDOWS,
+    WindowsDevice,
+    WindowsDeviceState,
+    {
+        'touch': WindowsTouchEvent,
+        'long_touch': WindowsLongTouchEvent,
+        'swipe': WindowsSwipeEvent,
+        'scroll': WindowsScrollEvent,
+        'set_text': WindowsSetTextEvent,
+        'key': WindowsKeyEvent,
+        'kill_app': WindowsKillAppEvent,
+    }
+)
+
+__all__ = [
+    'WindowsDevice',
+    'WindowsDeviceState',
+    'WindowsTouchEvent',
+    'WindowsLongTouchEvent',
+    'WindowsSwipeEvent',
+    'WindowsScrollEvent',
+    'WindowsSetTextEvent',
+    'WindowsKeyEvent',
+    'WindowsKillAppEvent',
+]

DroidBot/platforms/windows/windows_device.py

@@ -0,0 +1,674 @@
+"""
+Windows Device Implementation
+Concrete implementation of AbstractDevice for Windows desktop applications.
+Uses CV mode for UI element detection.
+"""
+import logging
+import os
+import re
+import subprocess
+import sys
+import time
+from typing import Optional, Dict, Any, List, Tuple
+
+from ...core.abstract_device import AbstractDevice
+
+class WindowsApp:
+    """
+    Windows 应用封装类
+    用于兼容 DroidBot 的 App 接口
+    """
+    def __init__(self, window_title):
+        self._window_title = window_title
+        self.main_activity = None
+        self.activities = []
+        
+    def get_package_name(self):
+        return self._window_title
+
+
+class WindowsDevice(AbstractDevice):
+    """
+    Windows 设备的具体实现（CV 模式）
+    
+    使用 OmniParser 进行 UI 元素检测，pyautogui 进行输入模拟。
+    """
+    
+    def __init__(self, 
+                 window_title: str = None,
+                 exe_path: str = None,
+                 output_dir: str = None,
+                 cv_mode: bool = True,
+                 steam_game_id: str = None,  # Steam 游戏 ID
+
+                 **kwargs):
+        """
+        初始化 Windows 设备连接
+        
+        :param window_title: 目标窗口标题（支持部分匹配）
+        :param exe_path: 可执行文件路径（用于启动应用）
+        :param output_dir: 输出目录
+        :param cv_mode: 是否使用 CV 模式（默认 True）
+        :param steam_game_id: Steam 游戏 ID，例如 '730' (CS2)
+        :param app_path: (兼容参数)
+        :param device_serial: (兼容参数)
+        :param kwargs: 忽略的其他参数
+        """
+        super().__init__(output_dir)
+        
+        self._window_title = window_title 
+        self._exe_path = exe_path 
+        self._steam_game_id = steam_game_id
+        self.cv_mode = cv_mode
+        
+
+        # 兼容性：初始化 _app 对象
+        self._app = WindowsApp(self._window_title) 
+        
+        # 窗口句柄相关
+        self._hwnd = None
+        self._window_rect = None
+        
+        # 显示信息缓存
+        self._display_info = None
+        
+        # 截图计数器
+        self._screenshot_count = 0
+        
+        # 控制标志
+        self.pause_sending_event = False
+        
+        self.logger = logging.getLogger(self.__class__.__name__)
+    
+    def get_platform_name(self) -> str:
+        """获取平台名称"""
+        return "windows"
+    
+    # ==================== 连接管理 ====================
+    
+    def set_up(self) -> None:
+        """设置设备连接前的准备工作"""
+        # 检查依赖库
+        try:
+            import pyautogui
+            import mss
+            import win32gui
+            import win32con
+        except ImportError as e:
+            raise ImportError(
+                f"Missing required library: {e}. "
+                "Please install: pip install pyautogui mss pywin32"
+            )
+        
+        # 禁用 pyautogui 的安全暂停
+        pyautogui.PAUSE = 0.1
+        pyautogui.FAILSAFE = True
+
+        # 预加载必要的 win32 API
+        try:
+            import win32process
+            import win32api
+        except ImportError:
+            pass
+    
+    def connect(self) -> bool:
+        """连接到目标窗口"""
+        self.set_up()
+        
+        # 查找目标窗口
+        self._hwnd = self._find_window()
+        
+        if self._hwnd is None:
+            self.logger.warning(f"Window not found: {self._window_title}")
+            # 如果提供了 exe_path，尝试启动应用
+            if self._exe_path and os.path.exists(self._exe_path):
+                self.logger.info(f"Attempting to start app: {self._exe_path}")
+                if self.start_app():
+                    time.sleep(2)  # 等待应用启动
+                    self._hwnd = self._find_window()
+        
+        if self._hwnd:
+            self.connected = True
+            self._update_window_rect()
+            self.logger.info(f"Connected to window: {self._get_window_title()}")
+            return True
+        else:
+            self.connected = False
+            self.logger.error("Failed to connect to window")
+            return False
+    
+
+
+    def run_initial_setup(self) -> bool:
+        from DroidBot.guiagent_bridge import GuiAgentBridge
+
+        bridge = GuiAgentBridge(device=self, app=self._app, app_name=self._window_title)
+        current_state = self.get_current_state()
+        return bridge.handle_with_guiagent("game_initial", {"state": current_state})
+
+    def disconnect(self) -> None:
+        """断开设备连接"""
+        self._hwnd = None
+        self._window_rect = None
+        self.connected = False
+        self.logger.info("Disconnected from window")
+    
+    def tear_down(self) -> None:
+        """清理设备资源"""
+        self.disconnect()
+    
+    def check_connectivity(self) -> bool:
+        """检查窗口是否仍然存在 (包含句柄自愈机制)"""
+        import win32gui
+        
+        # 1. 检查现有句柄是否有效
+        if self._hwnd and win32gui.IsWindow(self._hwnd):
+            return True
+            
+        # 2. 句柄失效，尝试自愈 (重新查找窗口)
+        # 这种情况常见于游戏崩溃重启、更新重启、或从 Launcher 切换到 Game 主窗口
+        self.logger.warning(f"Window handle {self._hwnd} invalid, attempting to reconnect to '{self._window_title}'...")
+        new_hwnd = self._find_window()
+        
+        if new_hwnd:
+            self._hwnd = new_hwnd
+            self.logger.info(f"Reconnected to window: {self._hwnd}")
+            return True
+            
+        return False
+    
+    # ==================== 窗口查找 ====================
+    
+    def _find_window(self) -> Optional[int]:
+        """查找目标窗口句柄 (优先完全匹配)"""
+        import win32gui
+        
+        if self._window_title is None:
+            return None
+        
+        exact_matches = []
+        partial_matches = []
+        
+        def enum_callback(hwnd, _):
+            if win32gui.IsWindowVisible(hwnd):
+                title = win32gui.GetWindowText(hwnd)
+                if not title:
+                    return True
+
+                # 优先完全匹配 (忽略大小写)
+                if self._window_title.lower() == title.lower():
+                    exact_matches.append((hwnd, title))
+                # 其次部分匹配
+                elif self._window_title.lower() in title.lower():
+                    partial_matches.append((hwnd, title))
+            return True
+        
+        try:
+            win32gui.EnumWindows(enum_callback, None)
+        except Exception as e:
+            self.logger.error(f"EnumWindows failed: {e}")
+            return None
+        
+        # 1. 优先返回完全匹配
+        if exact_matches:
+            self.logger.info(f"Found exact match window: {exact_matches[0][1]} ({exact_matches[0][0]})")
+            return exact_matches[0][0]
+            
+        # 2. 其次返回部分匹配
+        if partial_matches:
+            self.logger.info(f"Found partial match window: {partial_matches[0][1]} ({partial_matches[0][0]})")
+            return partial_matches[0][0]
+        
+        return None
+    
+    def _get_window_title(self) -> str:
+        """获取当前窗口标题"""
+        if self._hwnd is None:
+            return ""
+        
+        import win32gui
+        try:
+            return win32gui.GetWindowText(self._hwnd)
+        except Exception:
+            return ""
+    
+    def _update_window_rect(self) -> None:
+        """更新窗口位置和大小"""
+        if self._hwnd is None:
+            return
+        
+        import win32gui
+        try:
+            rect = win32gui.GetWindowRect(self._hwnd)
+            self._window_rect = {
+                'left': rect[0],
+                'top': rect[1],
+                'right': rect[2],
+                'bottom': rect[3],
+                'width': rect[2] - rect[0],
+                'height': rect[3] - rect[1],
+            }
+        except Exception as e:
+            self.logger.warning(f"Failed to get window rect: {e}")
+    
+    # ==================== 状态获取 ====================
+
+    def check_network(self, host: str = "8.8.8.8") -> bool:
+        return True
+    
+    def get_current_state(self):
+        """获取当前设备状态"""
+        from .windows_device_state import WindowsDeviceState
+        from ...cv import cv
+        
+        # 更新窗口位置
+        self._update_window_rect()
+        
+        # 截图
+        screenshot_path = self.take_screenshot()
+        
+        # CV 检测
+        cv_views = []
+        if screenshot_path and os.path.exists(screenshot_path):
+            try:
+                img = cv.load_image_from_path(screenshot_path)
+                if img is not None:
+                    cv_views = cv.find_views(img)
+                    self.logger.info(f"CV detected {len(cv_views)} views")
+            except Exception as e:
+                self.logger.warning(f"CV detection failed: {e}")
+        
+        # 创建状态对象
+        state = WindowsDeviceState(
+            device=self,
+            cv_views=cv_views,
+            window_title=self._get_window_title(),
+            screenshot_path=screenshot_path
+        )
+        
+        return state
+    
+    def get_display_info(self, refresh: bool = False) -> Dict[str, Any]:
+        """获取显示信息"""
+        if self._display_info is None or refresh:
+            self._update_window_rect()
+            
+            if self._window_rect:
+                self._display_info = {
+                    'width': self._window_rect['width'],
+                    'height': self._window_rect['height'],
+                    'left': self._window_rect['left'],
+                    'top': self._window_rect['top'],
+                }
+            else:
+                # 默认值
+                self._display_info = {
+                    'width': 1920,
+                    'height': 1080,
+                    'left': 0,
+                    'top': 0,
+                }
+        
+        return self._display_info
+    
+    # ==================== 屏幕操作 ====================
+    
+    def take_screenshot(self, path: str = None) -> Optional[str]:
+        """截取窗口屏幕"""
+        import mss
+        
+        if path is None:
+            if self.output_dir:
+                self._screenshot_count += 1
+                path = os.path.join(
+                    self.output_dir, 
+                    f"screen_{self._screenshot_count}.png"
+                )
+            else:
+                path = f"screen_{int(time.time())}.png"
+        
+        # 确保目录存在
+        os.makedirs(os.path.dirname(path) if os.path.dirname(path) else '.', exist_ok=True)
+        
+        self._update_window_rect()
+        
+        try:
+            with mss.mss() as sct:
+                if self._window_rect:
+                    # 截取窗口区域
+                    monitor = {
+                        'left': self._window_rect['left'],
+                        'top': self._window_rect['top'],
+                        'width': self._window_rect['width'],
+                        'height': self._window_rect['height'],
+                    }
+                else:
+                    # 截取整个屏幕
+                    monitor = sct.monitors[1]
+                
+                screenshot = sct.grab(monitor)
+                
+                # 保存截图
+                from PIL import Image
+                img = Image.frombytes('RGB', screenshot.size, screenshot.bgra, 'raw', 'BGRX')
+                img.save(path)
+                
+                self.logger.debug(f"Screenshot saved to: {path}")
+                return path
+                
+        except Exception as e:
+            self.logger.error(f"Failed to take screenshot: {e}")
+            return None
+    
+    def unlock(self) -> None:
+        """解锁屏幕（Windows 上无需实现）"""
+        pass
+    
+    # ==================== 输入操作 ====================
+    
+    def send_event(self, event) -> bool:
+        """发送输入事件"""
+        return event.send(self)
+    
+    def view_touch(self, x: int, y: int, button: str = 'left') -> None:
+        """点击指定坐标"""
+        import pyautogui
+        
+        # 如果是相对于窗口的坐标，转换为屏幕坐标
+        if self._window_rect:
+            screen_x = self._window_rect['left'] + x
+            screen_y = self._window_rect['top'] + y
+        else:
+            screen_x, screen_y = x, y
+        
+        pyautogui.click(screen_x, screen_y, button=button)
+        self.logger.debug(f"Touch ({button}) at ({screen_x}, {screen_y})")
+    
+    def view_long_touch(self, x: int, y: int, duration: int = 2000) -> None:
+        """长按指定坐标"""
+        import pyautogui
+        
+        if self._window_rect:
+            screen_x = self._window_rect['left'] + x
+            screen_y = self._window_rect['top'] + y
+        else:
+            screen_x, screen_y = x, y
+        
+        # 移动到位置，按下，等待，松开
+        pyautogui.moveTo(screen_x, screen_y)
+        pyautogui.mouseDown()
+        time.sleep(duration / 1000.0)
+        pyautogui.mouseUp()
+        
+        self.logger.debug(f"Long touch at ({screen_x}, {screen_y}) for {duration}ms")
+    
+    def view_drag(self, start_xy: Tuple[int, int], end_xy: Tuple[int, int], 
+                  duration: int = 500) -> None:
+        """拖拽操作"""
+        import pyautogui
+        
+        start_x, start_y = start_xy
+        end_x, end_y = end_xy
+        
+        if self._window_rect:
+            start_x += self._window_rect['left']
+            start_y += self._window_rect['top']
+            end_x += self._window_rect['left']
+            end_y += self._window_rect['top']
+        
+        pyautogui.moveTo(start_x, start_y)
+        pyautogui.drag(
+            end_x - start_x, 
+            end_y - start_y, 
+            duration=duration / 1000.0
+        )
+        
+        self.logger.debug(f"Drag from ({start_x}, {start_y}) to ({end_x}, {end_y})")
+    
+    def view_set_text(self, text: str) -> None:
+        """输入文本"""
+        import pyautogui
+        
+        # 对于中文等非ASCII字符，使用 pyperclip 和 Ctrl+V
+        try:
+            # 检查是否包含非ASCII字符
+            text.encode('ascii')
+            # 纯ASCII字符，使用 typewrite
+            pyautogui.typewrite(text, interval=0.05)
+        except UnicodeEncodeError:
+            # 包含非ASCII字符，使用剪贴板
+            import pyperclip
+            pyperclip.copy(text)
+            pyautogui.hotkey('ctrl', 'v')
+        
+        self.logger.debug(f"Set text: {text[:20]}...")
+    
+    def key_press(self, key_code: str) -> None:
+        """按键操作"""
+        import pyautogui
+        pyautogui.press(key_code)
+        self.logger.debug(f"Key press: {key_code}")
+    
+    # ==================== 应用管理 ====================
+    
+    @property
+    def app_identifier(self) -> str:
+        """获取应用标识符"""
+        if self._window_title:
+            return self._window_title
+        if self._exe_path:
+            return os.path.basename(self._exe_path)
+        return "unknown"
+    
+    def is_foreground(self) -> bool:
+        """检查目标窗口是否在前台"""
+        if self._hwnd is None:
+            return False
+        
+        import win32gui
+        try:
+            foreground_hwnd = win32gui.GetForegroundWindow()
+            return foreground_hwnd == self._hwnd
+        except Exception:
+            return False
+    
+    def pull_back_to_app(self) -> bool:
+        """
+        将目标窗口拉到前台 (多策略尝试)
+        """
+        if self._hwnd is None:
+            # 尝试重新查找窗口
+            self._hwnd = self._find_window()
+            if self._hwnd is None:
+                self.logger.warning("Cannot find window to bring to foreground")
+                return False
+        
+        import win32gui
+        import win32con
+        import win32process
+        import win32api
+        import pyautogui
+        
+        # 先检查窗口句柄是否仍然有效
+        if not win32gui.IsWindow(self._hwnd):
+            self.logger.warning(f"Window handle {self._hwnd} is no longer valid")
+            self._hwnd = self._find_window()
+            if self._hwnd is None:
+                return False
+        
+        # 检查当前是否已经是前台窗口
+        try:
+            current_foreground = win32gui.GetForegroundWindow()
+            if current_foreground == self._hwnd:
+                self.logger.debug("Window is already in foreground")
+                return True
+        except Exception:
+            pass
+        
+        self.logger.info(f"Attempting to bring window {self._hwnd} to foreground...")
+        
+        # === Method 1: AttachThreadInput + SetForegroundWindow ===
+        try:
+            current_thread_id = win32api.GetCurrentThreadId()
+            target_thread_id, target_process_id = win32process.GetWindowThreadProcessId(self._hwnd)
+            foreground_hwnd = win32gui.GetForegroundWindow()
+            foreground_thread_id, _ = win32process.GetWindowThreadProcessId(foreground_hwnd)
+            
+            attached_to_foreground = False
+            attached_to_target = False
+            
+            try:
+                # 先依附到当前前台窗口的线程（获取输入权限）
+                if current_thread_id != foreground_thread_id:
+                    win32process.AttachThreadInput(current_thread_id, foreground_thread_id, True)
+                    attached_to_foreground = True
+                
+                # 再依附到目标窗口的线程
+                if current_thread_id != target_thread_id:
+                    win32process.AttachThreadInput(current_thread_id, target_thread_id, True)
+                    attached_to_target = True
+                
+                # 如果最小化了，先还原
+                if win32gui.IsIconic(self._hwnd):
+                    win32gui.ShowWindow(self._hwnd, win32con.SW_RESTORE)
+                
+                # 显示窗口
+                win32gui.ShowWindow(self._hwnd, win32con.SW_SHOW)
+                
+                # 尝试多种置顶方法
+                win32gui.BringWindowToTop(self._hwnd)
+                win32gui.SetForegroundWindow(self._hwnd)
+                
+            finally:
+                # 务必解除依附
+                if attached_to_target:
+                    try:
+                        win32process.AttachThreadInput(current_thread_id, target_thread_id, False)
+                    except Exception:
+                        pass
+                if attached_to_foreground:
+                    try:
+                        win32process.AttachThreadInput(current_thread_id, foreground_thread_id, False)
+                    except Exception:
+                        pass
+
+            time.sleep(0.3)
+            if win32gui.GetForegroundWindow() == self._hwnd:
+                self.logger.info("Method 1 (AttachThreadInput) succeeded")
+                return True
+                
+        except Exception as e:
+            self.logger.warning(f"Method 1 (AttachThreadInput) failed: {e}")
+        
+        # === Method 2: Alt Key Trick ===
+        try:
+            self.logger.debug("Trying Alt-Key trick...")
+            # 模拟 Alt 键按下释放，欺骗 Windows 认为有用户输入
+            pyautogui.keyDown('alt')
+            time.sleep(0.02)
+            pyautogui.keyUp('alt')
+            time.sleep(0.02)
+            
+            if win32gui.IsIconic(self._hwnd):
+                win32gui.ShowWindow(self._hwnd, win32con.SW_RESTORE)
+            win32gui.SetForegroundWindow(self._hwnd)
+            
+            time.sleep(0.3)
+            if win32gui.GetForegroundWindow() == self._hwnd:
+                self.logger.info("Method 2 (Alt-Key) succeeded")
+                return True
+        except Exception as e:
+            self.logger.warning(f"Method 2 (Alt-Key) failed: {e}")
+        
+        # === Method 3: Minimize then Restore ===
+        try:
+            self.logger.debug("Trying Minimize-Restore trick...")
+            win32gui.ShowWindow(self._hwnd, win32con.SW_MINIMIZE)
+            time.sleep(0.1)
+            win32gui.ShowWindow(self._hwnd, win32con.SW_RESTORE)
+            win32gui.SetForegroundWindow(self._hwnd)
+            
+            time.sleep(0.3)
+            if win32gui.GetForegroundWindow() == self._hwnd:
+                self.logger.info("Method 3 (Minimize-Restore) succeeded")
+                return True
+        except Exception as e:
+            self.logger.warning(f"Method 3 (Minimize-Restore) failed: {e}")
+        
+        # === Method 4: Click on Window ===
+        try:
+            self.logger.debug("Trying direct click on window...")
+            self._update_window_rect()
+            if self._window_rect:
+                # 点击窗口中心
+                center_x = self._window_rect['left'] + self._window_rect['width'] // 2
+                center_y = self._window_rect['top'] + self._window_rect['height'] // 2
+                pyautogui.click(center_x, center_y)
+                time.sleep(0.3)
+                if win32gui.GetForegroundWindow() == self._hwnd:
+                    self.logger.info("Method 4 (Direct Click) succeeded")
+                    return True
+        except Exception as e:
+            self.logger.warning(f"Method 4 (Direct Click) failed: {e}")
+
+        self.logger.error("All methods to bring window to foreground failed.")
+        return False
+    
+    def start_app(self) -> bool:
+        """启动应用 (支持 Steam 游戏和普通应用)"""
+        # 1. 首先尝试查找并连接现有窗口
+        if self._find_window():
+            self._hwnd = self._find_window()
+            if self.pull_back_to_app():
+                self.logger.info("App is already running, brought to foreground.")
+                return True
+
+        # 2. 确定启动方式
+        launch_command = None
+        
+        # 优先使用 Steam 协议启动
+        if self._steam_game_id:
+            launch_command = f"start steam://rungameid/{self._steam_game_id}"
+            self.logger.info(f"Launching via Steam protocol: {launch_command}")
+        elif self._exe_path:
+            if not os.path.exists(self._exe_path):
+                self.logger.error(f"Executable not found: {self._exe_path}")
+                return False
+            launch_command = self._exe_path
+            self.logger.info(f"Launching via executable: {launch_command}")
+        else:
+            self.logger.warning("No steam_game_id or exe_path specified, cannot start app")
+            return False
+        
+        try:
+            # 3. 启动进程
+            if self._steam_game_id:
+                # Steam 协议需要使用 shell=True
+                subprocess.Popen(launch_command, shell=True)
+            else:
+                subprocess.Popen(
+                    [self._exe_path],
+                    cwd=os.path.dirname(self._exe_path),
+                    shell=True
+                )
+            self.logger.info(f"Started app process")
+            
+            # 4. 轮询等待窗口出现 (Timeout: 120s for Steam games which may need loading)
+            self.logger.info(f"Waiting for window '{self._window_title}' to appear...")
+            max_retries = 120  # Steam 游戏启动可能比较慢
+            for i in range(max_retries):
+                hwnd = self._find_window()
+                if hwnd:
+                    self._hwnd = hwnd
+                    self.logger.info(f"Window appeared after {i} seconds.")
+                    time.sleep(3)  # Steam 游戏窗口出现后稍等一下，等待完全初始化
+                    return True
+                time.sleep(1)
+            
+            self.logger.error("Timed out waiting for window to appear.")
+            return False
+            
+        except Exception as e:
+            self.logger.error(f"Failed to start app: {e}")
+            return False
+

DroidBot/platforms/windows/windows_device_state.py

@@ -0,0 +1,207 @@
+"""
+Windows Device State Implementation
+Concrete implementation of AbstractDeviceState for Windows desktop applications.
+Uses CV mode for UI element detection.
+"""
+import hashlib
+from typing import Optional, Dict, Any, List
+
+from ...core.abstract_device_state import AbstractDeviceState
+from ...core.abstract_input_event import EventType
+
+
+class WindowsDeviceState(AbstractDeviceState):
+    """
+    Windows 设备状态的具体实现（CV 模式）
+    
+    使用 OmniParser 进行 UI 元素检测，无需 UIAutomation。
+    """
+    
+    def __init__(self, device, cv_views: List[Dict[str, Any]], 
+                 window_title: str = None, tag: str = None, 
+                 screenshot_path: str = None):
+        """
+        初始化 Windows 设备状态
+        
+        :param device: WindowsDevice 实例
+        :param cv_views: CV 检测得到的 ViewDict 列表
+        :param window_title: 当前窗口标题
+        :param tag: 状态标签
+        :param screenshot_path: 截图路径
+        """
+        super().__init__(device, tag, screenshot_path)
+        
+        self._cv_views = cv_views or []
+        self._window_title = window_title
+        
+        # 生成视图字符串标识
+        self._generate_view_strs()
+        
+        # 缓存
+        self._state_str = None
+        self._structure_str = None
+    
+    # ==================== 视图信息 ====================
+    
+    @property
+    def views(self) -> List[Dict[str, Any]]:
+        """获取 CV 检测的视图列表"""
+        return self._cv_views
+    
+    @property
+    def cv_views(self) -> List[Dict[str, Any]]:
+        """获取 CV 视图列表（与 views 相同）"""
+        return self._cv_views
+    
+    # ==================== 状态标识 ====================
+
+    @property
+    def state_str(self) -> str:
+        """获取状态的唯一标识字符串"""
+        if self._state_str is None:
+            self._state_str = self._get_state_str()
+        return self._state_str
+    
+    @property
+    def structure_str(self) -> str:
+        """获取状态的结构标识（忽略内容）"""
+        if self._structure_str is None:
+            self._structure_str = self._get_content_free_state_str()
+        return self._structure_str
+    
+    @property
+    def foreground_page(self) -> Optional[str]:
+        """返回当前窗口标题作为页面标识"""
+        return self._window_title
+    
+    @property
+    def search_content(self) -> str:
+        """获取用于搜索的文本内容"""
+        texts = []
+        for view in self._cv_views:
+            text = view.get('text', '')
+            if text:
+                texts.append(text)
+        return ' '.join(texts)
+    
+    # ==================== 私有方法 ====================
+    
+    def _generate_view_strs(self):
+        """为每个视图生成唯一标识符"""
+        for idx, view in enumerate(self._cv_views):
+            if 'view_str' not in view or not view['view_str']:
+                bounds = view.get('bounds', [[0, 0], [0, 0]])
+                x1, y1 = bounds[0]
+                x2, y2 = bounds[1]
+                text = view.get('text', '')[:20]  # 截取前20个字符
+                view['view_str'] = f"win_cv_{idx}_{x1}_{y1}_{x2}_{y2}_{text}"
+    
+    def _get_state_str(self) -> str:
+        """生成状态唯一标识"""
+        state_raw = self._get_state_str_raw()
+        return hashlib.md5(state_raw.encode('utf-8')).hexdigest()
+    
+    def _get_state_str_raw(self) -> str:
+        """获取原始状态字符串"""
+        view_signatures = []
+        for view in self._cv_views:
+            sig = self._get_view_signature(view)
+            view_signatures.append(sig)
+        
+        view_signatures.sort()
+        state_str = f"window={self._window_title}&views=" + ','.join(view_signatures)
+        return state_str
+    
+    def _get_content_free_state_str(self) -> str:
+        """获取内容无关的状态标识"""
+        view_signatures = []
+        for view in self._cv_views:
+            sig = self._get_content_free_view_signature(view)
+            view_signatures.append(sig)
+        
+        view_signatures.sort()
+        state_str = f"window={self._window_title}&structure=" + ','.join(view_signatures)
+        return hashlib.md5(state_str.encode('utf-8')).hexdigest()
+    
+    @staticmethod
+    def _get_view_signature(view_dict: Dict[str, Any]) -> str:
+        """获取视图签名（包含内容）"""
+        # 如果已有签名则直接返回（缓存）
+        if 'signature' in view_dict:
+            return view_dict['signature']
+        
+        bounds = view_dict.get('bounds', [[0, 0], [0, 0]])
+        text = view_dict.get('text', '')
+        class_name = view_dict.get('class_name', 'unknown')
+        clickable = view_dict.get('clickable', False)
+        
+        signature = f"{class_name}:{bounds}:{text}:{clickable}"
+        view_dict['signature'] = signature  # 存回字典供其他地方使用
+        return signature
+    
+    @staticmethod
+    def _get_content_free_view_signature(view_dict: Dict[str, Any]) -> str:
+        """获取内容无关的视图签名"""
+        bounds = view_dict.get('bounds', [[0, 0], [0, 0]])
+        class_name = view_dict.get('class_name', 'unknown')
+        clickable = view_dict.get('clickable', False)
+        
+        # 只使用位置和类型，忽略文本内容
+        return f"{class_name}:{bounds}:{clickable}"
+    
+    # ==================== 输入事件 ====================
+    
+    def get_possible_input(self) -> List:
+        """获取当前状态可能的输入事件列表"""
+        from .windows_input_event import (
+            WindowsTouchEvent, WindowsScrollEvent, WindowsKeyEvent
+        )
+        
+        possible_events = []
+        
+        # 为每个可点击的视图生成点击事件
+        for view in self._cv_views:
+            if view.get('clickable', False) and view.get('visible', True):
+                touch_event = WindowsTouchEvent(view=view)
+                possible_events.append(touch_event)
+        
+        # 添加滚动事件
+        possible_events.append(WindowsScrollEvent(direction='up'))
+        possible_events.append(WindowsScrollEvent(direction='down'))
+        
+        # 添加常用按键事件
+        possible_events.append(WindowsKeyEvent('ESCAPE'))
+        possible_events.append(WindowsKeyEvent('ENTER'))
+        
+        return possible_events
+    
+    # ==================== 序列化 ====================
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """序列化为字典"""
+        return {
+            'tag': self.tag,
+            'window_title': self._window_title,
+            'state_str': self.state_str,
+            'structure_str': self.structure_str,
+            'foreground_page': self.foreground_page,
+            'views': self._cv_views,
+            'screenshot_path': self.screenshot_path,
+            'width': self.width,
+            'height': self.height,
+        }
+    
+    # ==================== 应用信息 ====================
+    
+    def get_app_page_depth(self) -> int:
+        """
+        获取应用页面深度
+        
+        Windows 应用通常没有明确的页面栈概念。
+        返回 0 表示应用处于正常状态（在应用内），
+        这样 MemoryGuidedPolicy 的记忆学习和卡住检测才能正常工作。
+        
+        注意：返回 -1 会导致 Memory._memorize_state() 跳过学习，
+        且 input_policy.py 中的 stuck 检测也会被跳过。
+        """
+        return 0

DroidBot/platforms/windows/windows_input_event.py

@@ -0,0 +1,200 @@
+"""
+Windows Input Event Implementations
+Concrete implementations of input events for Windows desktop applications.
+"""
+from typing import Optional, Dict, Any, List
+
+from ...core.abstract_input_event import (
+    AbstractInputEvent, EventType,
+    BaseTouchEvent, BaseLongTouchEvent, BaseSwipeEvent,
+    BaseScrollEvent, BaseSetTextEvent, BaseKeyEvent, BaseKillAppEvent
+)
+
+
+
+class WindowsTouchEvent(BaseTouchEvent):
+    """Windows 点击事件"""
+    
+    def send(self, device) -> bool:
+        """发送点击事件到 Windows 设备"""
+        x, y = self.x, self.y
+        if self.view is not None:
+            from .windows_device_state import WindowsDeviceState
+            x, y = WindowsDeviceState.get_view_center(self.view)
+        
+        # 默认左键点击
+        device.view_touch(int(x), int(y), button='left')
+        return True
+
+
+class WindowsRightClickEvent(BaseTouchEvent):
+    """Windows 右键点击事件 (新增)"""
+    
+    def send(self, device) -> bool:
+        """发送右键点击事件到 Windows 设备"""
+        x, y = self.x, self.y
+        if self.view is not None:
+            from .windows_device_state import WindowsDeviceState
+            x, y = WindowsDeviceState.get_view_center(self.view)
+        
+        device.view_touch(int(x), int(y), button='right')
+        return True
+
+
+class WindowsLongTouchEvent(BaseLongTouchEvent):
+    """Windows 长按事件"""
+    
+    def send(self, device) -> bool:
+        """发送长按事件到 Windows 设备"""
+        x, y = self.x, self.y
+        if self.view is not None:
+            from .windows_device_state import WindowsDeviceState
+            x, y = WindowsDeviceState.get_view_center(self.view)
+        device.view_long_touch(int(x), int(y), self.duration)
+        return True
+
+
+class WindowsSwipeEvent(BaseSwipeEvent):
+    """Windows 滑动/拖拽事件"""
+    
+    def send(self, device) -> bool:
+        """发送滑动事件到 Windows 设备"""
+        device.view_drag(
+            (self.start_x, self.start_y),
+            (self.end_x, self.end_y),
+            self.duration
+        )
+        return True
+
+
+class WindowsScrollEvent(BaseScrollEvent):
+    """Windows 滚动事件"""
+    
+    def send(self, device) -> bool:
+        """发送滚动事件到 Windows 设备"""
+        import pyautogui
+        
+        # 1. 确定滚动位置
+        start_x, start_y = None, None
+        
+        # 如果指定了目标 View，移动到 View 中心
+        if self.view is not None:
+            from .windows_device_state import WindowsDeviceState
+            start_x, start_y = WindowsDeviceState.get_view_center(self.view)
+            # 转换为屏幕坐标 (device.view_drag 会处理，但这里是直接调用 pyautogui)
+            # 注意：get_view_center 返回的是相对窗口坐标（如果是基于 device state 的话）
+            # 我们需要检查 view_center 的逻辑。
+            # 现在的 WindowsDeviceState.get_view_center 是基于 cv_views，bounds 是相对窗口的吗？
+            # 即使 Device 实现了 _update_window_rect，CV 识别是基于截图的。
+            # 如果截图是全屏截图(view_file 335行: path is None -> path=screen...png -> with mss -> grab monitor)，
+            # 那么 bounds 就是屏幕绝对坐标！
+            pass
+
+        if start_x is None:
+            # 默认：窗口中心
+            display_info = device.get_display_info()
+            width = display_info.get('width', 1920)
+            height = display_info.get('height', 1080)
+            target_x = display_info.get('left', 0) + width // 2
+            target_y = display_info.get('top', 0) + height // 2
+        else:
+            
+            # 必须转换为屏幕坐标
+            display_info = device.get_display_info()
+            target_x = display_info.get('left', 0) + start_x
+            target_y = display_info.get('top', 0) + start_y
+
+        # 移动到目标位置
+        pyautogui.moveTo(target_x, target_y)
+        
+        # 执行滚动
+        scroll_amount = 3  # 滚动量
+        if self.direction == self.DIRECTION_UP:
+            pyautogui.scroll(scroll_amount) # scroll up
+        elif self.direction == self.DIRECTION_DOWN:
+            pyautogui.scroll(-scroll_amount) # scroll down
+        elif self.direction == self.DIRECTION_LEFT:
+            pyautogui.hscroll(-scroll_amount)
+        elif self.direction == self.DIRECTION_RIGHT:
+            pyautogui.hscroll(scroll_amount)
+        
+        return True
+
+
+class WindowsSetTextEvent(BaseSetTextEvent):
+    """Windows 文本输入事件"""
+    
+    def send(self, device) -> bool:
+        """发送文本输入事件到 Windows 设备"""
+        # 先点击目标视图
+        if self.view is not None:
+            from .windows_device_state import WindowsDeviceState
+            x, y = WindowsDeviceState.get_view_center(self.view)
+            device.view_touch(int(x), int(y))
+            import time
+            time.sleep(0.3)
+        
+        # 输入文本
+        device.view_set_text(self.text)
+        
+        return True
+
+
+class WindowsKeyEvent(BaseKeyEvent):
+    """Windows 按键事件"""
+    
+    # Windows 按键映射 (通用名称 -> pyautogui 按键名)
+    KEY_MAP = {
+        'BACK': 'backspace',
+        'HOME': 'win',
+        'MENU': 'apps',
+        'ENTER': 'enter',
+        'ESCAPE': 'escape',
+        'TAB': 'tab',
+        'DELETE': 'delete',
+        'UP': 'up',
+        'DOWN': 'down',
+        'LEFT': 'left',
+        'RIGHT': 'right',
+        'SPACE': 'space',
+        'CTRL': 'ctrl',
+        'ALT': 'alt',
+        'SHIFT': 'shift',
+    }
+    
+    def send(self, device) -> bool:
+        """发送按键事件到 Windows 设备"""
+        key_code = self.KEY_MAP.get(self.key_name.upper(), self.key_name.lower())
+        device.key_press(key_code)
+        return True
+
+
+class WindowsKillAppEvent(BaseKillAppEvent):
+    """Windows 终止应用事件"""
+    
+    def send(self, device) -> bool:
+        """终止 Windows 应用"""
+        import subprocess
+        
+        if self.app is None:
+            return False
+        
+        # 获取进程名
+        process_name = self.app
+        if hasattr(self.app, 'get_process_name'):
+            process_name = self.app.get_process_name()
+        elif hasattr(self.app, 'exe_path'):
+            import os
+            process_name = os.path.basename(self.app.exe_path)
+        
+        try:
+            # 使用 taskkill 强制终止进程
+            subprocess.run(
+                ['taskkill', '/F', '/IM', process_name],
+                capture_output=True,
+                timeout=5
+            )
+            return True
+        except Exception as e:
+            device.logger.warning(f"Failed to kill app {process_name}: {e}")
+            return False