paper_framework.md

Papers with Keyword: framework

• Proposer-Agent-Evaluator (PAE): Autonomous Skill Discovery For Foundation Model Internet Agents

  • Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, Erran Li
  • 🏛️ Institutions: UCB, UIUC, Amazon
  • 📅 Date: December 17, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [reinforcement learning], [skill discovery], [PAE]
  • 📖 TLDR: This paper introduces the Proposer-Agent-Evaluator (PAE) system, enabling foundation model agents to autonomously discover and practice skills in real-world web environments. PAE comprises a context-aware task proposer, an agent policy for task execution, and a vision-language model-based success evaluator. Validated on vision-based web navigation tasks, PAE significantly enhances zero-shot generalization capabilities of vision-language model Internet agents, achieving over 30% relative improvement on unseen tasks and websites, and surpassing state-of-the-art open-source agents by more than 10%.

• Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining

  • Zhiqi Ge, Juncheng Li, Xinglei Pang, Minghe Gao, Kaihang Pan, Wang Lin, Hao Fei, Wenqiao Zhang, Siliang Tang, Yueting Zhuang
  • 🏛️ Institutions: Zhejiang University, NUS
  • 📅 Date: December 13, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [Information-Sensitive Cropping], [Self-Refining Dual Learning], [visual grounding], [model]
  • 📖 TLDR: This paper introduces Iris, a visual agent designed to enhance GUI automation by addressing challenges in high-resolution, complex digital environments. It employs two key innovations: Information-Sensitive Cropping (ISC), which dynamically identifies and prioritizes visually dense regions using an edge detection algorithm for efficient processing, and Self-Refining Dual Learning (SRDL), which enhances the agent's ability to handle complex tasks through a dual-learning loop that iteratively refines its performance without requiring additional annotated data. Empirical evaluations demonstrate that Iris achieves state-of-the-art performance across multiple benchmarks with only 850K GUI annotations, outperforming methods using ten times more training data.

• The BrowserGym Ecosystem for Web Agent Research

  • Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, Alexandre Lacoste
  • 🏛️ Institutions: ServiceNow Research, Mila, Polytechnique Montréal, CMU, McGill University, Tel Aviv University, Université de Montréal, iMean AI
  • 📅 Date: December 6, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [framework], [LLM], [automation], [BrowserGym], [AgentLab]
  • 📖 TLDR: This paper presents BrowserGym, an ecosystem designed to standardize the evaluation and benchmarking of web agents, particularly those leveraging Large Language Models (LLMs). It addresses the challenges posed by fragmented benchmarks and inconsistent methodologies in web agent research. BrowserGym provides a unified, gym-like environment with clearly defined observation and action spaces, enabling reproducible comparisons across various benchmarks. Additionally, AgentLab, a complementary framework, supports agent creation, testing, and analysis. The paper also features a large-scale experiment comparing the performance of 6 leading LLMs, highlighting the strengths and weaknesses of different models in real-world web tasks, while emphasizing the ongoing challenges in building efficient and robust web agents.

• ShowUI: One Vision-Language-Action Model for GUI Visual Agent

  • Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, Mike Zheng Shou
  • 🏛️ Institutions: NUS, Microsoft
  • 📅 Date: November 26, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [model], [framework], [dataset], [UI-Guided Visual Token Selection], [Interleaved Vision-Language-Action Streaming], [ShowUI]
  • 📖 TLDR: This paper introduces ShowUI, a vision-language-action model designed to enhance GUI automation by addressing challenges in UI visual perception and action modeling. It features innovations like UI-Guided Visual Token Selection to reduce computational costs and Interleaved Vision-Language-Action Streaming for effective management of visual-action history. Trained on a curated dataset, ShowUI achieves 75.1% accuracy in zero-shot screenshot grounding and demonstrates competitive performance across web, mobile, and online environments.

• AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

  • Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, Manuela Veloso
  • 🏛️ Institutions: J.P. Morgan AI Research
  • 📅 Date: November 24, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [few-shot learning], [meta-learning], [AdaptAgent]
  • 📖 TLDR: This paper introduces AdaptAgent, a framework that enables multimodal web agents to adapt to new websites and domains using few human demonstrations (up to 2). The approach enhances agents' adaptability beyond large-scale pre-training and fine-tuning by leveraging in-context learning and meta-learning techniques. Experiments on benchmarks like Mind2Web and VisualWebArena show that incorporating minimal human demonstrations boosts task success rates significantly, highlighting the effectiveness of multimodal demonstrations over text-only ones and the impact of data selection strategies during meta-learning on agent generalization.

• Improved GUI Grounding via Iterative Narrowing

  • Anthony Nguyen
  • 🏛️ Institutions: Algoma University
  • 📅 Date: November 18, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [grounding], [visual grounding], [iterative narrowing]
  • 📖 TLDR: This paper introduces a visual framework to enhance GUI grounding. By iteratively refining model predictions through progressively focused image crops, the proposed method improves the performance of both general and fine-tuned Vision-Language Models (VLMs) in GUI grounding tasks.

• The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use

  • Siyuan Hu, Mingyu Ouyang, Difei Gao, Mike Zheng Shou
  • 🏛️ Institutions: NUS
  • 📅 Date: Nov 15, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [Claude 3.5 Computer Use], [GUI automation], [planning], [action], [critic]
  • 📖 TLDR: This study evaluates Claude 3.5 Computer Use, an AI model enabling end-to-end language-to-desktop actions, through curated tasks across various domains. It introduces an out-of-the-box framework for deploying API-based GUI automation models, analyzing the model's planning, action execution, and adaptability to dynamic environments.

• Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents

  • Yu Gu, Boyuan Zheng, Boyu Gou, Kai Zhang, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, Yu Su
  • 🏛️ Institutions: OSU, Orby AI
  • 📅 Date: November 10, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [WebDreamer], [model-based planning], [world model]
  • 📖 TLDR: This paper investigates whether Large Language Models (LLMs) can function as world models within web environments, enabling model-based planning for web agents. Introducing WebDreamer, a framework that leverages LLMs to simulate potential action sequences in web environments, the study demonstrates significant performance improvements over reactive baselines on benchmarks like VisualWebArena and Mind2Web-live. The findings suggest that LLMs possess the capability to model the dynamic nature of the internet, paving the way for advancements in automated web interaction and opening new research avenues in optimizing LLMs for complex, evolving environments.

• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning

  • Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Xinyue Yang, Jiadai Sun, Yu Yang, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, Yuxiao Dong
  • 🏛️ Institutions: Tsinghua University, BAAI
  • 📅 Date: November 4, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [reinforcement learning], [self-evolving curriculum], [WebRL], [outcome-supervised reward model]
  • 📖 TLDR: This paper introduces WebRL, a self-evolving online curriculum reinforcement learning framework designed to train high-performance web agents using open large language models (LLMs). WebRL addresses challenges such as the scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning. It incorporates a self-evolving curriculum that generates new tasks from unsuccessful attempts, a robust outcome-supervised reward model (ORM), and adaptive reinforcement learning strategies to ensure consistent improvements. Applied to Llama-3.1 and GLM-4 models, WebRL significantly enhances their performance on web-based tasks, surpassing existing state-of-the-art web agents.

• AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents

  • Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, Yuxiao Dong
  • 🏛️ Institutions: Tsinghua University, Peking University
  • 📅 Date: October 31, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [dataset], [benchmark], [AndroidLab]
  • 📖 TLDR: This paper introduces AndroidLab, a comprehensive framework for training and systematically benchmarking Android autonomous agents. It provides an operational environment with diverse modalities and action spaces, supporting both large language models (LLMs) and multimodal models (LMMs). The benchmark includes 138 tasks across nine apps on predefined Android virtual devices. Utilizing AndroidLab, the authors developed an Android Instruction dataset and trained six open-source LLMs and LMMs, significantly improving their average success rates.

• From Context to Action: Analysis of the Impact of State Representation and Context on the Generalization of Multi-Turn Web Navigation Agents

  • Nalin Tiwary, Vardhan Dongre, Sanil Arun Chawla, Ashwin Lamani, Dilek Hakkani-Tür
  • 🏛️ Institutions: UIUC
  • 📅 Date: October 31, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [context management], [generalization], [multi-turn navigation], [CWA]
  • 📖 TLDR: This study examines how different contextual elements affect the performance and generalization of Conversational Web Agents (CWAs) in multi-turn web navigation tasks. By optimizing context management—specifically interaction history and web page representation—the research demonstrates enhanced agent performance across various out-of-distribution scenarios, including unseen websites, categories, and geographic locations.

• Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents

  • Jaekyeom Kim, Dong-Ki Kim, Lajanugen Logeswaran, Sungryull Sohn, Honglak Lee
  • 🏛️ Institutions: LG AI Research, Field AI, University of Michigan
  • 📅 Date: October 29, 2024
  • 📑 Publisher: EMNLP 2024 (Findings)
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [Auto-Intent]
  • 📖 TLDR: The paper presents Auto-Intent, a method to adapt pre-trained large language models for web navigation tasks without direct fine-tuning. It discovers underlying intents from domain demonstrations and trains an intent predictor to enhance decision-making. Auto-Intent improves the performance of GPT-3.5, GPT-4, and Llama-3.1 agents on benchmarks like Mind2Web and WebArena.

• EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

  • Xuetian Chen, Hangcheng Li, Jiaqing Liang, Sihang Jiang, Deqing Yang
  • 🏛️ Institutions: Fudan University
  • 📅 Date: October 25, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [dataset], [framework], [synthetic data]
  • 📖 TLDR: The EDGE framework proposes an innovative approach to improve GUI understanding and interaction capabilities in vision-language models through large-scale, multi-granularity synthetic data generation. By leveraging webpage data, EDGE minimizes the need for manual annotations and enhances the adaptability of models across desktop and mobile GUI environments. Evaluations show its effectiveness in diverse GUI-related tasks, contributing significantly to autonomous agent development in GUI navigation and interaction.

• OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization

  • Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, Dong Yu
  • 🏛️ Institutions: Zhejiang University, Tencent AI Lab, Westlake University
  • 📅 Date: October 25, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [learning], [imitation learning], [exploration], [AI feedback]
  • 📖 TLDR: The paper presents OpenWebVoyager, an open-source framework for training web agents that explore real-world online environments autonomously. The framework employs a cycle of exploration, feedback, and optimization, enhancing agent capabilities through multimodal perception and iterative learning. Initial skills are acquired through imitation learning, followed by real-world exploration, where the agent’s performance is evaluated and refined through feedback loops.

• AutoGLM: Autonomous Foundation Agents for GUIs

  • Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, Junjie Gao, Junjun Shan, Kangning Liu, Shudan Zhang, Shuntian Yao, Siyi Cheng, Wentao Yao, Wenyi Zhao, Xinghan Liu, Xinyi Liu, Xinying Chen, Xinyue Yang, Yang Yang, Yifan Xu, Yu Yang, Yujia Wang, Yulin Xu, Zehan Qi, Yuxiao Dong, Jie Tang
  • 🏛️ Institutions: Zhipu AI, Tsinghua University
  • 📅 Date: October 25, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [model], [learning], [AutoGLM]
  • 📖 TLDR: This paper introduces AutoGLM, a new series in the ChatGLM family, designed as foundation agents for autonomous control of digital devices through GUIs. It addresses the challenges foundation models face in decision-making within dynamic environments by developing agents capable of learning through autonomous interactions. Focusing on web browsers and Android devices, AutoGLM integrates various techniques to create deployable agent systems. Key insights include the importance of designing an appropriate "intermediate interface" for GUI control and a novel progressive training framework for self-evolving online curriculum reinforcement learning. Evaluations demonstrate AutoGLM's effectiveness across multiple domains, achieving notable success rates in web browsing and Android device control tasks.

• AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant

  • Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, Zhiyong Wu
  • 🏛️ Institutions: XJTU, Shanghai AI Lab, HKU
  • 📅 Date: October 24, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [multi-agent systems], [specialized generalist agent], [OSWorld benchmark]
  • 📖 TLDR: AgentStore introduces a scalable platform to integrate and manage heterogeneous agents, designed to enhance generalist assistant capabilities for diverse computer tasks. Using a MetaAgent and AgentToken strategy, AgentStore shows improved generalization on the OSWorld benchmark.

• Lightweight Neural App Control

  • Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, Kun Shao
  • 🏛️ Institutions: Huawei Noah's Ark Lab, UCL
  • 📅 Date: October 23, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [vision language model], [Action Transformer], [app agent], [Android control], [multi-modal]
  • 📖 TLDR: This paper introduces LiMAC, a mobile control framework for Android that integrates an Action Transformer and fine-tuned vision-language models to execute precise actions in mobile apps. Tested on open-source datasets, LiMAC improves action accuracy by up to 42% over traditional prompt engineering baselines, demonstrating enhanced efficiency and accuracy in mobile app control tasks.

• Large Language Models Empowered Personalized Web Agents

  • Hongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu, Xiaoyu Shen, Wenjie Li, Tat-Seng Chua
  • 🏛️ Institutions: HK PolyU, NTU Singapore
  • 📅 Date: Oct 22, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [benchmark], [personalized web agent], [user behavior alignment], [memory-enhanced alignment]
  • 📖 TLDR: This paper proposes a novel framework, Personalized User Memory-enhanced Alignment (PUMA), enabling large language models to serve as personalized web agents by incorporating user-specific data and historical web interactions. The authors also introduce a benchmark, PersonalWAB, to evaluate these agents on various personalized web tasks. Results show that PUMA improves web agent performance by optimizing action execution based on user-specific preferences.

• SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation

  • Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, Kun Shao
  • 🏛️ Institutions: Huawei Noah’s Ark Lab, Harbin Institute of Technology, Shenzhen, UCL
  • 📅 Date: October 19, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [AI agent], [smartphone control], [framework]
  • 📖 TLDR: SPA-Bench is introduced as a benchmark designed to evaluate multimodal large language model (MLLM)-based smartphone agents, offering a task set that spans common smartphone functionalities across system and third-party applications. It includes a plug-and-play framework for real-time agent interactions on Android, integrating over ten agents with an adaptable evaluation pipeline measuring success across diverse metrics. Through this, the benchmark exposes challenges such as UI interpretation, action grounding, and memory retention in mobile environments, advancing research in smartphone-based agent applications.

• AutoWebGLM: A Large Language Model-based Web Navigating Agent

  • Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, Jie Tang
  • 🏛️ Institutions: THU, OSU
  • 📅 Date: October 12, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [dataset], [benchmark], [reinforcement learning]
  • 📖 TLDR: AutoWebGLM introduces a web navigation agent based on ChatGLM3-6B, designed to autonomously navigate and interact with webpages for complex tasks. The paper highlights a two-phase data construction approach using a hybrid human-AI methodology for diverse, curriculum-based web task training. It also presents AutoWebBench, a benchmark for evaluating agent performance in web tasks, and uses reinforcement learning to fine-tune operations, addressing complex webpage interaction and grounding.

• Agent S: An Open Agentic Framework that Uses Computers Like a Human

  • Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang
  • 🏛️ Institutions: Simular Research
  • 📅 Date: October 10, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [autonomous GUI interaction], [experience-augmented hierarchical planning]
  • 📖 TLDR: This paper introduces Agent S, an open agentic framework that enables autonomous interaction with computers through a Graphical User Interface (GUI). The system addresses key challenges in automating computer tasks through experience-augmented hierarchical planning and an Agent-Computer Interface (ACI). Agent S demonstrates significant improvements over baselines on the OSWorld benchmark, achieving a 20.58% success rate (83.6% relative improvement). The framework shows generalizability across different operating systems and provides insights for developing more effective GUI agents.

• ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents

  • Jakub Hoscilowicz, Bartosz Maj, Bartosz Kozakiewicz, Oleksii Tymoschuk, Artur Janicki
  • 🏛️ Institutions: Samsung R&D Poland, Warsaw University of Technology
  • 📅 Date: October 9, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [model], [SeeClick], [AITW benchmark]
  • 📖 TLDR: The paper introduces ClickAgent, a framework that enhances autonomous agents' interaction with mobile UIs by improving their ability to locate interface elements accurately. This is achieved through a dual-component system where an MLLM performs reasoning and action planning, while a dedicated UI location model (e.g., SeeClick) handles element identification. ClickAgent, evaluated on the AITW benchmark and tested on both emulators and real Android devices, surpasses other agents like CogAgent and AppAgent in task success rate, advancing automation reliability on mobile platforms.

• TinyClick: Single-Turn Agent for Empowering GUI Automation

  • Pawel Pawlowski, Krystian Zawistowski, Wojciech Lapacz, Marcin Skorupa, Adam Wiacek, Sebastien Postansque, Jakub Hoscilowicz
  • 🏛️ Institutions: Samsung R&D Poland, Warsaw University of Technology
  • 📅 Date: October 9, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [vision language model], [Screenspot], [OmniAct]
  • 📖 TLDR: TinyClick is a compact, single-turn agent designed to automate GUI tasks by precisely locating screen elements via the Vision-Language Model Florence-2-Base. Trained with multi-task strategies and MLLM-based data augmentation, TinyClick achieves high accuracy on Screenspot and OmniAct, outperforming specialized GUI interaction models and general MLLMs like GPT-4V. The model's lightweight design (0.27B parameters) ensures fast processing and minimal latency, making it efficient for real-world applications on multiple platforms.

• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

  • Boyu Gou, Ruochen Wang, Boyuan Zheng, Yucheng Xie, Cheng Chang, Yiheng Shu, Haotian Sun, Yu Su
  • 🏛️ Institutions: OSU, Orby AI
  • 📅 Date: October 7, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [visual grounding], [GUI agents], [cross-platform generalization], [UGround], [SeeAct-V], [synthetic data]
  • 📖 TLDR: This paper introduces UGround, a universal visual grounding model for GUI agents that enables human-like navigation of digital interfaces. The authors advocate for GUI agents with human-like embodiment that perceive the environment entirely visually and take pixel-level actions. UGround is trained on a large-scale synthetic dataset of 10M GUI elements across 1.3M screenshots. Evaluated on six benchmarks spanning grounding, offline, and online agent tasks, UGround significantly outperforms existing visual grounding models by up to 20% absolute. Agents using UGround achieve comparable or better performance than state-of-the-art agents that rely on additional textual input, demonstrating the feasibility of vision-only GUI agents.

• ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning

  • Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, Zhou Yu
  • 🏛️ Institutions: Columbia Univ., MSR
  • 📅 Date: Oct 2, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [learning], [R-MCTS], [Exploratory Learning], [VisualWebArena]
  • 📖 TLDR: This paper introduces ExACT, an approach that combines Reflective Monte Carlo Tree Search (R-MCTS) and Exploratory Learning to enhance AI agents' exploration and decision-making capabilities in complex web environments. R-MCTS incorporates contrastive reflection and multi-agent debate for improved search efficiency and reliable state evaluation. Evaluated on the VisualWebArena benchmark, the GPT-4o-based R-MCTS agent demonstrates significant performance improvements over previous state-of-the-art methods. Additionally, knowledge gained from test-time search is effectively transferred back to GPT-4o through fine-tuning, enabling the model to explore, evaluate, and backtrack without external search algorithms, achieving 87% of R-MCTS's performance with reduced computational resources.

• Dynamic Planning for LLM-based Graphical User Interface Automation

  • Shaoqing Zhang, Zhuosheng Zhang, Kehai Chen, Xinbei Ma, Muyun Yang, Tiejun Zhao, Min Zhang
  • 🏛️ Institutions: SJTU
  • 📅 Date: October 1, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [dynamic planning]
  • 📖 TLDR: This paper introduces a novel method called Dynamic Planning of Thoughts (D-PoT) aimed at enhancing LLM-based agents for GUI tasks. It addresses the challenges of task execution by dynamically adjusting planning based on environmental feedback and action history, outperforming existing methods such as ReAct by improving accuracy significantly in navigating GUI environments. The study emphasizes the importance of integrating execution history and contextual cues to optimize decision-making processes for autonomous agents.

• Turn Every Application into an Agent: Towards Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents

  • Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
  • 🏛️ Institutions: Peking University, Microsoft
  • 📅 Date: September 26, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [API interaction], [HACI], [Agent OS]
  • 📖 TLDR: This paper proposes an API-centered framework called AXIS, enhancing the efficiency and reliability of LLM-based agents by prioritizing API interactions over UI-based actions. This approach aims to reduce the high latency and error rates of traditional UI-interaction models. AXIS not only supports the rapid creation and extension of APIs through automated application exploration but also contributes to a new Human-Agent-Computer Interaction (HACI) framework. The paper outlines the development of an agent-centric operating system (Agent OS), which improves task completion times by up to 70% and reduces cognitive load on users while maintaining high accuracy across complex multi-application tasks.

• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

  • Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, Zack Hui
  • 🏛️ Institutions: Microsoft
  • 📅 Date: September 13, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [benchmark], [Navi]
  • 📖 TLDR: This paper introduces the Windows Agent Arena (WAA), a scalable platform for testing and benchmarking multi-modal AI agents within a realistic Windows OS environment. WAA enables researchers to evaluate agentic workflows across diverse tasks and supports large-scale deployment using Azure ML. The study also presents Navi, a multi-modal agent achieving a 19.5% success rate on Windows tasks, highlighting the platform's potential for advancing AI agent development.

• Agent Workflow Memory

  • Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, Graham Neubig
  • 🏛️ Institutions: CMU, MIT
  • 📅 Date: September 11, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [memory], [AWM]
  • 📖 TLDR: The paper proposes Agent Workflow Memory (AWM), a method enabling language model-based agents to induce and utilize reusable workflows from past experiences to guide future actions in web navigation tasks. AWM operates in both offline and online settings, significantly improving performance on benchmarks like Mind2Web and WebArena, and demonstrating robust generalization across tasks, websites, and domains.

• TinyAgent: Function Calling at the Edge

  • Lutfi Eren Erdogan, Nicholas Lee, Siddharth Jha, Sehoon Kim, Ryan Tabrizi, Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, Amir Gholami
  • 🏛️ Institutions: UC Berkeley, ICSI
  • 📅 Date: September 1, 2024
  • 📑 Publisher: EMNLP 2024
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [dataset], [quantization], [LLMCompiler], [TinyAgent-1.1B], [TinyAgent-7B]
  • 📖 TLDR: This paper introduces TinyAgent, an end-to-end framework for training and deploying task-specific small language model agents capable of function calling at the edge. By fine-tuning small models with curated datasets and employing techniques like quantization and a novel tool retrieval method, TinyAgent enables efficient, real-time execution of user commands on local devices without relying on cloud infrastructure. The framework demonstrates that these small models can match or even surpass the function-calling capabilities of larger models like GPT-4-Turbo while operating entirely on edge devices.

• WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration

  • Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, Volker Tresp
  • 🏛️ Institutions: LMU Munich, Technical University of Munich, Munich Center for Machine Learning (MCML)
  • 📅 Date: August 28, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [Monte Carlo Tree Search], [reinforcement learning], [WebPilot]
  • 📖 TLDR: This paper introduces WebPilot, a multi-agent system designed to execute complex web tasks requiring dynamic interaction. By employing a dual optimization strategy grounded in Monte Carlo Tree Search (MCTS), WebPilot enhances adaptability in complex web environments. The system's Global Optimization phase generates high-level plans by decomposing tasks into manageable subtasks, while the Local Optimization phase executes each subtask using a tailored MCTS approach. Experimental results on WebArena and MiniWoB++ demonstrate WebPilot's effectiveness, achieving state-of-the-art performance with GPT-4 and marking a significant advancement in autonomous web agent capabilities. :contentReference[oaicite:0]{index=0}

• Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents

  • Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
  • 🏛️ Institutions: MultiOn, Stanford
  • 📅 Date: August 13, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [MCTS], [Tree Search], [DPO], [Reinforcement Learning]
  • 📖 TLDR: TBD

• AppAgent v2: Advanced Agent for Flexible Mobile Interactions

  • Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, Yunchao Wei
  • 🏛️ Institutions: University of Technology Sydney, Tencent, Beijing Jiaotong University, Westlake University
  • 📅 Date: August 5, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [AppAgent v2]
  • 📖 TLDR: This work presents AppAgent v2, a novel LLM-based multimodal agent framework for mobile devices capable of navigating applications by emulating human-like interactions such as tapping and swiping. The agent constructs a flexible action space that enhances adaptability across various applications, including parsing text and vision descriptions. It operates through two main phases: exploration and deployment, utilizing retrieval-augmented generation (RAG) technology to efficiently retrieve and update information from a knowledge base, thereby empowering the agent to perform tasks effectively and accurately.

• CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

  • Xinbei Ma, Zhuosheng Zhang, Hai Zhao
  • 🏛️ Institutions: SJTU
  • 📅 Date: August 2024
  • 📑 Publisher: ACL 2024
  • 💻 Env: [Mobile]
  • 🔑 Key: [model], [framework], [benchmark]
  • 📖 TLDR: This paper presents CoCo-Agent, a multimodal large language model (MLLM) designed for smartphone GUI automation. It introduces two novel approaches: Comprehensive Environment Perception (CEP) for enhanced GUI understanding, and Conditional Action Prediction (CAP) to improve action response accuracy. The proposed agent achieves state-of-the-art performance on GUI automation benchmarks such as AITW and META-GUI, showcasing its capabilities in realistic scenarios.

• OmniParser for Pure Vision Based GUI Agent

  • Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah
  • 🏛️ Institutions: MSR, Microsoft Gen AI
  • 📅 Date: August 1, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [dataset], [OmniParser]
  • 📖 TLDR: This paper introduces OmniParser, a method for parsing user interface screenshots into structured elements, enhancing the ability of models like GPT-4V to generate actions accurately grounded in corresponding UI regions. The authors curated datasets for interactable icon detection and icon description, fine-tuning models to parse interactable regions and extract functional semantics of UI elements.

• Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems

  • Aditya Vempaty, [Other authors not provided in the search results]
  • 🏛️ Institutions: Emergence AI
  • 📅 Date: July 17, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [autonomous web navigation], [hierarchical architecture], [DOM distillation]
  • 📖 TLDR: This paper presents Agent-E, a novel web agent that introduces several architectural improvements over previous state-of-the-art systems. Key features include a hierarchical architecture, flexible DOM distillation and denoising methods, and a "change observation" concept for improved performance. Agent-E outperforms existing text and multi-modal web agents by 10-30% on the WebVoyager benchmark. The authors synthesize their findings into general design principles for developing agentic systems, including the use of domain-specific primitive skills, hierarchical architectures, and agentic self-improvement.

• AUITestAgent: Automatic Requirements Oriented GUI Function Testing

  • Yongxiang Hu, Xuan Wang, Yingchuan Wang, Yu Zhang, Shiyu Guo, Chaoyi Chen, Xin Wang, Yangfan Zhou
  • 🏛️ Institutions: Fudan University, Meituan
  • 📅 Date: July 12, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [GUI testing], [AUITestAgent]
  • 📖 TLDR: This paper presents AUITestAgent, the first automatic, natural language-driven GUI testing tool for mobile apps. It automates the entire process of GUI interaction and function verification by extracting GUI interactions from test requirements via dynamically organized agents and employing a multi-dimensional data extraction strategy for verification.

• Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence

  • Weize Chen, Ziming You, Ran Li, Yitong Guan, Chen Qian, Chenyang Zhao, Cheng Yang, Ruobing Xie, Zhiyuan Liu, Maosong Sun
  • 🏛️ Institutions: Tsinghua University, Peking University, BUPT, Tencent
  • 📅 Date: July 7, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Misc]
  • 🔑 Key: [framework], [IoA]
  • 📖 TLDR: The paper proposes the Internet of Agents (IoA), a framework inspired by the Internet to facilitate collaboration among diverse autonomous agents. IoA introduces an agent integration protocol, dynamic teaming mechanisms, and conversation flow control, enabling flexible and scalable multi-agent collaboration. Experiments demonstrate IoA's superior performance across various tasks, highlighting its effectiveness in integrating heterogeneous agents.

• MobileFlow: A Multimodal LLM For Mobile GUI Agent

  • Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, Wenhao Xu
  • 🏛️ Institutions: Ant Group
  • 📅 Date: July 5, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [model], [framework], [MobileFlow]
  • 📖 TLDR: This paper introduces MobileFlow, a multimodal large language model tailored for mobile GUI agents. With approximately 21 billion parameters and hybrid visual encoders, it supports variable image resolutions and multilingual GUIs, enhancing the model's ability to interpret image data and comprehend user instructions for GUI interaction tasks.

• MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices

  • Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, Jianping Fan
  • 🏛️ Institutions: HKUST, Ant Group
  • 📅 Date: July 4, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [tool formulation], [multi-agent collaboration], [MobileExperts]
  • 📖 TLDR: This paper introduces MobileExperts, a framework that enhances autonomous operations on mobile devices by dynamically assembling agent teams based on user requirements. Each agent independently explores and formulates tools to evolve into an expert, improving efficiency and reducing reasoning costs.

• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents

  • Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Philip Torr, Bernard Ghanem, Guohao Li
  • 🏛️ Institutions: KAUST, UTokyo, CMU, Stanford, Harvard, Tsinghua University, SUSTech, Oxford
  • 📅 Date: July 3, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [benchmark], [framework], [evaluation], [CRAB]
  • 📖 TLDR: The authors present CRAB, a benchmark framework designed to evaluate Multimodal Language Model agents across multiple environments. It features a graph-based fine-grained evaluation method and supports automatic task generation, addressing limitations in existing benchmarks.

• Vision-driven Automated Mobile GUI Testing via Multimodal Large Language Model

  • Zhe Liu, Cheng Li, Chunyang Chen, Junjie Wang, Boyu Wu, Yawen Wang, Jun Hu, Qing Wang
  • 🏛️ Institutions: Institute of Software, Chinese Academy of Sciences, Monash University, Beijing Institute of Technology, University of Chinese Academy of Sciences
  • 📅 Date: July 3, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [VisionDroid]
  • 📖 TLDR: The paper presents VisionDroid, a vision-driven automated GUI testing approach utilizing Multimodal Large Language Models (MLLM) to detect non-crash functional bugs in mobile applications. By extracting GUI text information and aligning it with screenshots, VisionDroid enables MLLM to understand GUI context, facilitating deeper and function-oriented exploration. The approach segments exploration history into logically cohesive parts, prompting MLLM for bug detection, demonstrating superior performance over existing methods.

• Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding

  • Yue Fan, Lei Ding, Ching-Chen Kuo, Shan Jiang, Yang Zhao, Xinze Guan, Jie Yang, Yi Zhang, Xin Eric Wang
  • 🏛️ Institutions: UCSC, MSR
  • 📅 Date: June 27, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [dataset], [ToL], [screen reading], [accessibility]
  • 📖 TLDR: The authors propose the Tree-of-Lens (ToL) agent to address the Screen Point-and-Read (ScreenPR) task, which involves generating natural language descriptions of screen regions based on user-indicated points. The ToL agent constructs a Hierarchical Layout Tree to comprehend the content and articulate the layout and spatial relationships between elements. The authors also introduce the ScreenPR benchmark, consisting of 650 screenshots from web, mobile, and operating system GUIs, manually annotated with 1,500 target points and regions.

• Octo-planner: On-device Language Model for Planner-Action Agents

  • Nexa AI Team
  • 🏛️ Institutions: Nexa AI
  • 📅 Date: June 26, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Misc]
  • 🔑 Key: [model], [framework], [Octo-planner], [on-device], [planning]
  • 📖 TLDR: This paper presents Octo-planner, an on-device planning model designed for the Planner-Action Agents Framework. Octo-planner utilizes a fine-tuned model based on Phi-3 Mini (3.8 billion parameters) for high efficiency and low power consumption. It separates planning and action execution into two distinct components: a planner agent optimized for edge devices and an action agent using the Octopus model for function execution. The model achieves a planning success rate of 98.1% on benchmark datasets, providing reliable and effective performance.

• VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning

  • Ziyang Meng, Yu Dai, Zezheng Gong, Shaoxiong Guo, Minglong Tang, Tongquan Wei
  • 🏛️ Institutions: SJTU
  • 📅 Date: June 20, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [model], [dataset], [framework], [VGA], [hallucination]
  • 📖 TLDR: This paper introduces VGA, a fine-tuned model designed to enhance GUI comprehension by reducing hallucinations. The authors constructed a Vision Question Answering (VQA) dataset of 63.8k high-quality examples using a Referent Method, ensuring model responses are highly dependent on visual content. They also propose a two-stage fine-tuning method called Foundation and Advanced Comprehension (FAC) to improve the model's ability to extract information from images and align with human intent.

• VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought

  • Gabriel Sarch, Lawrence Jang, Michael J. Tarr, William W. Cohen, Kenneth Marino, Katerina Fragkiadaki
  • 🏛️ Institutions: CMU, Google DeepMind
  • 📅 Date: June 20, 2024
  • 📑 Publisher: NeurIPS 2024
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [memory], [in-context learning], [ICAL]
  • 📖 TLDR: This paper introduces In-Context Abstraction Learning (ICAL), a method enabling Vision-Language Models (VLMs) to generate their own examples from sub-optimal demonstrations and human feedback. By abstracting trajectories into generalized programs of thought, ICAL enhances decision-making in retrieval-augmented LLM and VLM agents, reducing reliance on manual prompt engineering and improving performance across various tasks.

• GUI Action Narrator: Where and When Did That Action Take Place?

  • Qinchen Wu, Difei Gao, Kevin Qinghong Lin, Zhuoyu Wu, Xiangwu Guo, Peiran Li, Weichen Zhang, Hengxu Wang, Mike Zheng Shou
  • 🏛️ Institutions: NUS, Chinese Academy of Sciences
  • 📅 Date: June 19, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [dataset], [framework], [Act2Cap], [GUI Narrator]
  • 📖 TLDR: The authors present Act2Cap, a GUI action dataset containing 4,189 video-caption pairs depicting various GUI actions such as clicks, drags, and typing across multiple software environments. They also propose GUI Narrator, a framework that leverages cursor detection as a visual prompt to enhance the interpretation of high-resolution screenshots for GUI video captioning. Evaluations reveal that even advanced multimodal models face challenges in this domain, highlighting the need for specialized approaches to improve performance.

• WebCanvas: Benchmarking Web Agents in Online Environments

  • Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, Zhengyang Wu
  • 🏛️ Institutions: iMean AI, CMU
  • 📅 Date: June 18, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [dataset], [benchmark], [Mind2Web-Live], [key-node evaluation]
  • 📖 TLDR: This paper presents WebCanvas, an online evaluation framework for web agents designed to address the dynamic nature of web interactions. It introduces a key-node-based evaluation metric to capture critical actions or states necessary for task completion while disregarding noise from insignificant events or changed web elements. The framework includes the Mind2Web-Live dataset, a refined version of the original Mind2Web static dataset, containing 542 tasks with 2,439 intermediate evaluation states. Despite advancements, the best-performing model achieves a task success rate of 23.1%, highlighting substantial room for improvement.

• GUICourse: From General Vision Language Models to Versatile GUI Agents

  • Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, Maosong Sun
  • 🏛️ Institutions: Tsinghua University, Rhapsody AI, University of Electronic Science and Technology of China
  • 📅 Date: June 17, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [dataset], [framework], [GUICourse]
  • 📖 TLDR: This paper introduces GUICourse, a suite of datasets aimed at training visual-based GUI agents from general vision-language models. It addresses challenges in OCR, grounding, and GUI knowledge, enhancing the models' capabilities in GUI navigation tasks.

• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

  • Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar
  • 🏛️ Institutions: UC Berkeley, UIUC, Google DeepMind
  • 📅 Date: June 14, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [reinforcement learning], [DigiRL]
  • 📖 TLDR: The authors present DigiRL, an autonomous reinforcement learning approach for training device-control agents. By fine-tuning a pre-trained vision-language model in two stages—offline and offline-to-online RL—DigiRL achieves a significant improvement in success rates on the Android-in-the-Wild dataset, establishing a new state-of-the-art for digital agents in device control.

• Practical, Automated Scenario-based Mobile App Testing

  • Shengcheng Yu, Chunrong Fang, Mingzhe Du, Zimin Ding, Zhenyu Chen, Zhendong Su
  • 🏛️ Institutions: Nanjing University, ETH Zurich
  • 📅 Date: June 12, 2024
  • 📑 Publisher: IEEE Transactions on Software Engineering
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [ScenTest], [event knowledge graph], [GUI image understanding]
  • 📖 TLDR: This paper introduces ScenTest, a novel approach for scenario-based mobile app testing that integrates event knowledge graphs (EKGs) with GUI image understanding. By extracting entities and relationships from crowdsourced test reports, ScenTest constructs EKGs for specific scenarios, guiding automated testing processes. This method bridges the gap between testing execution and app business logic, achieving fully automated testing on target scenarios for the first time.

• Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

  • Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, Jitao Sang
  • 🏛️ Institutions: Alibaba Group, Beijing University of Posts and Telecommunications
  • 📅 Date: June 3, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [multi-agent], [planning], [decision-making], [reflection]
  • 📖 TLDR: The paper presents Mobile-Agent-v2, a multi-agent architecture designed to assist with mobile device operations. It comprises three agents: a planning agent that generates task progress, a decision agent that navigates tasks using a memory unit, and a reflection agent that corrects erroneous operations. This collaborative approach addresses challenges in navigation and long-context input scenarios, achieving over a 30% improvement in task completion compared to single-agent architectures.

• WebSuite: Systematically Evaluating Why Web Agents Fail

  • Eric Li, Jim Waldo
  • 🏛️ Institutions: Harvard
  • 📅 Date: June 1, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [framework], [failure analysis], [analysis], [task disaggregation]
  • 📖 TLDR: This paper introduces WebSuite, a diagnostic benchmark to investigate the causes of web agent failures. By categorizing agent tasks using a taxonomy of operational, informational, and navigational actions, WebSuite offers granular insights into the specific actions where agents struggle, like filtering or form completion. It enables detailed comparison across agents, identifying areas for architectural and UX adaptation to improve agent reliability and task success on the web.

• Visual Grounding for User Interfaces

  • Yijun Qian, Yujie Lu, Alexander Hauptmann, Oriana Riva
  • 🏛️ Institutions: CMU, UCSB
  • 📅 Date: June 2024
  • 📑 Publisher: NAACL 2024
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [visual grounding], [UI element localization], [LVG]
  • 📖 TLDR: This work introduces the task of visual UI grounding, which unifies detection and grounding by enabling models to identify UI elements referenced by natural language commands solely from visual input. The authors propose LVG, a model that outperforms baselines pre-trained on larger datasets by over 4.9 points in top-1 accuracy, demonstrating its effectiveness in localizing referenced UI elements without relying on UI metadata.

• Unveiling Disparities in Web Task Handling Between Human and Web Agent

  • Kihoon Son, Jinhyeon Kwon, DaEun Choi, Tae Soo Kim, Young-Ho Kim, Sangdoo Yun, Juho Kim
  • 🏛️ Institutions: KAIST, Seoul National University
  • 📅 Date: May 7, 2024
  • 📑 Publisher: CHI 2024 Workshop
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [cognitive comparison], [task analysis]
  • 📖 TLDR: This paper examines how humans and web agents differ in handling web-based tasks, focusing on key aspects such as planning, action-taking, and reflection. Using a think-aloud protocol, the study highlights the cognitive processes humans employ, like exploration and adjustment, versus the more rigid task execution patterns observed in web agents. The authors identify several limitations in current web agents, proposing the need for improved frameworks to enhance adaptability and knowledge update mechanisms in agent-based systems.

• Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning

  • Lucas-Andreï Thil, Mirela Popa, Gerasimos Spanakis
  • 🏛️ Institutions: Maastricht University the Netherlands
  • 📅 Date: May 1, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [large language models], [reinforcement learning]
  • 📖 TLDR: This paper proposes a novel approach combining supervised learning (SL) and reinforcement learning (RL) techniques to train web navigation agents using large language models. The authors address limitations in previous models' understanding of HTML content and introduce methods to enhance true comprehension. Their approach, evaluated on the MiniWoB benchmark, outperforms previous SL methods on certain tasks using less data and narrows the performance gap with RL models. The study achieves 43.58% average accuracy in SL and 36.69% when combined with a multimodal RL approach, setting a new direction for future web navigation research.

• Search Beyond Queries: Training Smaller Language Models for Web Interactions via Reinforcement Learning

  • Moghis Fereidouni, A.B. Siddique
  • 🏛️ Institutions: University of Kentucky
  • 📅 Date: April 16, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [reinforcement learning], [grounded language agent], [Flan-T5], [unsupervised domain adaptation]
  • 📖 TLDR: This paper introduces GLAINTEL, a grounded language agent framework designed to enhance web interaction using instruction-finetuned language models, particularly Flan-T5, with reinforcement learning (PPO) to tackle interactive web navigation challenges. The study explores unsupervised and supervised training methods, evaluating the effects of human demonstration on agent performance. Results indicate that combining human feedback with reinforcement learning yields effective outcomes, rivaling larger models like GPT-4 on web navigation tasks.

• MMInA: Benchmarking Multihop Multimodal Internet Agents

  • Ziniu Zhang, Shulin Tian, Liangyu Chen, Ziwei Liu
  • 🏛️ Institutions: NTU
  • 📅 Date: April 15, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [framework], [multihop web browsing], [multimodal tasks], [long-range reasoning]
  • 📖 TLDR: The MMInA benchmark is designed to evaluate agents' capacity to complete complex, multihop web tasks by navigating and extracting information across evolving real-world websites. Composed of 1,050 tasks across diverse domains, MMInA challenges agents with realistic, multimodal information retrieval and reasoning tasks, such as comparative shopping and travel inquiries. Despite recent advances, agents show difficulties in handling tasks requiring sequential steps across multiple sites, underscoring the need for enhanced multimodal and memory-augmented models.

• LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Automation Task Evaluation

  • Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, Mengwei Xu
  • 🏛️ Institutions: BUPT, Tsinghua University
  • 📅 Date: April 12, 2024
  • 📑 Publisher: UIST 2024
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [dataset], [benchmark], [UI automation], [mobile agent evaluation]
  • 📖 TLDR: LlamaTouch is an evaluation testbed designed for mobile UI automation, enabling reliable task assessment across 495 annotated tasks. It provides a scalable solution to evaluate agents in real-world mobile settings, comparing agent actions to essential UI states for accurate task completion. LlamaTouch supports dynamic environments, advancing mobile agent reliability and scalability in task automation.

• Autonomous Evaluation and Refinement of Digital Agents

  • Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, Alane Suhr
  • 🏛️ Institutions: UCB, UMich
  • 📅 Date: April 9, 2024
  • 📑 Publisher: COLM 2024
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [benchmark], [evaluation model], [domain transfer]
  • 📖 TLDR: This paper presents an autonomous evaluation framework for digital agents to enhance performance on web navigation and device control. The study introduces modular, cost-effective evaluators achieving up to 92.9% accuracy in benchmarks like WebArena and outlines their use in fine-tuning agents, improving state-of-the-art by 29% without additional supervision.

• Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

  • Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, Zhe Gan
  • 🏛️ Institutions: Apple
  • 📅 Date: April 8, 2024
  • 📑 Publisher: ECCV 2024
  • 💻 Env: [Mobile]
  • 🔑 Key: [model], [framework], [dataset], [benchmark], [mobile UI understanding]
  • 📖 TLDR: This paper presents Ferret-UI, a multimodal large language model (MLLM) designed to understand and interact with mobile user interfaces. The model incorporates advanced capabilities for referring, grounding, and reasoning about UI elements. By training on a variety of UI tasks, Ferret-UI achieves high performance in tasks such as icon recognition and text extraction. The authors introduce a unique architecture that allows for improved visual feature extraction from mobile screens, paving the way for applications in accessibility and user interaction.

• Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Reranking

  • Lei Ding, Jeshwanth Bheemanpally, Yi Zhang
  • 🏛️ Institutions: UCSC
  • 📅 Date: April 2024
  • 📑 Publisher: SIGIR 2024
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [benchmark], [reranking], [verification], [mobile task automation]
  • 📖 TLDR: This paper presents a system that enhances mobile "how-to" queries by verifying and reranking search results through automated instruction extraction, on-device action execution, and reranking based on relevance. The method improves on traditional ranking by analyzing device-specific execution success. The approach comprises a three-stage pipeline: 1) extracting step-by-step instructions from top search results, 2) validating these instructions on mobile devices, and 3) reranking based on performance. The system leverages a pre-trained GPT model for initial processing, ensuring adaptability across diverse apps and systems.

• AgentStudio: A Toolkit for Building General Virtual Agents

  • Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, Shuicheng Yan
  • 🏛️ Institutions: NTU, Skywork AI, ETH Zurich
  • 📅 Date: March 26, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [dataset], [general virtual agents], [open-ended learning], [tool creation], [GroundUI], [benchmark]
  • 📖 TLDR: AgentStudio is a robust toolkit for developing virtual agents with versatile actions, such as GUI automation and code execution. It unifies real-world human-computer interactions across OS platforms and includes diverse observation and action spaces, facilitating comprehensive training and benchmarking in complex settings. The toolkit's flexibility promotes agent generalization across varied tasks, supporting tool creation and a multimodal interaction interface to advance agent adaptability and learning.

• WebVLN: Vision-and-Language Navigation on Websites

  • Qi Chen, Dileepa Pitawela, Chongyang Zhao, Gengze Zhou, Hsiang-Ting Chen, Qi Wu
  • 🏛️ Institutions: The University of Adelaide
  • 📅 Date: March 24, 2024
  • 📑 Publisher: AAAI 2024
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [dataset], [web-based VLN], [HTML content integration], [multimodal navigation]
  • 📖 TLDR: This paper introduces the WebVLN task, where agents navigate websites by following natural language instructions that include questions and descriptions. Aimed at emulating real-world browsing behavior, the task allows the agent to interact with elements not directly visible in the rendered content by integrating HTML-specific information. A new WebVLN-Net model, based on the VLN BERT framework, is introduced alongside the WebVLN-v1 dataset, supporting question-answer navigation across web pages. This framework demonstrated significant improvement over existing web-based navigation methods, marking a new direction in vision-and-language navigation research.

• Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as a Case Study

  • Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, Yifei Bi, Pengjie Gu, Xinrun Wang, Börje F. Karlsson, Bo An, Zongqing Lu
  • 🏛️ Institutions: NTU, BAAI, PKU
  • 📅 Date: March 5, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [Cradle], [General Computer Control], [multimodal], [keyboard and mouse control], [long-term memory], [reasoning], [self-improvement]
  • 📖 TLDR: This paper introduces Cradle, a framework designed to achieve General Computer Control (GCC) by enabling agents to perform any computer task using only screen images (and possibly audio) as input and producing keyboard and mouse operations as output. The authors deploy Cradle in the complex AAA game Red Dead Redemption II, demonstrating its capability to follow the main storyline and complete real missions with minimal reliance on prior knowledge or resources.

• Cradle: Empowering Foundation Agents Towards General Computer Control

  • Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, Zongqing Lu
  • 🏛️ Institutions: Skywork AI, BAAI, NTU, PKU, Institute of Software - Chinese Academy of Sciences, HKU, CUHK
  • 📅 Date: March 5, 2024
  • 📑 Publisher: TBD
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [model], [general computer control], [skill curation], [self-improvement]
  • 📖 TLDR: This paper introduces the Cradle framework, designed to enable general computer control (GCC) through multimodal input (e.g., screen images and optional audio) and outputs (keyboard and mouse). Cradle’s six core modules, including self-reflection, skill curation, and memory, allow for generalized task handling in complex environments like AAA games. Demonstrated in Red Dead Redemption II, the framework exhibits adaptability by performing real missions and following the storyline with minimal prior knowledge, showcasing its potential as a generalist agent for diverse computer tasks.

• Android in the Zoo: Chain-of-Action-Thought for GUI Agents

  • Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, Duyu Tang
  • 🏛️ Institutions: Fudan University, Huawei
  • 📅 Date: March 5, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [dataset], [Android GUI], [Chain-of-Action-Thought], [autonomous GUI agents]
  • 📖 TLDR: This paper introduces Chain-of-Action-Thought (CoAT), a novel paradigm to improve GUI agent task completion by enabling agents to interpret previous actions, current screen content, and action rationale for next steps. The authors present the Android-In-The-Zoo (AitZ) dataset, which includes 18,643 screen-action pairs with detailed annotations, supporting CoAT's development and evaluation. The study demonstrates that fine-tuning with the AitZ dataset improves performance of a baseline large language model in predicting correct action sequences in Android tasks.

• Improving Language Understanding from Screenshots

  • Tianyu Gao, Zirui Wang, Adithya Bhaskar, Danqi Chen
  • 🏛️ Institutions: Princeton University
  • 📅 Date: February 22, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [model], [framework], [screenshot language models], [patch-and-text prediction]
  • 📖 TLDR: This paper introduces a novel approach to improve the language understanding capabilities of screenshot language models (LMs). The authors propose a Patch-and-Text Prediction (PTP) objective, which masks and recovers both image patches and text within screenshots. The method significantly narrows the performance gap between screenshot LMs and text-only models on language understanding tasks, achieving comparable results to BERT on most GLUE tasks. The research also extends PTP to train autoregressive screenshot LMs, demonstrating improved perplexity by utilizing screenshot context.

• UFO: A UI-Focused Agent for Windows OS Interaction

  • Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
  • 🏛️ Institutions: Microsoft
  • 📅 Date: February 14, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [UI automation], [Windows], [UFO]
  • 📖 TLDR: This paper presents UFO, a pioneering multimodal LLM-based agent designed to fulfill user requests on Windows OS. UFO employs a dual-agent architecture—comprising AppAgent and ActAgent—that can interpret and execute complex tasks across multiple Windows applications by observing UI elements and utilizing control interactions. The framework allows UFO to handle intricate, cross-application workflows and execute commands seamlessly based on natural language prompts. It integrates GPT-Vision to recognize and interact with graphical elements, enabling flexible, autonomous task completion within and across diverse Windows applications.

• ScreenAgent: A Computer Control Agent Driven by Visual Language Large Model

  • Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, Qi Wang
  • 🏛️ Institutions: Jilin University
  • 📅 Date: February 13, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [visual language model], [computer control agent]
  • 📖 TLDR: This paper introduces ScreenAgent, a computer control agent powered by a visual language large model. The system can interpret natural language instructions and execute them on various computer applications by analyzing screen content. ScreenAgent employs a novel action grounding mechanism to map high-level instructions to specific UI interactions. Evaluated on a diverse set of tasks across different applications, ScreenAgent demonstrates superior performance in task completion and generalization compared to existing methods.

• OS-Copilot: Towards Generalist Computer Agents with Self-Improvement

  • Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, Lingpeng Kong
  • 🏛️ Institutions: Shanghai AI Lab, East China Normal University, Princeton University, University of Hong Kong
  • 📅 Date: February 12, 2024
  • 📑 Publisher: ICLR 2024 Workshop LLMAgents
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [self-directed learning], [GAIA], [FRIDAY], [OS-Copilot]
  • 📖 TLDR: The OS-Copilot framework supports building generalist agents capable of performing diverse tasks across an operating system (OS). This work introduces FRIDAY, an embodied agent using OS-Copilot to self-improve by learning from task outcomes. It operates with a memory-based architecture to tackle OS-level tasks across applications like terminals, web browsers, and third-party tools. Tested on the GAIA benchmark, FRIDAY achieved 35% higher performance than prior methods, proving effective in adapting to unfamiliar applications and refining its capabilities with minimal guidance.

• Dual-View Visual Contextualization for Web Navigation

  • Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, Wei-Lun Chao
  • 🏛️ Institutions: OSU
  • 📅 Date: February 6, 2024
  • 📑 Publisher: CVPR 2024
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [visual contextualization]
  • 📖 TLDR: This paper proposes a novel approach to web navigation by contextualizing HTML elements through their "dual views" in webpage screenshots. The method leverages both the textual content of HTML elements and their visual representation in the screenshot to create more informative representations for web agents. Evaluated on the Mind2Web dataset, the approach demonstrates consistent improvements over baseline methods across various scenarios, including cross-task, cross-website, and cross-domain navigation tasks.

• WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

  • Xing Han Lu, Zdeněk Kasner, Siva Reddy
  • 🏛️ Institutions: Mila, McGill University
  • 📅 Date: February 2024
  • 📑 Publisher: ICML 2024
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [dataset], [benchmark], [multi-turn dialogue], [real-world navigation], [WebLINX]
  • 📖 TLDR: WebLINX addresses the complexity of real-world website navigation for conversational agents, with a benchmark featuring over 2,300 demonstrations across 150+ websites. The benchmark allows agents to handle multi-turn instructions and interact dynamically across diverse domains, including geographic and thematic categories. The study proposes a retrieval-inspired model that selectively extracts key HTML elements and browser actions, achieving efficient task-specific representations. Experiments reveal that smaller finetuned decoders outperform larger zero-shot multimodal models, though generalization to new environments remains challenging.

• Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception

  • Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, Jitao Sang
  • 🏛️ Institutions: Beijing Jiaotong University, Alibaba
  • 📅 Date: January 29, 2024
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [benchmark]
  • 📖 TLDR: This paper presents Mobile-Agent, an autonomous multi-modal agent designed for mobile device interaction. The system integrates visual perception, natural language processing, and action prediction to navigate and operate mobile applications. The authors introduce a new dataset and benchmark for evaluating mobile agents, demonstrating Mobile-Agent's superior performance in task completion and generalization across various apps compared to existing methods.

• VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

  • Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, Daniel Fried
  • 🏛️ Institutions: CMU
  • 📅 Date: January 24, 2024
  • 📑 Publisher: ACL 2024
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [benchmark], [dataset], [multimodal agent evaluation], [visually grounded tasks]
  • 📖 TLDR: VisualWebArena is a benchmark designed for testing multimodal web agents on complex, visually grounded web tasks. It provides a reproducible framework with 910 task scenarios across real-world web applications, emphasizing open-ended, visually guided interactions. The tasks are modeled within a partially observable Markov decision process to assess agents’ capacity to interpret multimodal inputs, execute navigation, and accomplish user-defined objectives across complex visual and textual information on websites.

• GPT-4V(ision) is a Generalist Web Agent, if Grounded

  • Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su
  • 🏛️ Institutions: OSU
  • 📅 Date: January 1, 2024
  • 📑 Publisher: ICML 2024
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [dataset], [benchmark], [grounding], [SeeAct], [Multimodal-Mind2web]
  • 📖 TLDR: This paper explores the capability of GPT-4V(ision), a multimodal model, as a web agent that can perform tasks across various websites by following natural language instructions. It introduces the SEEACT framework, enabling GPT-4V to navigate, interpret, and interact with elements on websites. Evaluated using the Mind2Web benchmark and an online test environment, the framework demonstrates high performance on complex web tasks by integrating grounding strategies like element attributes and image annotations to improve HTML element targeting. However, grounding remains challenging, presenting opportunities for further improvement.

• AppAgent: Multimodal Agents as Smartphone Users

  • Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, Gang Yu
  • 🏛️ Institutions: Tencent
  • 📅 Date: December 21, 2023
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [smartphone interaction], [autonomous exploration], [self-improve]
  • 📖 TLDR: This paper introduces AppAgent, a novel multimodal agent framework designed to operate smartphone applications. The agent uses a simplified action space to mimic human-like interactions such as tapping and swiping. AppAgent learns to navigate and use new apps through autonomous exploration or by observing human demonstrations, creating a knowledge base for executing complex tasks across different applications. The framework's effectiveness is demonstrated through extensive testing on 50 tasks across 10 diverse applications.

• AssistGUI: Task-Oriented Desktop Graphical User Interface Automation

  • Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, Mike Zheng Shou
  • 🏛️ Institutions: NUS
  • 📅 Date: December 20, 2023
  • 📑 Publisher: CVPR 2024
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [dataset], [benchmark], [desktop productivity tasks]
  • 📖 TLDR: This study presents AssistGUI, a benchmark and framework for desktop GUI automation, featuring an LLM-based agent capable of completing complex user requests by analyzing instructional videos and performing actions on the desktop. Utilizing a novel Actor-Critic framework and GUI parser, AssistGUI was tested on 100 tasks across nine applications, such as MS Word and After Effects. Despite advances, the top-performing model achieved only a 46% success rate, illustrating the challenge of comprehensive desktop automation and underscoring areas for future research in agent-driven GUI tasks.

• GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation

  • An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, Lijuan Wang
  • 🏛️ Institutions: UCSD, Microsoft, UCSB, UWM
  • 📅 Date: November 13, 2023
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [benchmark], [zero-shot GUI navigation], [multimodal LLMs]
  • 📖 TLDR: This paper explores the capabilities of GPT-4V in navigating smartphone GUIs without prior training. The authors introduce a novel framework for GUI navigation and a new benchmark, MobileNav, featuring 1,000 navigation tasks across 100 mobile apps. The study demonstrates GPT-4V's impressive zero-shot performance in understanding and interacting with mobile interfaces, outperforming previous methods and even approaching human-level performance on some tasks.

• Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large Language Models

  • Fangzhi Xu, Qiushi Sun, Kanzhi Cheng, Jun Liu, Yu Qiao, Zhiyong Wu
  • 🏛️ Institutions: Xi'an Jiaotong University, Shanghai AI Lab, HKU, Nanjing University
  • 📅 Date: November 2023
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI (evaluated on web, math reasoning, and logic reasoning environments)]
  • 🔑 Key: [framework], [dataset], [neural-symbolic self-training], [online exploration], [self-refinement]
  • 📖 TLDR: This paper introduces ENVISIONS, a neural-symbolic self-training framework designed to improve large language models (LLMs) by enabling self-training through interaction with a symbolic environment. The framework addresses symbolic data scarcity and enhances LLMs' symbolic reasoning proficiency by iteratively exploring, refining, and learning from symbolic tasks without reinforcement learning. Extensive evaluations across web navigation, math, and logical reasoning tasks highlight ENVISIONS as a promising approach for enhancing LLM symbolic processing.

• Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

  • Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao
  • 🏛️ Institutions: MSR
  • 📅 Date: October 17, 2023
  • 📑 Publisher: arXiv
  • 💻 Env: [Misc]
  • 🔑 Key: [visual prompting], [framework], [benchmark], [visual grounding], [zero-shot]
  • 📖 TLDR: This paper introduces Set-of-Mark (SoM), a novel visual prompting approach designed to enhance the visual grounding capabilities of multimodal models like GPT-4V. By overlaying images with spatially and semantically distinct marks, SoM enables fine-grained object recognition and interaction within visual data, surpassing conventional zero-shot segmentation methods in accuracy. The framework is validated on tasks requiring detailed spatial reasoning, demonstrating a significant improvement over existing visual-language models without fine-tuning.

• OpenAgents: An Open Platform for Language Agents in the Wild

  • Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, Tao Yu
  • 🏛️ Institutions: HKU, XLang Lab, Sea AI Lab, Salesforce Research
  • 📅 Date: October 16, 2023
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [Data Agent], [Plugins Agent], [Web Agent]
  • 📖 TLDR: This paper introduces OpenAgents, an open-source platform designed to facilitate the use and hosting of language agents in real-world scenarios. It features three agents: Data Agent for data analysis using Python and SQL, Plugins Agent with access to over 200 daily API tools, and Web Agent for autonomous web browsing. OpenAgents aims to provide a user-friendly web interface for general users and a seamless deployment experience for developers and researchers, promoting the development and evaluation of innovative language agents in practical applications.

• Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API

  • Zhizheng Zhang, Wenxuan Xie, Xiaoyi Zhang, Yan Lu
  • 🏛️ Institutions: MSRA
  • 📅 Date: October 7, 2023
  • 📑 Publisher: arXiv
  • 💻 Env: [GUI]
  • 🔑 Key: [model], [framework], [reinforcement learning], [UI task automation], [instruction grounding]
  • 📖 TLDR: This paper introduces a multimodal model, termed RUIG (Reinforced UI Instruction Grounding), for automating UI tasks through natural language instructions. By leveraging a pixel-to-sequence approach, the model directly decodes UI element locations from screenshots based on user commands, removing the need for metadata like element coordinates. The framework uses a transformer-based encoder-decoder setup optimized through reinforcement learning to improve spatial accuracy. This novel approach outperforms prior methods, offering a generalized solution for UI task automation.

• SteP: Stacked LLM Policies for Web Actions

  • Paloma Sodhi, S.R.K. Branavan, Yoav Artzi, Ryan McDonald
  • 🏛️ Institutions: ASAPP Research, Cornell University
  • 📅 Date: October 5, 2023
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [policy composition], [dynamic control], [SteP]
  • 📖 TLDR: This paper introduces SteP (Stacked LLM Policies), a framework that dynamically composes policies to tackle diverse web tasks. By defining a Markov Decision Process where the state is a stack of policies, SteP enables adaptive control that adjusts to task complexity. Evaluations on WebArena, MiniWoB++, and a CRM simulator demonstrate that SteP significantly outperforms existing methods, achieving a success rate improvement from 14.9% to 35.8% over state-of-the-art GPT-4 policies.

• You Only Look at Screens: Multimodal Chain-of-Action Agents

  • Zhuosheng Zhang, Aston Zhang
  • 🏛️ Institutions: SJTU
  • 📅 Date: September 20, 2023
  • 📑 Publisher: ICLR 2024
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [dataset], [benchmark], [multimodal agent], [chain-of-action technique]
  • 📖 TLDR: This paper presents Auto-GUI, a multimodal agent capable of directly interacting with graphical user interfaces without relying on environment parsing or application-specific APIs. The authors introduce a novel chain-of-action technique that leverages previous action histories and future action plans to improve decision-making. Auto-GUI is evaluated on a new device-control benchmark, AITW, demonstrating state-of-the-art performance in action prediction and task completion across various applications and web-based tasks.

• LASER: LLM Agent with State-Space Exploration for Web Navigation

  • Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Dong Yu, Jianshu Chen
  • 🏛️ Institutions: Tencent AI Lab
  • 📅 Date: September 15, 2023
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [state-space exploration], [backtracking]
  • 📖 TLDR: This paper introduces LASER, an LLM agent that models interactive web navigation tasks as state-space exploration. The approach defines a set of high-level states and associated actions, allowing the agent to transition between states and backtrack from errors. LASER significantly outperforms previous methods on the WebShop task without using in-context examples, demonstrating improved handling of novel situations and mistakes during task execution.

• AutoDroid: LLM-powered Task Automation in Android

  • Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, Yunxin Liu
  • 🏛️ Institutions: Tsinghua University, Shanghai AI Lab, University of Notre Dame, MSR
  • 📅 Date: August 29, 2023
  • 📑 Publisher: MobiCom 2024
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [dataset], [benchmark], [Android task automation], [LLM-powered agent]
  • 📖 TLDR: This paper introduces AutoDroid, a novel mobile task automation system capable of handling arbitrary tasks on any Android application without manual efforts. The framework combines the commonsense knowledge of LLMs with domain-specific knowledge of apps through automated dynamic analysis. AutoDroid features a functionality-aware UI representation method, exploration-based memory injection techniques, and a multi-granularity query optimization module. Evaluated on a new benchmark with 158 common tasks, AutoDroid achieves a 90.9% action generation accuracy and a 71.3% task completion rate, significantly outperforming GPT-4-powered baselines.

• MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

  • Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, Feng Zhao
  • 🏛️ Institutions: USTC, Shanghai AI Lab
  • 📅 Date: July 29, 2023
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [information seeking], [planning], [AI search], [MindSearch]
  • 📖 TLDR: This paper presents MindSearch, a novel approach to web information seeking and integration that mimics human cognitive processes. The system uses a multi-agent framework consisting of a WebPlanner and WebSearcher. The WebPlanner models multi-step information seeking as a dynamic graph construction process, decomposing complex queries into sub-questions. The WebSearcher performs hierarchical information retrieval for each sub-question. MindSearch demonstrates significant improvements in response quality and depth compared to existing AI search solutions, processing information from over 300 web pages in just 3 minutes.

• WebArena: A Realistic Web Environment for Building Autonomous Agents

  • Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, Graham Neubig
  • 🏛️ Institutions: CMU
  • 📅 Date: July 26, 2023
  • 📑 Publisher: NeurIPS 2023
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [benchmark], [multi-tab navigation], [web-based interaction], [agent simulation]
  • 📖 TLDR: WebArena provides a standalone, realistic web simulation environment where autonomous agents can perform complex web-based tasks. The platform offers functionalities such as multi-tab browsing, element interaction, and customized user profiles. Its benchmark suite contains 812 tasks grounded in high-level natural language commands. WebArena uses multi-modal observations, including HTML and accessibility tree views, supporting advanced tasks that require contextual understanding across diverse web pages, making it suitable for evaluating generalist agents in real-world web environments.

• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

  • Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, Aleksandra Faust
  • 🏛️ Institutions: Google DeepMind, The University of Tokyo
  • 📅 Date: July 2023
  • 📑 Publisher: ICLR 2024
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [program synthesis], [HTML comprehension], [web automation], [self-supervised learning]
  • 📖 TLDR: WebAgent leverages two LLMs—HTML-T5 for HTML comprehension and Flan-U-PaLM for program synthesis—to complete web automation tasks. It combines planning, HTML summarization, and code generation to navigate and interact with real-world web environments, improving success rates on HTML-based tasks and achieving state-of-the-art performance in benchmarks like MiniWoB and Mind2Web. The modular architecture adapts well to open-domain tasks, using local-global attention mechanisms to manage long HTML contexts.

• Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control

  • Longtao Zheng, Rundong Wang, Xinrun Wang, Bo An
  • 🏛️ Institutions: NTU
  • 📅 Date: June 13, 2023
  • 📑 Publisher: ICLR 2024
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [benchmark], [trajectory prompting], [state abstraction], [memory retrieval]
  • 📖 TLDR: Synapse introduces a novel framework for computer control tasks, leveraging trajectory-as-exemplar prompting and memory to enhance LLM performance in complex, multi-step computer tasks. The system combines state abstraction, trajectory-based prompts, and memory retrieval, overcoming LLM limitations by filtering task-irrelevant data, storing exemplar trajectories, and retrieving relevant instances for improved decision-making. Synapse achieves significant performance gains on benchmarks such as MiniWoB++ and Mind2Web, demonstrating enhanced task success rates and generalization across diverse web-based tasks.

• SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models

  • Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, Zhaoxiang Zhang
  • 🏛️ Institutions: UCAS, HKISI-CAS, PolyU, Shanghai AI Lab
  • 📅 Date: May 30, 2023
  • 📑 Publisher: NeurIPS 2023
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [spreadsheet automation], [natural language interface]
  • 📖 TLDR: This paper introduces SheetCopilot, an innovative system that leverages large language models to automate spreadsheet tasks through natural language interactions. The framework includes a novel prompt design for task decomposition and execution, and a feedback loop for error correction. SheetCopilot demonstrates significant improvements in task completion rates and efficiency across various spreadsheet operations, outperforming existing methods and showing potential for enhancing productivity in spreadsheet software.

• Augmenting Autotelic Agents with Large Language Models

  • Cédric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, Marc-Alexandre Côté
  • 🏛️ Institutions: MIT, Inria, Microsoft
  • 📅 Date: May 22, 2023
  • 📑 Publisher: CoLLAs 2023
  • 💻 Env: [GUI]
  • 🔑 Key: [framework], [reinforcement learning], [goal generation], [large language models], [autotelic learning]
  • 📖 TLDR: This study introduces the Language Model-Augmented Autotelic Agent (LMA3), a framework leveraging large language models to help agents autonomously generate, represent, and learn diverse goals in a task-agnostic, text-based environment. LMA3 integrates pretrained language models to emulate human cultural knowledge, aiming to dynamically relabel goals, generate new goals, and create goal-driven reward functions without manual inputs. This approach supports skill development by autonomously expanding goal repertoires in ways that resemble human open-ended learning, showcasing potential for achieving complex, self-directed learning in AI.

• Language Models can Solve Computer Tasks

  • Geunwoo Kim, Pierre Baldi, Stephen McAleer
  • 🏛️ Institutions: UCI
  • 📅 Date: March 30, 2023
  • 📑 Publisher: NeurIPS 2023
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [benchmark], [Recursive Critique and Improve], [RCI], [MiniWoB++], [general computer tasks]
  • 📖 TLDR: This study demonstrates that large language models (LLMs) can effectively automate computer tasks using a Recursive Critique and Improve (RCI) prompting method, enabling agents to handle complex desktop tasks like email and file management. By combining RCI with existing Chain of Thought (CoT) prompting, the method outperforms prior LLM approaches and traditional supervised and reinforcement learning models on the MiniWoB++ benchmark, showing potential for broad computer task automation.

• Reflexion: Language Agents with Verbal Reinforcement Learning

  • Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao
  • 🏛️ Institutions: Northeastern University, MIT, Princeton University
  • 📅 Date: March 20, 2023
  • 📑 Publisher: NeurIPS 2023
  • 💻 Env: [Misc]
  • 🔑 Key: [framework], [learning], [verbal reinforcement learning], [Reflexion]
  • 📖 TLDR: This paper introduces Reflexion, a framework that enhances language agents by enabling them to reflect on task feedback linguistically, storing these reflections in an episodic memory to improve decision-making in future trials. Reflexion allows agents to learn from various feedback types without traditional weight updates, achieving significant performance improvements across tasks like decision-making, coding, and reasoning. For instance, Reflexion attains a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4's 80%.

• Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding

  • Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova
  • 🏛️ Institutions: Google
  • 📅 Date: February 1, 2023
  • 📑 Publisher: ICML 2023
  • 💻 Env: [Web], [Doc]
  • 🔑 Key: [model], [framework], [vision encoder], [visual language understanding], [screenshot parsing], [image-to-text]
  • 📖 TLDR: This paper introduces Pix2Struct, a model pre-trained to parse masked screenshots into simplified HTML for tasks requiring visual language understanding. By leveraging the structure of HTML and diverse web page elements, Pix2Struct captures pretraining signals like OCR and image captioning, achieving state-of-the-art performance across tasks in domains including documents, user interfaces, and illustrations.

• ReAct: Synergizing Reasoning and Acting in Language Models

  • Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao
  • 🏛️ Institutions: Princeton University, Google Research
  • 📅 Date: October 6, 2022
  • 📑 Publisher: ICLR 2023
  • 💻 Env: [Misc]
  • 🔑 Key: [framework], [reasoning], [ReAct]
  • 📖 TLDR: This paper introduces ReAct, a framework that enables large language models to generate reasoning traces and task-specific actions in an interleaved manner. By combining reasoning and acting, ReAct enhances the model's ability to perform complex tasks in language understanding and interactive decision making. The approach is validated across various benchmarks, demonstrating improved performance and interpretability over existing methods.

• Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus

  • Gang Li, Yang Li
  • 🏛️ Institutions: Google Research
  • 📅 Date: September 29, 2022
  • 📑 Publisher: ICLR 2023
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [model], [dataset], [mobile UI tasks], [region-based focus]
  • 📖 TLDR: This paper introduces "Spotlight," a vision-language model for mobile UI understanding that operates solely on visual inputs (screenshots) and a specified focus region on the screen. By leveraging a large-scale dataset and training strategies tailored to mobile interfaces, Spotlight performs multiple UI-related tasks, including widget captioning, screen summarization, command grounding, and tappability prediction. It utilizes a vision-only approach, avoiding reliance on view hierarchies to achieve greater robustness and scalability across different mobile UI environments.

• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

  • Shunyu Yao, Howard Chen, John Yang, Karthik Narasimhan
  • 🏛️ Institutions: Princeton University
  • 📅 Date: July 2022
  • 📑 Publisher: NeurIPS 2022
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [dataset], [benchmark], [e-commerce web interaction], [language grounding]
  • 📖 TLDR: This paper introduces WebShop, a simulated web-based shopping environment with over 1 million real-world products and 12,087 annotated instructions. It allows language agents to navigate, search, and make purchases based on natural language commands. The study explores how agents handle compositional instructions and noisy web data, providing a robust environment for reinforcement learning and imitation learning. The best models show effective sim-to-real transfer on websites like Amazon, illustrating WebShop’s potential for training grounded agents.

• A Data-Driven Approach for Learning to Control Computers

  • Peter C. Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, Timothy Lillicrap
  • 🏛️ Institutions: DeepMind
  • 📅 Date: February 16, 2022
  • 📑 Publisher: ICML 2022
  • 💻 Env: [Desktop]
  • 🔑 Key: [dataset], [framework], [computer control], [reinforcement learning], [multimodal transformer]
  • 📖 TLDR: This study presents a reinforcement learning-based approach to train agents for computer control tasks, using keyboard and mouse interactions guided by natural language. By leveraging human demonstration data, agents trained in this environment achieved strong cross-task generalization across the MiniWob++ benchmark. This framework demonstrates how agents can control computers as humans would, enabling enhanced performance in complex computer tasks with high transferability.

• Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning

  • Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, Yang Li
  • 🏛️ Institutions: University of Toronto
  • 📅 Date: August 6, 2021
  • 📑 Publisher: UIST 2021
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [dataset], [mobile UI summarization], [multimodal learning], [Transformer model]
  • 📖 TLDR: The paper introduces Screen2Words, an approach that utilizes multimodal learning to generate descriptive language summaries for mobile UI screens, combining textual, visual, and structural data from screens. The study created a large-scale dataset with 112,085 annotated screen summaries for 22,417 unique UIs, aiming to support model training for mobile UI understanding. The dataset facilitates a Transformer-based model trained to summarize screens by highlighting main functionalities, and the approach is validated with benchmarks in the mobile environment.

• UIBert: Learning Generic Multimodal Representations for UI Understanding

  • Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, Blaise Agüera y Arcas
  • 🏛️ Institutions: Google Research
  • 📅 Date: July 29, 2021
  • 📑 Publisher: IJCAI 2021
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [model], [dataset], [multimodal representation learning], [UI understanding]
  • 📖 TLDR: This paper presents UIBert, a multimodal model aimed at understanding user interfaces (UIs) by combining visual, textual, and structural metadata. UIBert is designed for tasks such as component retrieval and expression resolution, using a transformer-based joint image-text model. The authors introduce five novel pre-training tasks to leverage UI-specific features, enhancing accessibility and task completion in mobile applications. UIBert demonstrates superior performance on nine downstream UI tasks, highlighting the potential of multimodal pre-training in UI understanding.

• Grounding Open-Domain Instructions to Automate Web Support Tasks

  • Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, Monica Lam
  • 🏛️ Institutions: Stanford
  • 📅 Date: March 30, 2021
  • 📑 Publisher: NAACL 2021
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [framework], [grounding], [task automation], [open-domain instructions], [RUSS]
  • 📖 TLDR: This paper introduces RUSS (Rapid Universal Support Service), a framework designed to interpret and execute open-domain, step-by-step web instructions automatically. RUSS uses a BERT-LSTM model for semantic parsing into a custom language, ThingTalk, which allows the system to map language to actions across various web elements. The framework, including a dataset of instructions, facilitates agent-based web support task automation by grounding natural language to interactive commands.

• Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations

  • Toby Jia-Jun Li, Tom Mitchell, Brad Myers
  • 🏛️ Institutions: CMU
  • 📅 Date: July 2020
  • 📑 Publisher: ACL 2020
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [Sugilite], [programming-by-demonstration]
  • 📖 TLDR: This paper introduces Sugilite, an intelligent task automation agent that learns new tasks and associated concepts interactively from users' natural language instructions and demonstrations on third-party mobile app GUIs. The system allows users to teach procedures and concepts through verbal instructions combined with GUI demonstrations, supports intent clarification for demonstrated actions, infers task parameters using hierarchical app GUI structures, and generalizes taught concepts across different contexts and domains. A prototype is presented as a conversational assistant on Android. oai_citation_attribution:0‡ACL Anthology

• Mapping Natural Language Instructions to Mobile UI Action Sequences

  • Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, Jason Baldridge
  • 🏛️ Institutions: Google Researc
  • 📅 Date: July 2020
  • 📑 Publisher: ACL 2020
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [dataset], [mobile UI automation], [natural language instructions], [action grounding], [RicoSCA]
  • 📖 TLDR: This paper introduces a method for grounding natural language instructions to mobile UI actions, aiming to automate mobile task execution through user interface manipulation. It introduces three key datasets: PixelHelp for task instruction-performance mappings on a Pixel emulator, AndroidHowTo for detailed phrase extraction, and RicoSCA for synthetic UI command training. The system utilizes a Transformer model to extract action phrase tuples, aligning them to UI elements with contextual screen positioning. Achieving over 70% accuracy in task completion, this approach is foundational for natural language-driven mobile UI automation.

• Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

  • Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, Percy Liang
  • 🏛️ Institutions: Stanford
  • 📅 Date: February 24, 2018
  • 📑 Publisher: ICLR 2018
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [benchmark], [reinforcement learning], [web tasks], [workflow-guided exploration]
  • 📖 TLDR: This paper presents a novel RL approach using workflow-guided exploration to efficiently train agents on web-based tasks, where actions are restricted based on demonstrated workflows to streamline learning. Evaluated on MiniWoB and MiniWoB++ benchmarks, the method significantly outperforms traditional RL techniques in sparse reward settings by structuring exploration according to high-level action constraints.

• World of Bits: An Open-Domain Platform for Web-Based Agents

  • Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, Percy Liang
  • 🏛️ Institutions: Stanford, OpenAI
  • 📅 Date: August 2017
  • 📑 Publisher: ICML 2017
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [dataset], [reinforcement learning], [open-domain]
  • 📖 TLDR: This paper introduces World of Bits (WoB), a platform enabling agents to perform complex web-based tasks using low-level keyboard and mouse actions, addressing the lack of open-domain realism in existing reinforcement learning environments. WoB leverages a novel framework where crowdworkers create tasks with structured rewards and reproducibility by caching web interactions, forming a stable training environment. The authors validate WoB by training agents via behavioral cloning and reinforcement learning to accomplish various real-world tasks, showcasing its potential as an effective platform for reinforcement learning on web tasks.

• SUGILITE: Creating Multimodal Smartphone Automation by Demonstration

  • Toby Jia-Jun Li, Amos Azaria, Brad A. Myers
  • 🏛️ Institutions: CMU, Ariel University
  • 📅 Date: May 6, 2017
  • 📑 Publisher: CHI 2017
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [PBD], [multimodal interaction], [SUGILITE], [programming-by-demonstration], [demonstration]
  • 📖 TLDR: This paper introduces SUGILITE, a programming-by-demonstration (PBD) system that enables users to automate tasks on smartphones through multimodal interactions. By leveraging Android's accessibility API, SUGILITE allows users to create generalized automation scripts for arbitrary third-party apps by demonstrating tasks using the regular app UI. The system combines verbal instructions, user demonstrations, and app UI hierarchies to generalize scripts from single demonstrations, facilitating task variations and parameterization. Extensive error handling and context checking enhance robustness against app UI changes. A lab study indicates that users with minimal programming knowledge can successfully automate smartphone tasks using SUGILITE.