Skip to content

Latest commit

 

History

History
46 lines (41 loc) · 4.32 KB

paper_Zhiyong_Wu.md

File metadata and controls

46 lines (41 loc) · 4.32 KB
Raw

Zhiyong Wu's Papers

  • OS-ATLAS: A Foundation Action Model For Generalist GUI Agents

    • Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, Yu Qiao
    • 🏛️ Institutions: Shanghai AI Lab, Shanghai Jiaotong University, HKU, MIT
    • 📅 Date: October 30, 2024
    • 📑 Publisher: arXiv
    • 💻 Env: [GUI]
    • 🔑 Key: [model], [dataset], [benchmark], [OS-Atlas]
    • 📖 TLDR: This paper introduces OS-Atlas, a foundational GUI action model designed to enhance GUI grounding and out-of-distribution tasks. The authors developed a toolkit to synthesize multi-platform GUI grounding data, resulting in a cross-platform corpus of over 13 million GUI elements. OS-Atlas demonstrates significant performance improvements across six benchmarks spanning mobile, desktop, and web platforms.

  • AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant

    • Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, Zhiyong Wu
    • 🏛️ Institutions: XJTU, Shanghai AI Lab, HKU
    • 📅 Date: October 24, 2024
    • 📑 Publisher: arXiv
    • 💻 Env: [GUI]
    • 🔑 Key: [framework], [multi-agent systems], [specialized generalist agent], [OSWorld benchmark]
    • 📖 TLDR: AgentStore introduces a scalable platform to integrate and manage heterogeneous agents, designed to enhance generalist assistant capabilities for diverse computer tasks. Using a MetaAgent and AgentToken strategy, AgentStore shows improved generalization on the OSWorld benchmark.

  • OS-Copilot: Towards Generalist Computer Agents with Self-Improvement

    • Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, Lingpeng Kong
    • 🏛️ Institutions: Shanghai AI Lab, East China Normal University, Princeton University, University of Hong Kong
    • 📅 Date: February 12, 2024
    • 📑 Publisher: ICLR 2024 Workshop LLMAgents
    • 💻 Env: [Desktop]
    • 🔑 Key: [framework], [self-directed learning], [GAIA], [FRIDAY], [OS-Copilot]
    • 📖 TLDR: The OS-Copilot framework supports building generalist agents capable of performing diverse tasks across an operating system (OS). This work introduces FRIDAY, an embodied agent using OS-Copilot to self-improve by learning from task outcomes. It operates with a memory-based architecture to tackle OS-level tasks across applications like terminals, web browsers, and third-party tools. Tested on the GAIA benchmark, FRIDAY achieved 35% higher performance than prior methods, proving effective in adapting to unfamiliar applications and refining its capabilities with minimal guidance.

  • SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

    • Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, Zhiyong Wu
    • 🏛️ Institutions: Nanjing University, Shanghai AI Lab
    • 📅 Date: January 19, 2024
    • 📑 Publisher: ACL 2024
    • 💻 Env: [GUI]
    • 🔑 Key: [model], [benchmark], [GUI grounding], [visual grounding]
    • 📖 TLDR: TBD.

  • Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large Language Models

    • Fangzhi Xu, Qiushi Sun, Kanzhi Cheng, Jun Liu, Yu Qiao, Zhiyong Wu
    • 🏛️ Institutions: Xi'an Jiaotong University, Shanghai AI Lab, HKU, Nanjing University
    • 📅 Date: November 2023
    • 📑 Publisher: arXiv
    • 💻 Env: [GUI (evaluated on web, math reasoning, and logic reasoning environments)]
    • 🔑 Key: [framework], [dataset], [neural-symbolic self-training], [online exploration], [self-refinement]
    • 📖 TLDR: This paper introduces ENVISIONS, a neural-symbolic self-training framework designed to improve large language models (LLMs) by enabling self-training through interaction with a symbolic environment. The framework addresses symbolic data scarcity and enhances LLMs' symbolic reasoning proficiency by iteratively exploring, refining, and learning from symbolic tasks without reinforcement learning. Extensive evaluations across web navigation, math, and logical reasoning tasks highlight ENVISIONS as a promising approach for enhancing LLM symbolic processing.