Skip to content

Latest commit

 

History

History
28 lines (25 loc) · 3.27 KB

paper_programming-by-demonstration.md

File metadata and controls

28 lines (25 loc) · 3.27 KB
Raw

Papers with Keyword: programming-by-demonstration

  • Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations

    • Toby Jia-Jun Li, Tom Mitchell, Brad Myers
    • 🏛️ Institutions: CMU
    • 📅 Date: July 2020
    • 📑 Publisher: ACL 2020
    • 💻 Env: [Mobile]
    • 🔑 Key: [framework], [Sugilite], [programming-by-demonstration]
    • 📖 TLDR: This paper introduces Sugilite, an intelligent task automation agent that learns new tasks and associated concepts interactively from users' natural language instructions and demonstrations on third-party mobile app GUIs. The system allows users to teach procedures and concepts through verbal instructions combined with GUI demonstrations, supports intent clarification for demonstrated actions, infers task parameters using hierarchical app GUI structures, and generalizes taught concepts across different contexts and domains. A prototype is presented as a conversational assistant on Android. oai_citation_attribution:0‡ACL Anthology

  • PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

    • Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, Brad A. Myers
    • 🏛️ Institutions: CMU, Amherst College
    • 📅 Date: August 30, 2019
    • 📑 Publisher: UIST 2019
    • 💻 Env: [Mobile]
    • 🔑 Key: [programming-by-demonstration], [PUMICE]
    • 📖 TLDR: This paper introduces PUMICE, a multi-modal agent that combines natural language programming and programming-by-demonstration to enable end users to instruct intelligent agents in performing new tasks. By allowing users to describe tasks and conditions naturally and then collaboratively resolving ambiguities through conversation and demonstration, PUMICE facilitates the teaching of new concepts and procedures within existing mobile app GUIs. A lab study with 10 users demonstrated its usability and effectiveness.

  • SUGILITE: Creating Multimodal Smartphone Automation by Demonstration

    • Toby Jia-Jun Li, Amos Azaria, Brad A. Myers
    • 🏛️ Institutions: CMU, Ariel University
    • 📅 Date: May 6, 2017
    • 📑 Publisher: CHI 2017
    • 💻 Env: [Mobile]
    • 🔑 Key: [framework], [PBD], [multimodal interaction], [SUGILITE], [programming-by-demonstration], [demonstration]
    • 📖 TLDR: This paper introduces SUGILITE, a programming-by-demonstration (PBD) system that enables users to automate tasks on smartphones through multimodal interactions. By leveraging Android's accessibility API, SUGILITE allows users to create generalized automation scripts for arbitrary third-party apps by demonstrating tasks using the regular app UI. The system combines verbal instructions, user demonstrations, and app UI hierarchies to generalize scripts from single demonstrations, facilitating task variations and parameterization. Extensive error handling and context checking enhance robustness against app UI changes. A lab study indicates that users with minimal programming knowledge can successfully automate smartphone tasks using SUGILITE.