Skip to content

Latest commit

 

History

History
30 lines (21 loc) · 2.27 KB

README.md

File metadata and controls

30 lines (21 loc) · 2.27 KB
Raw

OpenSpaceGuide: steering OpenSpace with natural language

OpenSpaceGuide pictures in action Pictures taken in the Visualiseringscenter C dome theater.

This repo contains a script for piloting an OpenSpace instance with OpenAI's GPT4 and Whisper.

See the poster presented at IEEE Vis 2024 — poster.pdf

More details about this project on OSF — https://osf.io/jcgzt/

Instructions

  • Setup your OPENAI_API_KEY environment variable. You need a paid access to OpenAI's API, see https://platform.openai.com/api-keys.
  • Install the Python dependencies — pip install -R requirements.txt (it's recommended to setup a virtual environment first, see this).
  • Install OpenSpace — this fork contains a patch to display the text prompts on screen with --text-widget. Otherwise, install from the main repo.
  • Launch OpenSpace, then start the script (python main.py --input keyboard).

CLI reference

  • --address <addr>, --port <port>, --password <pwd>: OpenSpace server configuration. Default values will work for a locally running instance.
  • --input <keyboard|speech>: keyboard or speech. Whether to use a text input in the console or speech through the microphone via Whisper.
  • --trigger <key>: Set this to use a key press to start/stop listening (if --input is speech). Otherwise, it listens continuously.`
  • --microphone <index>: Microphone index to use. Run once to see the list.
  • --text-widget: Display prompts and responses in OpenSpace (requires using the fork)
  • --targets <comma,separated,list>: List of OpenSpace targets (Scene Identifiers) that the LLM should be aware of. Defaults to all visible renderables when the script is started.

Acknowledgements

This work is supported by the Knut and Alice Wallenberg Foundation (KAW 2019.0024), the Marcus and Amalia Wallenberg Foundation (MAW 2023.0130), and efforts from Visualiseringscenter C (Norrköping, Sweden).