Using iThor to Collect Live Multi-Agent Demonstrations

[akshathsa]

Hi, my advisor and I are looking to collect multi-agent demonstrations using iThor. We hope to have multiple agents being controlled by people in some sort of collective task and see the results of each other's movements/actions as they play. We don't know how to render both agents' views at the same time on the same machine. Would we still be utilizing Unity as we would with a single agent, and is it possible to initialize the Controller so that a separate Unity window pops up for each agent? Or would we have to extract image frames from each agent? What is the recommended/preferred way of accomplishing something like this with iThor?

[mattdeitke]

Hi @akshathsa,

Yes, you can use iTHOR to collect Multi-Agent observations. Checkout this notebook on using and setting up multi-agent settings:
https://colab.research.google.com/drive/1ityVEzHNN7k4Xek7-yfX_x_VRWNtO2wz?usp=sharing

Let me know if you have any questions :)

[akshathsa]

Thanks for the response @mattdeitke! The notebook was definitely a good resource to see how iThor works with multi-agent situations.

I did have some problems getting what's shown in the notebook to work on my end, and also for our use case we might need to do something a little different, so I do have some follow-up questions. Unfortunately, I was unable to run the colab notebook successfully on my end (I'm on Windows and use an Ubuntu VM and it didn't seem to work there though I am able to run iThor directly from my source code) so I'm just going off what I expect is the functionality based on reading your code, so my apologies if I make any mistakes.

It seems that you're extracting ego-centric frames from each agent and then using matplotlib to add them as subplots to get the end result which shows each of the perspectives after each action. I tried to do something where I initialize a controller and then use what's shown in the notebook to at least plot a single image from each agent. This is what I'm trying to run:

from ai2thor.controller import Controller
import matplotlib.pyplot as plt


def visualize_frames(rgb_frames, title='', figsize=(8, 2)):
    """Plots the rgb_frames for each agent."""
    fig, axs = plt.subplots(1, len(rgb_frames), figsize=figsize, facecolor='white', dpi=300)
    for i, frame in enumerate(rgb_frames):
        ax = axs[i]
        ax.imshow(frame)
        ax.set_title(f'AgentId: {i}')
        ax.axis('off')
    if title:
        fig.suptitle(title)
    return fig


controller = Controller(
    agentMode="default",
    visibilityDistance=1.5,
    scene="FloorPlan28",
    agentCount=2,

    # step sizes
    gridSize=0.125,
    snapToGrid=True,
    rotateStepDegrees=90,

    # image modalities
    renderDepthImage=False,
    renderInstanceSegmentation=False,

    # camera properties
    width=1920,
    height=984,
    fieldOfView=90,
)

# Properly align both agents.
controller.step(action='Teleport', x=-2.5, y=0.900998235, z=-3.0, agentId=0)
controller.step(action='Teleport', x=-2.5, y=0.900998235, z=-0.25, agentId=1)
controller.step(action="RotateRight", degrees=360, agentId=1)
controller.step(action="Crouch", agentId=1)
initial_event = controller.step(action='RotateRight', degrees=180, agentId=0)

rgb_frames = [event.frame for event in initial_event.events]
visualize_frames(rgb_frames, title=f'After Action', figsize=(8, 2.5))

This ends up opening the Unity window, going through those alignment movements and then quitting with the plot never showing up. It's the same case if I add some sort of statement to keep the process going until I manually terminate. I might be missing something obvious as to why this isn't working, but I've tried a couple things and I can't seem to find out why.

Actually what we need for our use case is to display images from each of the 2 agents in separate windows/figures and then update them as actions are taken by people with a keyboard and mouse so it looks like a co-op video game with each player looking at their agent's perspective. As a result, the first thing I tried was actually to use pylab and at least get the initial image frames displayed on two separate figures:

from ai2thor.controller import Controller
from ai2thor.platform import CloudRendering
import matplotlib.pylab as pl


def visualize_frames(rgb_frames, title='', figsize=(8, 2)):
    """Plots the rgb_frames for each agent."""
    fig, axs = plt.subplots(1, len(rgb_frames), figsize=figsize, facecolor='white', dpi=300)
    for i, frame in enumerate(rgb_frames):
        ax = axs[i]
        ax.imshow(frame)
        ax.set_title(f'AgentId: {i}')
        ax.axis('off')
    if title:
        fig.suptitle(title)
    return fig

controller = Controller(
    agentMode="default",
    # visibilityDistance=1.5,
    scene="FloorPlan28",
    agentCount=2,

    # step sizes
    gridSize=0.125,
    snapToGrid=True,
    rotateStepDegrees=90,

    # image modalities
    renderDepthImage=False,
    renderInstanceSegmentation=False,

    # camera properties
    width=1920,
    height=984,
    fieldOfView=90,
)

# Properly align both agents.
controller.step(action='Teleport', x=-2.5, y=0.900998235, z=-3.0, agentId=0)
controller.step(action='Teleport', x=-2.5, y=0.900998235, z=-0.25, agentId=1)
controller.step(action="RotateRight", degrees=360, agentId=1)
controller.step(action="Crouch", agentId=1)
initial_event = controller.step(action='RotateRight', degrees=180, agentId=0)

# Create 2 plot figures - 1 for each player
Update this with the last image.
frames = [event.frame for event in initial_event.events]
pl.figure(1)
img1 = pl.imshow(frames[0])
pl.figure(2)
img2 = pl.imshow(frames[1])

If I could get this to work the plan was to then incorporate a keyboard handling script I made (because as far as I'm aware iThor doesn't have this built-in, but let me know if I'm wrong on that) which would invoke a controller.step in response to certain keypresses and then we would do something similar and get the event object from that and access the ego-centric frames and update them for both plots.

Unfortunately, the first-step code above also doesn't work for some reason and it just exits after those alignment actions without ever making those 2 figures and plotting that single image frame. This methodology works fine with iThor image frames that I saved as jpgs to my disk that I load with imread, but it doesn't work for me using the frame directly for some reason. Whether it's event.frame or from an imread, the output if I understand correctly is an ndarray in both cases, so I'm not sure what the difference is.

I was initially unsure whether iThor's Unity environment was causing this problem, but if I try to use the plotting to plot those saved jpg's while iThor is running and showing something else in the Unity window, the plots still appear and everything works, so it's only while trying to directly use the frame from iThor to plot that I have a problem, it seems.

Sorry about the long message, I wanted to do my best to convey what I've tried and what I'm trying to do and how it's not working. Any help would be greatly appreciated!

[akshathsa]

Also, there's another related question. As I mentioned, our hope would be to have 2 human players controlling the agents and we would start the process and have 2 windows/plots with the perspectives of each agent. I guess it would be better if it is consistent and we either only have multiple Unity windows/instances, if possible, or if we have to use the plotting like the colab notebook suggests that we don't have the Unity window pop up at all since it would be redundant.

I saw the documentation for running iThor headless and tried specifying platform=CloudRendering in the controller initialization from above but I get the following exception:

Traceback (most recent call last):
  File "/mnt/hgfs/VM_Portal/cooperative-multi-agent-sim/src/main.py", line 22, in <module>
    controller = Controller(
  File "/home/akshath/.local/lib/python3.8/site-packages/ai2thor/controller.py", line 492, in __init__
    self.start(
  File "/home/akshath/.local/lib/python3.8/site-packages/ai2thor/controller.py", line 1301, in start
    self._start_unity_thread(env, width, height, unity_params, image_name)
  File "/home/akshath/.local/lib/python3.8/site-packages/ai2thor/controller.py", line 1025, in _start_unity_thread
    raise Exception(message)
Exception: Unity process has exited - check Player.log for errors. Confirm that Vulkan is properly configured on this system using vulkaninfo from the vulkan-utils package. returncode=-11

And the log file gives:

Preloaded 'libjpeg.so'
Preloaded 'libpngslz.so'
Preloaded 'libturbojpeg.so'
Unable to load player prefs
Initialize engine version: 2021.2.0b11 (0bffbba03cb9)
[Subsystems] Discovering subsystems at path /home/akshath/.ai2thor/releases/thor-CloudRendering-54535f6b9d76896c2ccb4532727aeda5741a9061/thor-CloudRendering-54535f6b9d76896c2ccb4532727aeda5741a9061_Data/UnitySubsystems
Forcing GfxDevice: Vulkan
GfxDevice: creating device client; threaded=1; jobified=1
[Vulkan init] extensions: count=18
[Vulkan init] extensions: name=VK_KHR_device_group_creation, enabled=0
[Vulkan init] extensions: name=VK_KHR_display, enabled=1
[Vulkan init] extensions: name=VK_KHR_external_fence_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_display_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_physical_device_properties2, enabled=1
[Vulkan init] extensions: name=VK_KHR_get_surface_capabilities2, enabled=0
[Vulkan init] extensions: name=VK_KHR_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_surface_protected_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_wayland_surface, enabled=0
[Vulkan init] extensions: name=VK_KHR_xcb_surface, enabled=0
[Vulkan init] extensions: name=VK_KHR_xlib_surface, enabled=0
[Vulkan init] extensions: name=VK_EXT_acquire_xlib_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_debug_report, enabled=0
[Vulkan init] extensions: name=VK_EXT_direct_mode_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_surface_counter, enabled=0
[Vulkan init] extensions: name=VK_EXT_debug_utils, enabled=0
Vulkan detection: 2
[Vulkan init] extensions: count=18
[Vulkan init] extensions: name=VK_KHR_device_group_creation, enabled=0
[Vulkan init] extensions: name=VK_KHR_display, enabled=1
[Vulkan init] extensions: name=VK_KHR_external_fence_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_display_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_physical_device_properties2, enabled=1
[Vulkan init] extensions: name=VK_KHR_get_surface_capabilities2, enabled=0
[Vulkan init] extensions: name=VK_KHR_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_surface_protected_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_wayland_surface, enabled=0
[Vulkan init] extensions: name=VK_KHR_xcb_surface, enabled=0
[Vulkan init] extensions: name=VK_KHR_xlib_surface, enabled=0
[Vulkan init] extensions: name=VK_EXT_acquire_xlib_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_debug_report, enabled=0
[Vulkan init] extensions: name=VK_EXT_direct_mode_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_surface_counter, enabled=0
[Vulkan init] extensions: name=VK_EXT_debug_utils, enabled=0
[Vulkan init] SelectPhysicalDevice requestedDeviceIndex=-1 xrDevice=(nil)
[Vulkan init] Physical Device 0x5606b8d39a20 [0]: "llvmpipe (LLVM 12.0.0, 256 bits)" deviceType=4 vendorID=10005 deviceID=0
[Vulkan init] Selected physical device (nil)

The same thing happens with trying to run just the basic code snippet from the docs too. So is there any way I can get around this problem or something that I should do differently? Once again, thanks in advance for all the help!

[akshathsa]

Hi, @mattdeitke it's been some time since my initial post, so I'm just following up on the above issues in case you have any insights on how I can get around the problems I have been having.