tutorials: Support meshcat running in a jupyter cell (as iframe) on Google Colab

[EricCousineau-TRI]

Problem

We don't have an example of doing 3D, animated visualization on Binder.

Reproduction

Based on the meshcat-python demo:
https://github.com/rdeits/meshcat-python/blob/549171bcf11ee422904fcf4858e231a354191eae/demo.ipynb
I briefly tried out showing a Jupyter cell in Binder:

import meshcat
vis = meshcat.Visualizer()
vis.url()
vis.jupyter_cell()

The Meshcat server could run, but I could not connect to the client.

I (very naively) tried to use http://{ip}.0.0.1:7000/static/, where {ip} is the publicly-visible IP address from the running instance on Binder:

import json
from urllib.request import urlopen
ip = urlopen('https://api.ipify.org').read().decode("utf-8")
print(ip)

But this did not work. I'm assuming we may have to get creative (if it's at all possible) to show the visualization on the client side.

Potential Solutions / Prototypes

Per this comment, meshcat may be viable, either via static HTML or by forwarding ports.

Other alternative being investigated are other direct / indirect three.js solutions, such as:

• itkwidgets (vtk.js)
• three.js directly

Prototypes

• Try tikwidgets (vtk.js)
  • WIP: Add itkwidgets examples to jupyter notebook #13182 Static viz
  • Animation viz
• Add meshcat animation (and binder tutorial) #13513 Try MeshCat render_static() on jupyter notebook #13498 Try meshcat static HTML
• Try meshcat via forwarding ports via Binder's mechanisms

Evaluation

TBD

---

UPDATE (2020/01/28): Per Jamie's comment below, the best solution for this may be to not use MeshCat, but instead another visualizer that can better handle the Binder-/JupyterHub- style workflow.

---

For displaying the widget / cell pointing to the appropriate cell, we may need to somehow specify the host option in meshcat, but it's unclear how to do so. Some relevant lines of code:
https://github.com/rdeits/meshcat-python/blob/549171bcf11ee422904fcf4858e231a354191eae/src/meshcat/visualizer.py#L42-L44
https://github.com/rdeits/meshcat-python/blob/549171bcf11ee422904fcf4858e231a354191eae/src/meshcat/servers/zmqserver.py#L167-L173

If the notebook is run locally, everything is fine; see #12646 for an example.

\cc @RussTedrake @TobiaMarcucci

[EricCousineau-TRI]

For reference, it seems like pyntcloud has notebooks that run on Binder and can show (WebGL?) visualization widgets:
https://github.com/daavoo/pyntcloud/blob/master/README.rst

Example Binder link:
https://mybinder.org/v2/gh/daavoo/pyntcloud/master?filepath=examples/[visualization]%20Polylines.ipynb

EDIT: Looks like the default backend uses pythreejs:
https://github.com/jupyter-widgets/pythreejs

[EricCousineau-TRI]

Just to expand, this difference in functionality can be reproduced in a local Docker container, per this bit:
https://github.com/RobotLocomotion/drake/tree/1ffb58bd1fa5e0c91e38c4bc7ec0de4cbcbfa56e/.binder#docker-image-for-binder
I cannot see the meshcat visualization in Jupyter within docker, but can see it on my local machine.

I can also install open3d==0.9.0 (with a newer version of Jupyter notebook), and can visualize point clouds with code like:

# http://www.open3d.org/docs/release/tutorial/Basic/working_with_numpy.html
import numpy as np
import open3d as o3d

# generate some neat n times 3 matrix using a variant of sync function
x = np.linspace(-3, 3, 401)
mesh_x, mesh_y = np.meshgrid(x, x)
z = np.sinc((np.power(mesh_x, 2) + np.power(mesh_y, 2)))
z_norm = (z - z.min()) / (z.max() - z.min())
xyz = np.zeros((np.size(mesh_x), 3))
xyz[:, 0] = np.reshape(mesh_x, -1)
xyz[:, 1] = np.reshape(mesh_y, -1)
xyz[:, 2] = np.reshape(z_norm, -1)

# Pass xyz to Open3D.o3d.geometry.PointCloud and visualize
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(xyz)

# http://www.open3d.org/docs/release/tutorial/Basic/jupyter.html
visualizer = JVisualizer()
visualizer.add_geometry(pcd)
visualizer.show()

[jamiesnape]

The first part of this is going to be whether we can actually expose a port when running on Binder. If it happens to run the equivalent of docker run -P (it uses kubernetes, I think, however) then that part is easy. Locally, you would want to add something like -p 7000:7000 to the docker run command.

[jamiesnape]

And (after just trying) as you would imagine, it does not allow you to expose 7000, so no websocket connection, so no meshcat visualization.

[jamiesnape]

(so the short answer would be that using a different visualizer would be a million times easier)

[EricCousineau-TRI]

Aye, makes sense...

[jamiesnape]

Are going to therefore close this as not possible or re-scope it as use a different visualizer or rewrite much of meshcat?

[EricCousineau-TRI]

Re-scoping to achieve some level of 3D visualization on Binder.

[EricCousineau-TRI]

Also filed: meshcat-dev/meshcat#67

[EricCousineau-TRI]

Next step is to try another three.js-based solution, but one that doesn't require WebSockets (may be slower... but works?)

[jamiesnape]

We also have something like itkwidgets based on vtk.js:

https://pypi.org/project/itkwidgets/

with Binder examples here:

https://mybinder.org/v2/gh/InsightSoftwareConsortium/itkwidgets/master?urlpath=lab/tree/examples

[BetsyMcPhail]

I did some experimentation with itkwidgets.

I tried some of the itk examples linked above (#13182). These examples work in Binder assuming itk and itkwidgets are available. Note that the version 0.27.0 of itkwidgets is not rendering in Binder (itkwidgets issue), I used itkwidgets==0.26.1 to test.

As a sidenote, I tried pip installing itk and itkwidgets directly from Binder , e.g.

import sys
!{sys.executable} -m pip install itk
!{sys.executable} -m pip install itkwidgets

But I couldn't import itkwidgets?

The vtk examples currently don't work in Binder (brief explanation).

I haven't explored this at all, but this disscussion seems to indicate that it might be possible?

[EricCousineau-TRI]

@BetsyMcPhail For your WIP branch, can you instead try to hack the .binder/Dockerfile (not mac prereqs) to install a custom-built version of VTK for the version you need and then post the exact command-line you used to run the Docker container image locally?

[EricCousineau-TRI]

If it takes too long for vtk.js-based solutions, then we may want to just do direct pythreejs / three.js solutions as shown in above examples.

[BetsyMcPhail]

#13182 has been updated with a new notebook (tutorials/3dvisualization_tests) that has working examples from itkwidgets, open3d and pyntcloud.

.binder/Dockerfile has been updated to install and setup any necessary modules.

Use the docker image for Binder to try locally.

[ToffeeAlbina-TRI]

@EricCousineau-TRI do you want to evaluate this as is and decide whether/what other visualizer options Betsy should explore?

[EricCousineau-TRI]

Thanks! I've just ran through the examples and provided some feedback in the PR.

[EricCousineau-TRI]

@jamiesnape When you have a chance, per your prior comment:

And (after just trying) as you would imagine, it does not allow you to expose 7000, so no websocket connection, so no meshcat visualization.

Would you or Betsy have time to add a permalink link to the JupyterHub code which indicates that this is not possible?

[jamiesnape]

There isn't public code. It would be a server firewall configuration.

(Also it is Binder, not JupyterHub.)

[EricCousineau-TRI]

Just for now, will re-assign this to you Russ until there're more action items for someone else.

[RussTedrake]

Just to update my thoughts on this...

After talking to @rdeits a week or so ago, he mentioned that the entire meshcat network setup was actually inspired by the way that jupyter kernels work (websockets and zmq). Assuming colab is the same, it might be that the most viable option here is actually to figure out if we can piggyback on the existing websocket connection between the browser and the server instead of trying to instantiate our own. @rdeits said he had managed to do that with Jupyter once before.

[duburcqa]

Hi guys ! I had a similar issue. I managed to get it work after "fixing" websocket, by enabling cross-traffic. Here is a working snippet enabling to render a custom javascript meshcat viewer in jupyter:

from IPython.display import HTML

HTML("""
<div style="height: 400px; width: 100%; overflow-x: auto; overflow-y: hidden; resize: both">
    <iframe id="testiframe" src='data:text/html,
        <div id="meshcat-pane" style="height: 100%; width: 100%; overflow-x: auto; overflow-y: hidden; resize: both">
        </div>

        <script type="text/javascript" src="http://{url}/static/main.min.js"></script>
        <script>
            var viewer = new MeshCat.Viewer(document.getElementById("meshcat-pane"));
            viewer.connect("ws://{url}");
        </script>

        <script id="embedded-json"></script>'; style="width: 100%; height: 100%; border: none">
    </iframe>
</div>
""".format(url='127.0.0.1:32784'))

[rdeits]

Ok, here's a direction which doesn't require any port forwarding, server configuration, certificates, or proxy. Rather than using a websocket, we can send commands from python to the frontend using the built-in Jupyter comms protocol. I've set up a dumb example that renders an iframe and then sends messages to it from Python here: https://github.com/rdeits/meshcat-python/blob/jupyter-comms-demo/comms.ipynb

This means:

• no websocket (other than what Jupyter already uses under the hood)
• no proxy
• no port forwarding
• no security holes (we're just using the comms that the notebook already provides)
• no certificates
• no ZMQ server at all

This would require some changes to meshcat to support sending messages this way instead of over ZMQ, but it might not be too bad. Essentially, we'd have to replace the calls to zeromq.send with command_channel.send(). Figuring out a nice way to do that without breaking the existing use-cases would require some thought, but I don't think it's too bad.

@RussTedrake this is what I was referring to when I talked about using the Jupyter comms protocol itself. It should also work on Colab (although I haven't tested) since it's the same underlying mechanism that things like altair and ipywidgets use.

[RussTedrake]

@duburcqa -- thanks. That one we knew how to do, and Robin already has it basically available in the jupyter_cell() method. The trick is that colab won't allow the connection via ws. It needs wss.

@rdeits -- awesome! it worked exactly as expected on my local machine, but my first copy/paste to colab did not:
https://colab.research.google.com/drive/1wsgE6EzRORqc2uXotRaL5lpbCZYvBYvH?usp=sharing
It looks like window.parent.Jupyter is undefined on colab. I found window.colab (also window.parent.colab), which has kernel, but not comm_manager. Will dig a little more.

[RussTedrake]

Ah, just found this: https://github.com/googlecolab/colabtools/blob/master/packages/outputframe/lib/index.d.ts
(from https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb#scrollTo=Ytn7tY-C9U0T)

which does have APIs that refer back to https://jupyter-notebook.readthedocs.io/en/stable/comms.html

It looks like this example is what we want:
https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb#scrollTo=1-FIHCdGKCyO

[RussTedrake]

Made some progress, but hit (I think) a missing feature/export in colab. I've asked on stackoverflow.

[RussTedrake]

OK, I have a proof of life, albeit with the probably inefficient version of opening a new comm channel for every message.
https://colab.research.google.com/drive/1O-WgLtqKGuHKEeD4CFT69870dPEJRsQJ
As @rdeits says, I think the rest is just plumbing, and shouldn't be that bad!

[RussTedrake]

Woot! I think I've get everything working now (though it will get more efficient if the stackoverflow generates a response).
https://colab.research.google.com/drive/1O-WgLtqKGuHKEeD4CFT69870dPEJRsQJ
meshcat PR is linked above.

[jcarpent]

@RussTedrake We would also like to integrate Pinocchio within Colab/Binder with Meshcat viewer. Did you manage to integrate the solution into Drake framework?
With @nim65s we can provide some force task on it.

@duburcqa Did you manage to also provide this portage within your framework?

[RussTedrake]

@jcarpent -- see also #13038, which is now almost complete. There are a lot of details and moving part that I could help you understand if you like.
Long story short:

• I've been using Meshcat successfully on Colab for a long time now; it can work quite well. This remaining issue was only about embedding colab into the actual notebook as a iframe. I have not gotten it working, mostly because most people actually prefer to have meshcat in a separate window anyhow.
• The primary challenge of putting meshcat into an iframe in the notebook is the combination of requirements that the notebook iframes require https/wss sources, but I currently use ngrok to tunnel http/ws out from the server. I've added wss support into meshcat, but the free ngrok won't support https/wss and their pricing plans don't make sense for this cloud workflow. If you hosted your own tunnel you could get around it.
• I'm actually now likely transitioning away from Colab. It's very generous in terms of compute resources, but it is not a stable environment to support. (the libraries and even python versions can change out from under you with no notice and no recourse). Port from colab to deepnote RussTedrake/manipulation#144
• I've implemented the versions of meshcat that work only through kernel comms, but Colab / Deepnote / etc all offer their own implementations (if at all) of the kernel comms, so writing one clean meshcat implementation that works for all of them is likely impossible.

[jcarpent]

Dear @RussTedrake,

Thanks a lot for the clear and detailed answer.
Having your feedback on Colab will for sure help us to not stuck into the same local minima.
I did not know about Deepnote, I will have a look into it.

I'm also following #13038: this is a very interesting feature that may be useful to a lot of people.
Could we imagine transferring it into an independent package meshcat-cpp? I may give some help on that and I know for instance @traversaro is also motivated.

Thanks again @RussTedrake for your very helpful feedback.

cc: @nim65s @proyan @ManifoldFR

[RussTedrake]

I definitely considered meshcat-cpp, but decided against in the first pass to streamline things. It currently depends on Drake's geometry types and rigid transform type, (plus build system / CI / helpers for memory/exception management, etc), as I didn't want to have to reinvent them for a more general implementation.

[jcarpent]

Perfect. I totally understand.
Do not hesitate to ping us if you need support.

[nim65s]

Hello,

I tried to embed meshcat in a jupyter cell on binder: https://github.com/nim65s/binder-meshcat-pinocchio-test.
You can try it at: https://mybinder.org/v2/gh/nim65s/binder-meshcat-pinocchio-test/main
It looks like it is working, but this probably need more tests, especially about speed.

It is mostly a copy-paste from meshcat-dev/meshcat-python#74, with the following changes:

1. removed the iframe. I'm not sure why it was needed in the first place. Did I miss something ? We can put it back if needed.
2. split meshcat inclusion into HTML() & Javascript()
3. host meshcat somewhere in HTTPS with Access-Control-Allow-Origin: *
4. use requirejs to get it (requirejs is already embedded in jupyter notebooks)
5. initialize communication from the client instead of the server (I was trying to ensure that if the communication is not properly established, the python part will notice)
6. removed parts related to Google Colab
7. instead of sending all the meshes by the communication bus, I hosted them somewhere in HTTPS, and I'm sending only an url. After that, they are fetched asynchronously on the client side: nim65s/meshcat@db88112

A few comments about that:

• Meshcat in COLAB duburcqa/jiminy#403 raised rate limit issues. I also managed to get them locally, so I implemented (3) & (7) to help with that.
• For this test, I'm using a small personal server for (3) & (7), but I think we should consider using a CDN for this kind of things, at least for (3). Anyway, this server is not a permanent one, it's just here for a few tests
• (6) can obviously be reverted, if we want this thing to work on multiple jupyter notebook providers. I didn't read the whole thread, but I guess there are stronger issues with it.
• my tests are tied to pinocchio and its MeshcatViewer abstraction, but we can easily extract the more generic parts

I'd be glad to discuss a bit about this, before going further and opening PRs ; so if you have any comment or subjection, please don't hesitate :)

[jwnimmer-tri]

For reference / archival or if anyone is interested --

Here's some code that I've used in Deepnote. It automatically opens up a Meshcat display in a new browser tab.

import meshcat
import os
from IPython.display import HTML, Javascript, display

with open(os.path.dirname(meshcat.__file__) + '/viewer/dist/main.min.js', 'r') as f:
    main_min_js = f.read()

def js_esc(data):
    bs = '\\'
    return data.replace(bs, bs*2).replace('\n', bs + '\n').replace("'", bs + "'")

body = f"""
<div id="meshcat-pane" style="height: 100%; width: 100%; overflow-x: auto; overflow-y: hidden; resize: both"></div>
"""
title = f"""
MeshCat
"""
mine = """
document.meshcatViewer = new MeshCat.Viewer(document.getElementById("meshcat-pane"));
"""
js = main_min_js + ";" + mine
escaped_body = body.replace("\n", "\\n")
js = f"""
var newWindow = window.open("", "meshcat");
newWindow.document.title = '{js_esc(title)}';
newWindow.document.body.innerHTML = '{js_esc(body)}';
var script = newWindow.document.createElement('script');
script.text = '{js_esc(js)}';
newWindow.document.head.appendChild(script);
window.document.meshcatViewer = newWindow.document.meshcatViewer;
console.log("MeshCat is ready!");
"""
display(Javascript(js))

This allows for opening the window without needing an https server for the HTML code. It also allows code in notebook cells to inject Javascript code into the meshcat tab.

It does not resolve any problems related to the meshcat websocket messages; those would still need to come in via a wss socket.

[RussTedrake]

I'm going to close this as "Won't fix", since nobody is asking for it currently. We can reopen it if/when there is interest.