For all experiments we utilize a Franka Emika Panda research robot. The only modification we made in terms of hardware is using a Robotiq 2F-85 gripper, as we found the gripper with the Franka was prone to failure when running longer RL experiments (1 hour+).
Before continuing, ensure that your Franka is setup with it's joints unlocked and is in the 'Ready' state for recieving movement commands for a controller. For instructions on how to do so for your setup, follow the instructions here.
We use the polymetis environment wrapper to work with our Franka robot, with the instructions to download found here. We thank the authors for nice codebase / wrapper to work with!
Our workflow assumes a controller computer. For our use case we used an intel NUC machine as recommended by the polymetis instructions. I
We recommend the local installation path option for polymetis has using conda forge caused us compilation issues.
As of this commit you will also need to also install dm_control (latest verison) and dm_robotics package to use the IK solver
pip install dm-robotics-moma
After installation, add the directory to the PYTHONPATH:
export PYTHONPATH=$PYTHONPATH:~/iris_robots/
The RobotEnv is a OpenAI gym-style interface with the polymetis stack under the hood.
If running on NUC itself, FrankaRobot will be instantiated. Otherwise, the RobotInterface
will be used. The documentation assumes the use of polymetis-local.
First, start a server to communicate with the robot on the NUC. Note, this requires a local installation of polymetis.
tmux new-session
sudo pkill -9 run_server
conda activate polymetis-local
bash ~/fairo/polymetis/polymetis/python/scripts/persist_server.sh
Then, create a new window in tmux and activate the gripper. The terminal should prompt that the robotiq gripper is activated. If you get an error, try once more and that should work.
sudo chmod a+rw /dev/ttyUSB0
conda activate polymetis-local
launch_gripper.py gripper=robotiq_2f gripper.comport=/dev/ttyUSB0
Create a new window and run the flask server, which communicates between the nuc / workstation.
conda activate polymetis-local
python ~/iris_robots/run_server.py
All instructions onwards assume you have activated the polymetis-local conda env. For demo collection, we utilize an Xbox One controller which is directly plugged into the NUC controller computer.
You can find one here. If you would like to get a sense of the commands, try running
python free_xbox_control.py
You can also modify the robot_env to use either 4-DoF or 5-DoF control, with an optional degree of freedom in rotating the orientation of the gripper up to 180 degrees. The commands are
'A': Close gripper. Must be held to continue closing gripper, and once released the gripper will open
'X': End demo collection episode early.
'Left stick (right to left)' : Move end-effector along y-axis with a delta depending on how far the stick is moved
'Left stick (upwards to downwards)' : Move end-effector along x-axis with a delta depending on how far the stick is moved
'Right trigger': Move end-effector upwards along z-axis with a delta depending on how much the trigger is pressed
'Left trigger': Move end-effector downwards along z-axis with a delta depending on how much the trigger is pressed
'Right stick (right to left)': Rotate end-effector clockwise/counter-clockwise depending on which direction the stick is moved
only used for 5 DoF control
The demo collection script used for the main experiments can be run by
python demo_collect_fb.py demos
Which will create a folder named 'demos' where all demo data and GIFs of data collection are stored. If a folder with the same name exists before, the script first prompts the user with the following text:
A demo file already exists. Continue appending to it? (y) or (n):
Be careful that you do NOT enter n on your keyboard if you wish to avoid overwriting your demos!
Afterwards demo collection will begin and you can control the robot with the Xbox controller for
a limited number of timesteps, which we configured per task but which can be changed in the robot_env.py
file. If you wish to end the demo early, or you reach the end of the episode, data collection will stop and
you will recieve the following prompt:
Save current episode?: (f) for forward buffer, (b) for backward buffer and (d) to discard episode and (q) to save and quit:
Enter either f or b depending on whether the demo you collected is for the forward / backward buffer as stated above. If you
would like another prompt which waits for you to start the next episode with a keyboard entry enter fj or bj. If you enter
d or dj you will discard the demo either directly continuing into the start of collecting the next demo or pausing before continuing.
If you do not enter either f or b then no demo will be saved. If you would like to save and quit you must enter fq or bq.
Running the launch robot command uses the yaml file at
{$HOME}/fairo/polymetis/polymetis/conf/launch_robot.yaml
Also gives different options for choice of robot client.
The file that determines the number of allowed automatic error recovery
attempts is at
fairo/polymetis/polymetis/src/clients/franka_panda_client/franka_panda_client.cpp
And the constants used for all of this are here
fairo/polymetis/polymetis/include/polymetis/clients/franka_panda_client.hpp
As for the server warning themselves, the warning for exceed the threshold of
force is logged here
fairo/polymetis/polymetis/src/polymetis_server.cpp
If when running the server, you see the following or similar error
dm_control/mjcf/physics.py", line 495, in from_mjcf_model
return cls.from_xml_string(xml_string=xml_string, assets=assets)
File "/home/panda5/anaconda3/envs/polymetis-local/lib/python3.8/site-packages/dm_control/mujoco/engine.py", line 424, in from_xml_string
return cls.from_model(model)
File "/home/panda5/anaconda3/envs/polymetis-local/lib/python3.8/site-packages/dm_control/mujoco/engine.py", line 407, in from_model
return cls(data)
File "/home/panda5/anaconda3/envs/polymetis-local/lib/python3.8/site-packages/dm_control/mujoco/engine.py", line 122, in __init__
self._reload_from_data(data)
File "/home/panda5/anaconda3/envs/polymetis-local/lib/python3.8/site-packages/dm_control/mjcf/physics.py", line 530, in _reload_from_data
super()._reload_from_data(data)
File "/home/panda5/anaconda3/envs/polymetis-local/lib/python3.8/site-packages/dm_control/mujoco/engine.py", line 370, in _reload_from_data
self._warnings_before = np.empty_like(self._warnings)
File "<__array_function__ internals>", line 180, in empty_like
RuntimeError: Caught an unknown exception!
There is a mismatch between the versions installed by the local version of polymetis and the versions you need
pip install mujoco==2.2.1
pip install dm-control==1.0.5
TypeError: bases must be types
to get around it uninstall then install earlier version of protobuf
pip uninstall protobuf
pip install protobuf==3.20.1
If you are using local polymetis to control the robot, make sure you are not running a server. The server would have taken over the cameras and for security reasons the OS will not let the local env cannot use them.
If no server is running and the cameras are still not detected, restart the NUC
If the robotiq server results in this issue: facebookresearch/fairo#1383 Kill the tmux session and restart. Otherwise restart the NUC
If the robot enters user stop mode, the polymetis-server will crash unfortunately. If the robot needs a manual reset, make sure the training script is halted, then perform the manual reset and ensure that a new polymetis server is running before continuing.