XR + Robotics = New Era
XR is the ultimate interface for robotics control in the real world
Intro - limits and opportunities
Robotic systems today are typically pre programmed with traditional computers. They are managed and operated by technicians working with laptops or desktops, perhaps tablets. The robotic systems are often pre-trained in simulated environments.
This is often a frustrating trial and error process that leaves the human supervisors often unable to completely understand the robot’s inner state at any given time. Yet XR offers us a way to make virtual state of robots into visible graphics mixing in the reality, and XR gives us the opportunity to go into the virtual training grounds as embodied participants. XR opens up an entirely new frontier for natural human/robot interaction.
Robots & human training / environment exploration today
Companies like Agility Robotics train its humanoid robot Digit in simulation using NVIDIA's robotics tools.
Source: The Robot Report
https://www.therobotreport.com/how-agility-robotics-crosses-the-sim2real-gap-with-nvidia-isaac-lab/
Meta has recently announced their work on XR control of robotics. So I think the space is heating up.
My Past work - 2019 onwards
Digital twin control and visualization. Direct manipulation of virtual robotics, this is the simplest demo possible. Even in this simple demo, we get to see the past, present and future states of the robotic arm directly superimposed on the reality itself. This makes operating the robot and understanding its next moves intuitive and direct.
I got to test this demo at the Star Trek convention (fitting lol) and found that laypeople could easily control the robot’s very basic functions.
Vision - robotic control using XR mediation brings us new powers
In my AWE 2024 talk I covered XR work for Nanome and Robotic experiments along with a vision of why I think robotics and XR are made for each other.
At a FeedbackXR session hosted by Iggy Love I went into it more and folks can ask questions. I think I should do another one soon.
Exploring the concept of robots helping people and learning from them in a shared context, I created this little video of asking a robot to identify plants and confirming its accuracy. Model Robot Crab by Zerg on Sketchfab
I see XR as able to break down the barriers of prior training and possible failures in the field, and instead working with robots becomes an enriching experience for both person and machine.
Robotics are not just for manufacturing and work, they also represent a whole new opportunity space for medicine and first response. The company AvatarMEDIC was founded to explore the possibilities for new tech in medicine. This longer talk from our XPRIZE challenge entry goes over the goals of the research and the experiments done.
Bret Victor’s work with Seeing Spaces is a major inspiration and I recommend everyone explore it and his thinking about what is needed to create a next generation science space.
I want to knock the walls down and use XR headsets to enable us to do new science, art, making and learning out in the world wherever we are.
Robotics and XR could unlock the powers of both, and if we are willing and brave, help us build a more empowering world.