Understanding one's own self extends to machines: A vision-based system is developed to enable robots to comprehend their physical forms
In a groundbreaking development, researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have created a new system called Neural Jacobian Fields (NJF) that enables robots to learn their own body control and motion using only visual data, without the need for embedded sensors or detailed prior models.
The system builds on neural radiance fields (NeRF), a technique that reconstructs 3D scenes from images. NJF extends this approach by learning not only the robot's shape, but also a Jacobian field, a function that predicts how any point on the robot's body moves in response to motor commands.
Robotic Self-Awareness and Control
NJF allows robots to develop an internal model of their 3D shape and how different parts move in response to commands through self-supervised learning from video observations. This visual self-modeling enables closed-loop control purely from monocular camera input at real-time speeds (~12 Hz), facilitating autonomous and adaptive motion control.
Soft Robotics
Soft robots, which are safe and adaptable but traditionally difficult to control due to their flexible, deformable bodies, benefit greatly from NJF. The technique predicts how soft structures respond to motor signals from visual feedback alone, overcoming challenges posed by floppy actuators or unpredictable dynamics.
Robotics in Dynamic, Unstructured Environments
Because NJF requires only a camera and no embedded sensors, it better equips robots to operate in complex settings such as construction sites, agricultural fields, or confined spaces where conditions change and sensor installation would be complex or costly. The visual approach enables robots to generalize their control despite occlusions, external loads, or imperfect hardware.
Lowering Barriers to Robotic Automation
NJF reduces reliance on human-engineered kinematic models, physical sensors, or laborious calibration. This democratizes robot design and deployment, broadening the design space to include soft or low-cost robots previously considered too difficult to control accurately.
Robotic Manipulation and Interaction
NJF enables precise control of complex manipulators, such as soft pneumatic hands or 3D-printed arms, by learning how control inputs translate to motion, potentially enhancing robot dexterity in both industrial and delicate tasks like surgery.
Overall, Neural Jacobian Fields pave the way for more adaptable, sensor-efficient, and visually supervised robots capable of robust operation in unstructured, dynamic environments across diverse applications.
In early simulations, simple 2D fingers and sliders were able to learn the mapping of control commands using just a few examples. The NJF system figures out which motors control which parts of the robot through learning, without any programming.
The motivation for the development of NJF is to expand the design space for robotics, particularly for soft and bio-inspired robots. The researchers envision a future where hobbyists can create a control model for a robot using footage from their phone, with no prior knowledge or special equipment required.
The system has proven robust across a range of robot types, including soft robotic hands, rigid hands, 3D-printed robotic arms, and even a rotating platform with no embedded sensors. However, the system doesn't yet generalize across different robots and lacks force or tactile sensing, limiting its effectiveness on contact-rich tasks.
An open-access paper about the work was published in Nature on June 25. The capability of controlling the robotic hand comes from a new system developed by CSAIL scientists. The goal of Neural Jacobian Fields is to lower the barrier for robotics, making it affordable, adaptable, and accessible to more people.
The research was supported by the Solomon Buchsbaum Research Fund, MIT Presidential Fellowship, National Science Foundation, and the Gwangju Institute of Science and Technology. NJF builds a dense map of controllability by modeling how specific points deform or shift in response to action. The core of NJF is a neural network that captures a robot's three-dimensional geometry and its sensitivity to control inputs. Vision is seen as a resilient, reliable sensor, opening the door to robots that can operate in messy, unstructured environments without expensive infrastructure.
- The groundbreaking Neural Jacobian Fields (NJF) system developed by MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) enables robots to learn their body control and motion using visual data alone, without the need for embedded sensors or detailed prior models.
- NJF extends the approach of neural radiance fields (NeRF) by not only learning the robot's shape but also a Jacobian field, a function that predicts how any point on the robot's body moves in response to motor commands.
- With NJF, robots can develop an internal model of their 3D shape and how different parts move in response to commands, making it possible for them to have closed-loop control purely from monocular camera input at real-time speeds.
- NJF's visual self-modeling enables robots to operate in complex, dynamic environments such as construction sites, agricultural fields, or confined spaces, where conditions change and sensor installation would be complex or costly.
- The system allows soft robots, which are difficult to control due to their flexible, deformable bodies, to be controlled more effectively by predicting how soft structures respond to motor signals from visual feedback alone.
- The development of Neural Jacobian Fields aims to lower the barrier for robotics, making it affordable, adaptable, and accessible to a broader range of people, potentially even allowing hobbyists to create control models for robots using footage from their smartphones.
- The research behind NJF has been supported by funding from the Solomon Buchsbaum Research Fund, MIT Presidential Fellowship, National Science Foundation, and the Gwangju Institute of Science and Technology.
- The NJF system is a component in the broader field of artificial intelligence and technology, contributing to advancements in robotics, engineering, physics, and computer science, and opening up new possibilities for learning, research, and teaching in these areas.
