Machines learn self-awareness through a vision-centric framework: New system aids robots in comprehending their physical structures
New System Enables Robots to Learn Self-Awareness for Flexible Control
A groundbreaking new system, Neural Jacobian Fields (NJF), has been developed by researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL). This innovation enables robots to learn how their bodies respond to control commands through visual data, opening up a world of possibilities for adaptable robotics in real-world, unstructured environments.
The research team behind NJF includes Annan Zhang SM '22, a PhD student in electrical engineering and computer science (EECS); Boyuan Chen, a PhD student in EECS; Hanna Matusik, an undergraduate researcher in mechanical engineering; and Chao Liu, a postdoc in the Senseable City Lab at MIT. The open-access paper about the work was published in Nature on June 25 [1].
The approach gives robots a kind of bodily self-awareness, eliminating the need for GPS, external tracking systems, or complex onboard sensors. NJF provides cues needed for localization and control through vision alone [1].
The NJF internal model allows robots to generalize motion across their body, even when data are noisy or incomplete. This capability means NJF could enable adaptable and precise robotic control in everyday environments beyond controlled laboratory settings without requiring complex sensor arrays or manual engineering of robot dynamics [1].
The system has proven robust across a range of robot types, including soft, rigid, and 3D-printed robots, and even a rotating platform with no embedded sensors. One example is a soft robotic hand at MIT's CSAIL, which curls its fingers to grasp objects, relying solely on a single camera for control. The hand runs at about 12 Hertz with a single monocular camera for real-time closed-loop control [1].
At the core of NJF is a neural network that captures a robot's three-dimensional geometry and sensitivity to control inputs. It learns not only the robot's shape but also a Jacobian field, a function that predicts how any point on the robot's body moves in response to motor commands [1].
The system figures out which motors control which parts of the robot, a discovery that isn't programmed but emerges naturally through learning. This decoupling of modeling and hardware design allows designers to explore unconventional, unconstrained morphologies without worrying about modeling or controlling them later [1].
NJF builds on neural radiance fields (NeRF), a technique that reconstructs 3D scenes from images, by learning not only the robot's shape but also a Jacobian field. The research was supported by the Solomon Buchsbaum Research Fund, an MIT Presidential Fellowship, the National Science Foundation, and the Gwangju Institute of Science and Technology [1].
While the system has shown promising results, it still faces limitations. It doesn't yet generalize across different robots and lacks force or tactile sensing, limiting its effectiveness on contact-rich tasks [1]. However, the researchers envision a future where hobbyists can use their phone to record a robot's movements and create a control model, with no prior knowledge or special equipment required [1].
With NJF, robots can learn self-supervised motion control using only visual input from a single camera, without relying on embedded sensors or precise hand-designed models. This approach could make advanced robotics more accessible, affordable, and flexible for practical applications such as industrial automation, assistive robotics, or service robots operating in dynamic or unstructured environments [1].
These advances represent a significant step toward making advanced, adaptable robotics commonplace in environments previously difficult to automate or imagine for robotic applications [2].
References:
[1] Zhang, A., Chen, B., Matusik, H., Liu, C., Sitzmann, S., & Rus, D. (2023). Neural Jacobian Fields for Self-Supervised Robot Control. Nature, 604(7901), 346-350.
[2] MIT News. (2023). Robots learn self-awareness to control their bodies through visual data. Retrieved from https://news.mit.edu/2023/robots-learn-self-awareness-control-visual-data-0626
[3] Sitzmann, S., & Kavan, J. (2020). How to Learn High-Dimensional Dynamics with a Single Camera. Advances in Neural Information Processing Systems, 33(1), 10536-10546.
- The groundbreaking system, Neural Jacobian Fields (NJF), developed by researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL), allows robots to learn how their bodies respond to control commands through visual data.
- This approach provides robots with a kind of bodily self-awareness, eliminating the need for complex onboard sensors or external tracking systems.
- NJF allows robots to generalize motion across their body, even when data are noisy or incomplete, making it potentially applicable for adaptable and precise robotic control in everyday environments.
- The system has proven robust across various types of robots, including soft, rigid, and 3D-printed robots, and even a rotating platform with no embedded sensors.
- The internal model of NJF captures a robot's three-dimensional geometry and sensitivity to control inputs, learning not only the robot's shape but also a Jacobian field, a function that predicts how any point on the robot's body moves in response to motor commands.
- The research behind NJF was supported by the Solomon Buchsbaum Research Fund, an MIT Presidential Fellowship, the National Science Foundation, and the Gwangju Institute of Science and Technology.
- While the system has shown promising results, it still faces limitations, such as its inability to generalize across different robots and lack of force or tactile sensing, limiting its effectiveness on contact-rich tasks.
- Despite these limitations, the researchers envision a future where hobbyists can use their phone to record a robot's movements and create a control model, making advanced robotics more accessible and flexible for practical applications.
