Key ideas
- FLEX combines natural-language instructions with a GPT-4o generalization step and an update to the object list based on the vision system.
- MIT describes tests in simulations of three tasks with a robotic arm: serving coffee, moving a candle, and carrying a bowling ball.
- The reported results are from simulated scenarios and do not establish FLEX’s performance in homes or with physical robots.
Interpreting more than the literal command
An instruction such as “Don’t spill coffee on the laptop” can express a broader precaution than avoiding a single object. FLEX, a platform presented by researchers from MIT and New York University, aims to enable a robot to draw these kinds of generalizations from everyday instructions and use them when planning its movements.
According to MIT, the study will be presented this month at the annual Conference on Robot Learning. The team demonstrated FLEX in simulated scenarios: people typed natural-language instructions while a virtual robotic arm moved objects. In one example, the robot pours coffee onto a table with several objects and is told not to spill it on a laptop; the system also changes its trajectory to avoid a nearby phone.
The platform aims to reduce the need for a person to name each object individually. Rather than simply adding an item to the list of relevant obstacles, FLEX tries to infer a broader category from the user’s corrections. The published information describes the method and its demonstrations, but does not establish how it would behave in real-world situations.
Three steps to update the list
FLEX organizes the process into three steps: Add, Generalize, and Mask. In Add, the robot adds an item identified by the person to its list. For example, if someone says “Don’t drop it on my mug” while the robot is carrying a bowling ball, the system adds the mug to the objects it must avoid.
In Generalize, the system expands the precaution. MIT’s description says that FLEX uses the GPT-4o language model to relate the items identified by the user and propose a broader category. In the mug example, the generalization is to avoid fragile objects, even though the person has not listed every object in that class. In this way, the platform combines the list of features relevant to the task with associations generated by the model.
The third step, Mask, uses the robot’s computer vision system to determine which objects are no longer present and remove them from the list. MIT says the entire process takes about five seconds. The description does not provide other performance measures, such as the frequency of successful inferences or the errors observed.
| FLEX stage | Described function | Source example |
|---|---|---|
| Add | Adds an object identified in the instruction to the list. | Add a mug to the objects the robot must avoid. |
| Generalize | Uses GPT-4o to infer a broader category from the identified items. | Extend the precaution to fragile objects. |
| Mask | The vision system removes objects that are no longer present from the list. | Update the list based on the detected environment. |
What the virtual tests showed
The researchers tested FLEX in simulations of three tasks with a robotic arm: serving a cup of coffee, moving a candle, and carrying a bowling ball. People typed commands such as “Don’t spill on my laptop,” “Stay away from the stack of paper,” and “Don’t drop it on the mug.” According to MIT, the robot followed each instruction and also generalized by avoiding electronic devices, flammable objects, and fragile items, respectively.
These examples show the method’s purpose in virtual environments, but do not establish how it would work in real-world situations, with other robots, or in response to different instructions. The source describes three simulated tasks and does not report tests with users in homes, quantitative comparisons with other methods, or error rates.
Team, funding, and scope
MIT identifies Andreea Bobu, an assistant professor of aeronautics and astronautics at the institution, as the study’s lead. Among the coauthors, it names Nathan Dennler, a postdoctoral researcher at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), and Audrey Lee and Ilia Sucholutsky, of New York University. The source says that the work received partial support from the MIT Generative AI Impact Consortium and the Social and Ethical Responsibilities of Computing program at MIT Schwarzman College of Computing.
The demonstration does not allow conclusions about how FLEX would respond to conflicting preferences or ambiguous instructions, or about its performance outside simulations. These questions remain open in the published information.
The NSF references included here provide context: one explains how the agency supports research, and another describes materials research programs in its Directorate for Mathematical and Physical Sciences. They do not independently corroborate FLEX’s results, which MIT attributes to its team’s study. NSF: how it supports research · NSF: materials research programs in the Directorate for Mathematical and Physical Sciences
References
What we know and its limits
The available information describes simulations of three tasks and does not provide trials in physical environments, quantitative performance results, or error rates. The NSF references provide context and do not independently corroborate the study attributed to MIT.
Original sources3
The original sources for exploring this explanation and its limits.
- Studies and Results: How NSF Supports Research NSF
-
How research into materials is organized and what it produces
NSF
Institutional reference for context. It does not constitute independent confirmation of the main source's findings. - AI-assisted learning platform helps robots reason beyond literal commands MIT
Image credits
Detailed view of a cutting-edge industrial robotic arm in action. · Photo by Freek Wolsink on Pexels · Original source · Pexels License · Resized and re-encoded as JPEG; full framing preserved.