• Home
  • Insight
  • Blog
  • Business
  • Entertainment
  • Health
  • Politics
  • Shop
    • Gift Shop
    • Value Shop
    • Store
    • Bargain Shop
    • Discount
  • Sports
  • Tech
  • Travel
  • USA
  • Video
  • World
    • Asia
    • Africa
    • South America
    • North America
    • Europe
    • Oceania
Thursday, November 19, 2026
No Result
View All Result
Subscribe Now
  • Home
  • Insight
  • Blog
  • Business
  • Entertainment
  • Health
  • Politics
  • Shop
    • Gift Shop
    • Value Shop
    • Store
    • Bargain Shop
    • Discount
  • Sports
  • Tech
  • Travel
  • USA
    FAA Administrator Takes Responsibility as Mullin Faces Tough Questions Over Nationwide Ground Stop

    FAA Administrator Takes Responsibility as Mullin Faces Tough Questions Over Nationwide Ground Stop

    Mary Peltola Emerges Victorious in Alaska Senate Race, Making History for Democrats

    Mary Peltola Emerges Victorious in Alaska Senate Race, Making History for Democrats

    Stranded in Conflict: Palestinians Navigate the Tenuous Balance of War and Peace Amid Ceasefire Efforts

    Stranded in Conflict: Palestinians Navigate the Tenuous Balance of War and Peace Amid Ceasefire Efforts

    Trump Administration Revokes Pay Increase for Head Start Educators

    Trump Administration Revokes Pay Increase for Head Start Educators

    Unexpected Triumph: Gay Candidates Achieve Unforeseen Victories in Challenging Races

    Unexpected Triumph: Gay Candidates Achieve Unforeseen Victories in Challenging Races

    Microsoft Outlook Experiences Prolonged Outage Affecting Users Nationwide

    Microsoft Outlook Experiences Prolonged Outage Affecting Users Nationwide

    Gunfire Erupts in D.C.: Authorities Search for Suspect Amid Rising Concerns Over Public Safety

    Gunfire Erupts in D.C.: Authorities Search for Suspect Amid Rising Concerns Over Public Safety

    LAUSD Takes Strong Action Against Hate Speech Following Racial Slur Incidents

    LAUSD Takes Strong Action Against Hate Speech Following Racial Slur Incidents

    Beloved Former Los Angeles Times Columnist Steve Harvey Passes Away at 80

    Beloved Former Los Angeles Times Columnist Steve Harvey Passes Away at 80

  • Video
  • World
    • Asia
    • Africa
    • South America
    • North America
    • Europe
    • Oceania
The Insight Post
  • Home
  • Insight
  • Blog
  • Business
  • Entertainment
  • Health
  • Politics
  • Shop
    • Gift Shop
    • Value Shop
    • Store
    • Bargain Shop
    • Discount
  • Sports
  • Tech
  • Travel
  • USA
    FAA Administrator Takes Responsibility as Mullin Faces Tough Questions Over Nationwide Ground Stop

    FAA Administrator Takes Responsibility as Mullin Faces Tough Questions Over Nationwide Ground Stop

    Mary Peltola Emerges Victorious in Alaska Senate Race, Making History for Democrats

    Mary Peltola Emerges Victorious in Alaska Senate Race, Making History for Democrats

    Stranded in Conflict: Palestinians Navigate the Tenuous Balance of War and Peace Amid Ceasefire Efforts

    Stranded in Conflict: Palestinians Navigate the Tenuous Balance of War and Peace Amid Ceasefire Efforts

    Trump Administration Revokes Pay Increase for Head Start Educators

    Trump Administration Revokes Pay Increase for Head Start Educators

    Unexpected Triumph: Gay Candidates Achieve Unforeseen Victories in Challenging Races

    Unexpected Triumph: Gay Candidates Achieve Unforeseen Victories in Challenging Races

    Microsoft Outlook Experiences Prolonged Outage Affecting Users Nationwide

    Microsoft Outlook Experiences Prolonged Outage Affecting Users Nationwide

    Gunfire Erupts in D.C.: Authorities Search for Suspect Amid Rising Concerns Over Public Safety

    Gunfire Erupts in D.C.: Authorities Search for Suspect Amid Rising Concerns Over Public Safety

    LAUSD Takes Strong Action Against Hate Speech Following Racial Slur Incidents

    LAUSD Takes Strong Action Against Hate Speech Following Racial Slur Incidents

    Beloved Former Los Angeles Times Columnist Steve Harvey Passes Away at 80

    Beloved Former Los Angeles Times Columnist Steve Harvey Passes Away at 80

  • Video
  • World
    • Asia
    • Africa
    • South America
    • North America
    • Europe
    • Oceania
No Result
View All Result
No Result
View All Result
Home Science

Researchers use large language models to help robots navigate

by Theinsightpost
June 15, 2024
in Science
0 0
0
Researchers use large language models to help robots navigate

Someday, you may want your home robot to carry a load of dirty clothes downstairs and deposit them in the washing machine in the far-left corner of the basement. The robot will need to combine your instructions with its visual observations to determine the steps it should take to complete this task.

For an AI agent, this is easier said than done. Current approaches often utilize multiple hand-crafted machine-learning models to tackle different parts of the task, which require a great deal of human effort and expertise to build. These methods, which use visual representations to directly make navigation decisions, demand massive amounts of visual data for training, which are often hard to come by.

To overcome these challenges, researchers from MIT and the MIT-IBM Watson AI Lab devised a navigation method that converts visual representations into pieces of language, which are then fed into one large language model that achieves all parts of the multistep navigation task.

Rather than encoding visual features from images of a robot’s surroundings as visual representations, which is computationally intensive, their method creates text captions that describe the robot’s point-of-view. A large language model uses the captions to predict the actions a robot should take to fulfill a user’s language-based instructions.

Because their method utilizes purely language-based representations, they can use a large language model to efficiently generate a huge amount of synthetic training data.

While this approach does not outperform techniques that use visual features, it performs well in situations that lack enough visual data for training. The researchers found that combining their language-based inputs with visual signals leads to better navigation performance.

“By purely using language as the perceptual representation, ours is a more straightforward approach. Since all the inputs can be encoded as language, we can generate a human-understandable trajectory,” says Bowen Pan, an electrical engineering and computer science (EECS) graduate student and lead author of a paper on this approach.

Pan’s co-authors include his advisor, Aude Oliva, director of strategic industry engagement at the MIT Schwarzman College of Computing, MIT director of the MIT-IBM Watson AI Lab, and a senior research scientist in the Computer Science and Artificial Intelligence Laboratory (CSAIL); Philip Isola, an associate professor of EECS and a member of CSAIL; senior author Yoon Kim, an assistant professor of EECS and a member of CSAIL; and others at the MIT-IBM Watson AI Lab and Dartmouth College. The research will be presented at the Conference of the North American Chapter of the Association for Computational Linguistics.

Solving a vision problem with language

Since large language models are the most powerful machine-learning models available, the researchers sought to incorporate them into the complex task known as vision-and-language navigation, Pan says.

But such models take text-based inputs and can’t process visual data from a robot’s camera. So, the team needed to find a way to use language instead.

Their technique utilizes a simple captioning model to obtain text descriptions of a robot’s visual observations. These captions are combined with language-based instructions and fed into a large language model, which decides what navigation step the robot should take next.

The large language model outputs a caption of the scene the robot should see after completing that step. This is used to update the trajectory history so the robot can keep track of where it has been.

The model repeats these processes to generate a trajectory that guides the robot to its goal, one step at a time.

To streamline the process, the researchers designed templates so observation information is presented to the model in a standard form — as a series of choices the robot can make based on its surroundings.

For instance, a caption might say “to your 30-degree left is a door with a potted plant beside it, to your back is a small office with a desk and a computer,” etc. The model chooses whether the robot should move toward the door or the office.

“One of the biggest challenges was figuring out how to encode this kind of information into language in a proper way to make the agent understand what the task is and how they should respond,” Pan says.

Advantages of language

When they tested this approach, while it could not outperform vision-based techniques, they found that it offered several advantages.

First, because text requires fewer computational resources to synthesize than complex image data, their method can be used to rapidly generate synthetic training data. In one test, they generated 10,000 synthetic trajectories based on 10 real-world, visual trajectories.

The technique can also bridge the gap that can prevent an agent trained with a simulated environment from performing well in the real world. This gap often occurs because computer-generated images can appear quite different from real-world scenes due to elements like lighting or color. But language that describes a synthetic versus a real image would be much harder to tell apart, Pan says.

Also, the representations their model uses are easier for a human to understand because they are written in natural language.

“If the agent fails to reach its goal, we can more easily determine where it failed and why it failed. Maybe the history information is not clear enough or the observation ignores some important details,” Pan says.

In addition, their method could be applied more easily to varied tasks and environments because it uses only one type of input. As long as data can be encoded as language, they can use the same model without making any modifications.

But one disadvantage is that their method naturally loses some information that would be captured by vision-based models, such as depth information.

However, the researchers were surprised to see that combining language-based representations with vision-based methods improves an agent’s ability to navigate.

“Maybe this means that language can capture some higher-level information than cannot be captured with pure vision features,” he says.

This is one area the researchers want to continue exploring. They also want to develop a navigation-oriented captioner that could boost the method’s performance. In addition, they want to probe the ability of large language models to exhibit spatial awareness and see how this could aid language-based navigation.

This research is funded, in part, by the MIT-IBM Watson AI Lab.

ShareTweetSend
Previous Post

Zelenskyy dismisses Putin ceasefire offer and says he’s on ‘same path’ as Hitler | World News

Next Post

World’s 50 most dangerous cities – and two are in the UK | World | News

Related News

Discover the Fascinating Secrets of NASA’s Voyager Mission: Test Your Knowledge!
Science

Discover the Fascinating Secrets of NASA’s Voyager Mission: Test Your Knowledge!

November 19, 2026
Groundbreaking Electric Comb Promises Hair Regrowth in Mice Within Four Weeks, Paving the Way for Future Hair-Loss Solutions
Science

Groundbreaking Electric Comb Promises Hair Regrowth in Mice Within Four Weeks, Paving the Way for Future Hair-Loss Solutions

November 18, 2026
Miniature LSD Doses Administered in Rapid Succession Show Antidepressant Potential in Mice: Exploring the Impact of Timing vs. Tolerance
Science

Miniature LSD Doses Administered in Rapid Succession Show Antidepressant Potential in Mice: Exploring the Impact of Timing vs. Tolerance

November 14, 2026
Investigation Underway: Experts Seek Answers Behind Sudden Brooklyn Building Collapse
Science

Investigation Underway: Experts Seek Answers Behind Sudden Brooklyn Building Collapse

November 12, 2026
Next Post
World’s 50 most dangerous cities – and two are in the UK | World | News

World’s 50 most dangerous cities - and two are in the UK | World | News

Discussion about this post

Subscribe To Our Newsletters

    Customer Support


    1251 Wilcrest Drive
    Houston, Texas
    77042 USA
    Call-832.795.1420
    e-mail – news@theinsightpost.com

    Subscribe To Our Newsletters

      Categories

      • Africa
      • Africa-East
      • African Sports
      • American Sports
      • Arts
      • Asia
      • Australia
      • Business
      • Business Asia
      • Business- Africa
      • Canada
      • Defense
      • Education
      • Egypt
      • Energy
      • Entertainment
      • Europe
      • European Soccer
      • Finance
      • Germany
      • Ghana
      • Health
      • Insight
      • International
      • Investing
      • Japan
      • Latest Headlines
      • Life & Living
      • Markets
      • Mobile
      • Movies
      • New Zealand
      • Nigeria
      • Politics
      • Scholarships
      • Science
      • South Africa
      • South America
      • Sports
      • Tech
      • Travel
      • Travel-Africa
      • UK
      • USA
      • Weather
      • World
      No Result
      View All Result

      Recent News

      Discover the Fascinating Secrets of NASA’s Voyager Mission: Test Your Knowledge!

      Discover the Fascinating Secrets of NASA’s Voyager Mission: Test Your Knowledge!

      November 19, 2026
      WNBA Begins Search for New Commissioner as Cathy Engelbert’s Tenure May Extend to 2027

      WNBA Begins Search for New Commissioner as Cathy Engelbert’s Tenure May Extend to 2027

      November 19, 2026
      Six Drainage Enhancement Projects Pending Financial Approval, Says Ahmed Ibrahim

      Six Drainage Enhancement Projects Pending Financial Approval, Says Ahmed Ibrahim

      November 19, 2026
      Jordan Poyer Takes Center Stage: A Look Ahead to November 18, 2026

      Jordan Poyer Takes Center Stage: A Look Ahead to November 18, 2026

      November 19, 2026
      • Home
      • Advertise With Us
      • About Us
      • Corporate
      • Consumer Rewards
      • Forum
      • Privacy Policy
      • Social Trends

      Theinsightpost ©2026 | All Rights Reserved. Theinsightpost is an Elnegy LLC company, registered in Texas, USA

      Welcome Back!

      Login to your account below

      Forgotten Password?

      Retrieve your password

      Please enter your username or email address to reset your password.

      Log In

      Add New Playlist

      We are using cookies to give you the best experience on our website.

      You can find out more about which cookies we are using or switch them off in .

      No Result
      View All Result
      • Home
      • Insight
      • Blog
      • Business
      • Entertainment
      • Health
      • Politics
      • Shop
        • Gift Shop
        • Value Shop
        • Store
        • Bargain Shop
        • Discount
      • Sports
      • Tech
      • Travel
      • USA
      • Video
      • World
        • Asia
        • Africa
        • South America
        • North America
        • Europe
        • Oceania

      Theinsightpost ©2026 | All Rights Reserved. Theinsightpost is an Elnegy LLC company, registered in Texas, USA

      The Insight Post
      Powered by  GDPR Cookie Compliance
      Privacy Overview

      This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

      Strictly Necessary Cookies

      Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

      Cookie Policy

      More information about our Cookie Policy