Data-driven Task Anticipation and Knowledge-driven Planning for Human-robot Collaboration

Abstract

An agent assisting humans in daily living activities can collaborate more effectively by anticipating upcoming tasks. Data-driven methods represent the state of the art in task anticipation, planning, and related problems, but these methods are resource-hungry and opaque. Our prior work introduced a proof of concept framework that used an LLM to anticipate 3 high-level tasks that served as goals for a classical planning system that computed a sequence of low-level actions for the agent to achieve these goals. This paper describes DaTAPlan, our framework that significantly extends our prior work toward human-robot collaboration. Specifically, DaTAPlan's planner computes actions for an agent and a human to collaboratively and jointly achieve the tasks anticipated by the LLM, and the agent automatically adapts to unexpected changes in human action outcomes and preferences. We evaluate DaTAPlan's capabilities in a realistic simulation environment, demonstrating accurate task anticipation, effective human-robot collaboration, and the ability to adapt to unexpected changes.

Framework

Our Framework consists of four parts:

In the section (a) of the above figure, our system uses LLM models to generate high level tasks as an output of anticipation. The LLM model is provided with 3 key information namely a Task List (consisting of all the possible tasks that the human can perform in the environment), User Preference (a set of tasks that can vary based on the type of user/human in the environment), Scene Description (highlighting the situation in the environment with object states and types). The system has also accommodated any failure cases (or rather Hallucinations caused by the LLM). A situation where any object is not present in the environment and an alternative is needed has been taken care of by the LLM Model itself (Please view the "Resource Availability" Section in the Supplementary Material)
In section (b), the tasks generated by the LLM are mapped to the Planning Domain Definition Language (PDDL) problem description. We used lama configuration provided the Fast-Downward system. The Planner using the domain and problem description files generates a plan with the primary focus on reducing the execution cost.
NOTE: The cost for all the type of actions can be viewed in the Supplementary material.
In the section (c), we show an example of plan generated by the planner. The generated plan is the expectation that the robot has regarding the distribution of actions in a collaborative setup in jointly completing the task. .
In the section (d) of the above figure, our system finally integrates the output of Task Planning with the actors present in the Coppeliasim Simulation Environment. We are faced with three challenges:
1. Reprompting: An immediate change in the course of action when a situation is changed by the human. For Example: When the tasks related to "preparation of breakfast" is being executed and the human immediately needs to attend the meeting, then subsequently the "setting up an office table" task would be performed by both the actors catering to the human's immediate requirement.
2. Replanning: There can be situations where there is a slight deviation in the expected robot action plan. An adaptive response is generated and an updated plan with the correction is generated.
  NOTE: An example of replanning can be seen in the video below.
3. Collision Avoidance: At the level of Motion Planning, the actors are tracked and the response to a possible future collision compels the robot to stop when transiting to it's goal position.
  NOTE: Checkout the video below

Video

Below is the PDDL problem file goal:

(goal: 
  (and
    (boiled boiled_egg)
    (food_served boiled_egg plate_2
    dining_table kitchen)
    (prepared_clothes office_clothes
    ironing_board livingroom)
    (charged cellphone)
  )
)

Plan

Italian Trulli