Integrating Multimodal Communication and Comprehension Evaluation during Human-Robot Collaboration for Increased Reliability of Foundation Model-based Task Planning Systems

Martin, Eden;Hasegawa, Shoichi;Solis, Jorge;Macq, Benoît;Taniguchi, Tadahiro;et.al.
(2025) 2025 IEEE/SICE International Symposium on System Integrations (SII 2025) — Location: Munich, Germany (21.January.2025)

Files

SII25_0261_FI.pdf
  • Open Access
  • Adobe PDF
  • 5.98 MB
Download

Details

Authors
  • Martin, EdenUCLouvain
    Author
  • Hasegawa, ShoichiRitsumeikan University
    Author
  • Solis, JorgeKarlstad University
    Author
  • Macq, Benoîtorcid-logoUCLouvain
    Author
  • Ronsse, Renaudorcid-logoUCLouvain
    Author
  • Taniguchi, TadahiroRitsumeikan University
    Author
Show more
Abstract
Foundation models provide the adaptability needed in robotics but often require explicit tasks or human verification due to potential unreliability in their responses, complicating human-robot collaboration (HRC). To enhance the reliability of such task-planning systems, we propose 1) an adaptive task-planning system for HRC that reliably performs non-predefined tasks implicitly instructed through HRC, and 2) an integrated system combining multimodal large language model (LLM)-based task planning with multimodal communication of human intention to increase the HRC success rate and comfort. The proposed system integrates GPT-4V for adaptive task planning and comprehension evaluation during HRC with multimodal communication of human intention through speech and deictic gestures. Four pick-and-place tasks of gradually increasing difficulty were used in three experiments, each evaluating a key aspect of the proposed system: task planning, comprehension evaluation, and multimodal communication. The quantitative results show that the proposed system can interpret implicitly instructed tabletop pick-and-place tasks through HRC, providing the next object to pick and the correct position to place it, achieving a mean success rate of 0.80. Additionally, the system can evaluate its comprehension of three of the four tasks with an average precision of 0.87. The qualitative results show that multimodal communication not only significantly enhances the success rate but also the feelings of trust and control, willingness to use again, and sense of collaboration during HRC.
Affiliations

Citations

  • APA
  • Chicago
  • FWB

Martin, E., Hasegawa, S., Solis, J., Macq, B., Ronsse, R., Garcia Ricardez, G. A., El Hafi, L., & Taniguchi, T. (2025). Integrating Multimodal Communication and Comprehension Evaluation during Human-Robot Collaboration for Increased Reliability of Foundation Model-based Task Planning Systems. 2025 IEEE/SICE International Symposium on System Integrations, 1-7. https://hdl.handle.net/2078.5/247443 (Original work published 2025)