Community interest in implementing a Reinforcement Learning local planner for Social Interactions with the NAV2 framework

[maxijsseldijk]

The Project

Overview

I am developing a Reinforcement Learning (RL) framework for learning social interactions as a local planner replacement in the NAV2 framework. Although previous work by @mhpanah on the ROS2 navigation stack has been halted, I am exploring the community's interest in an RL-based local planner and the features and implementations that would best integrate with the NAV2 stack.

Core Idea and Integration with NAV2

The NAV2 framework provides tools for navigation in static environments, featuring dedicated planner, controller, and behavior tree servers for programming complex behaviors. However, its current implementations are limited in dynamic environments, where it relies on cost maps to determine actions. My focus is on robot-human interactions, where traditional methods like social cost maps can lead to undesirable behavior in complex scenarios, such as when a human blocks a doorway. Tuning parameters for such scenarios is challenging and often not adaptable to all social situations. Therefore, an RL-based interaction module is more suitable, as it selects actions or local plans based on the scenario.

Current Implementation and Interaction with the NAV2 Stack

Although the paper and code are still in progress, the RL framework utilizes LiDAR data, NAV2 control input, NAV2 plan, and the location and velocity of other agents to train a neural network for socially aware behavior in a GAZEBO environment. This behavior is designed to be location-independent, making it applicable to various environments once trained. The local planner is invoked during an "interaction," which is determined by the direction, velocity, and distance between the robot and agents.

I am seeking community interest in a local RL planner or any aspect of this research. If there is interest in implementing an RL planner for the NAV2 stack, I can adjust my planning and code to facilitate integration. As I am undertaking this project alone and it has grown significantly, I would appreciate guidance on integrating functionality into the NAV2 stack effectively, if considered useful.

[SteveMacenski]

Certainly having an (a) RL controller, (b) a framework for retraining and documentation, (c) pre-trained on some reasonable objective functions to make good behavior, and (d) extensively tested for practical utility on real robots would be useful! As far as I'm aware, there aren't many companies using RL for their controllers (yet) due to the difficult of fine tuning for good social/comfortable behavior that MPC or sampling algorithms provide, unless they have 1000+ staff that are exceptional.

Is this part of a research project or a production-quality system? We'd be most interested if this was tested on hardware and shown to work well and mature enough to be considered deployable in robot applications. The previous RL work that you reference was very tutorial quality and the navigation was rather poor. A production-quality ready-to-use example that is fine-tunable with additional objective functions is sorely missing an welcome!

[maxijsseldijk]

Hey Steve,
This is part of academic research. Great to hear that it would be interesting to investigate the ability to apply such an RL controller in NAV2. The four points you mentioned are also important for my research, so I think it is interesting to look into this! I do think it will be a challenge, as training of an RL agent in a robust and universally applicable way has some challenges. For integration, I will need to think what the best way is, as there are many ways to do this. Currently, it utilizes a whole array of subscribers and 3rd party plugins that would need to be cleaned up. If I have questions on integration or documentation standards, where would be the best way to ask this? I will continue working on it for the coming months and would like to keep you updated!

[SteveMacenski]

I do think it will be a challenge, as training of an RL agent in a robust and universally applicable way has some challenge

That is the challenge of engineering 😉 We need to be able to make it work well in real-life to be able to use it in real-life! 😆

For integration, I will need to think what the best way is, as there are many ways to do this. Currently, it utilizes a whole array of subscribers and 3rd party plugins that would need to be cleaned up.

I think the first step is to make something that works well in your own github repository and/or lab's repository. Something that I can look at and we can look over the requirements that you made it under, how it meets those requirements, and how well it functionally works to be included into Nav2's codebase directly. Doing some cleanup would be good - you can see different examples like MPPI, RPP, etc for controller plugin examples in Nav2. Data subscribers make sense for pulling in lidar / camera data, so no issues with that.

The Nav2 slack (see Nav2 readme for a link) or Robotics Stack Exchange are good place for Q&A!