Code and Datasets

Examples of Application Domains

Human-Robot Teaming for Search and Rescue

When natural or manmade disasters occur, people who are injured or trapped rely on human rescue teams to save their lives. However, a rescue mission is highly dangerous and challenging, as human rescuers must perform the operation while faced with many dangers under high cognitive workload in a dynamic, potentially hazardous environment. In this application, intelligent robots who can naturally collaborate with human teams can improve team safety and rescue efficiency, thus potentially revolutionizing search and rescue missions. Collaborative human-robot teaming can also broadly impact many other time-critical, safety-critical applications, including homeland defense and security, law enforcement, among others. The image shows an example of teaming together our Jackal robot with humans for search and rescue.


Robot-Assisted Surveying, Inspection and Reconnaissance

Surveying, inspection, and reconnaissance of an environment has many real-world applications. For example, it is essential to detect pipeline erosions and track their growth rate over time across multiple inspections, as a pipeline accident can cost millions of dollars for healthcare bills, pipeline replacement, environmental response, and clean-up operations. Other applications include mine reconnaissance, geography and land surveying, and GI tract disease diagnosis using in vivo robots (developed by Prof. Rentschler at CU-Boulder). Many computational challenges are present in such applications, including how robots can recognize objects of interest (e.g., erosions in pipes), how to localize them in multiple runs potentially across a long time span, and how to track and predict their changes over time (e.g., growth rate of erosions).


Examples of Research Topics

Robot Awareness of Human Activities

To avoid overloading human's cognitive capability in safety-critical, time-critical applications, robots must have the ability to automatically understand human behaviors, thus enabling natural human-robot collaboration toward the same mission goal. Robot awareness of human activities is also necessary for natural human-robot interaction in applications such as robot-assisted living. A particular challenge what we focus on is how a robot can reason about the underlying temporal dependency of sequential human behaviors (e.g., "standing up" vs "sitting down").


Environmental Awareness

In human-robot teaming and robot-assisted inspection, environmental awareness is critical for robots to navigate and perform relevant tasks. To this end, we develop approaches to detect critical objects (e.g., tools) for response to emergency by integrating multisensory observations. We also investigate methods to address the problem of long-term place recognition, which is essential for robot localization and SLAM during long-term robot navigation with highly different weather, illumination, and vegetation conditions across days, months or even seasons.


Situational Decision Making and Planning

After reasoning about humans and the surrounding environment, collaborative robots need the crucial capability of making appropriately decisions to interact with people and facilitate ongoing tasks. Our research focuses on developing decision making methods that consider the uncertainty of robot reasoning results and risks of robot actions, and are aware of unexpected events that have never been experienced by robots. We also investigate methods that integrate planning and perception to enable new robot capabilities such as active perception and adaptive situational planning, especially in situations when humans are in the loop.


Human Detection and Tracking

Robust efficient detection and tracking of humans in complex environments is critical to ensure safe robot operations in human-centered environments, and the first step to enable robot awareness of human behaviors. Our research is focused on developing approaches to track multiple individuals in real time with non-upright body configurations, and human-object and human-human interactions, to address challenges of lighting variation, occlusion, and robot movement, and to integrate multimodal observations when a robot is equipped with multiple sensors.


Robot Internal Representation of People

Observations perceived by robots always contain a large amount of irrelevant, redundant and noisy data, which can significantly increase computational cost, and even worse distract robots to understand useful information. Thus, it is important to construct a compact, discriminative representation of people for robots to understand human behaviors in order to interact with and assist people. This research focuses on developing human representations by designing and integrating local spatio-temporal features and skeleton-based features to encode human body shape and motion.