Monitoring human physiological responses to improve interactions with robots

August 14, 2019 by Ingrid Fadelli, Phys.org
A system diagram for the conceptual framework for a Human Robot Collaboration as System of Systems that highlights the 3 aspects of an HRC system : Awareness, Intelligence and Compliance. Credit: Savur, Kumar & Sahin.

Researchers from the Century Mold Collaborative Robotics Laboratory (CMCRL) at Rochester Institute of Technology (RIT) in New York have recently developed a new framework for monitoring human physiological responses while users are collaborating with a robot to complete a task. Under the supervision of Prof. Ferat Sahin, director of the Multi Agent Bio-Robotics Laboratory, CMCRL researchers are working to develop systems, frameworks and software tools to study human-robot collaboration in an industrial setting. Their new approach, outlined in a paper pre-published on arXiv and set to be presented at the IEEE Systems, Man & Cybernetics conference in Bari (Italy), enables continuously collecting physiological data during human-robot interactions, while changing a robot's movements to trigger a response in the human with whom it's interacting.

Many experts believe that soon, humans and machines will be working together in a variety of fields, including industry and manufacturing. To enable efficient automation and human-robot collaboration, however, researchers will need to develop approaches that leverage human skills such as decision making, dexterity, flexibility and creativity, combining these with the speed, accuracy and power associated with robots.

In fact, while complete automation would be very expensive, manufacturing carried out manually by humans alone is often slow and prone to errors. Developing tools that enhance human-robot collaboration is hence important. According to RIA, this will be a central part of the Fifth Industrial Revolution, also known as Industry 5.0.

"In our opinion there are three main challenges to human-robot collaboration in industry. They are safety, human trust in automation and productivity," Shitij Kumar, one of the researchers who carried out the study, told TechXplore. "All of these are dependent on one another. In order to address these challenges, we first developed a human-robot collaboration (HRC) setup as a system of systems that would allow us to create different human-robot collaboration scenarios and understand and analyze human-robot interactions."

A system diagram of the implementation based on the proposed framework representing the data collection and monitoring during an HRC experiment. Credit: Savur, Kumar & Sahin.

As part of his Ph.D., under the guidance of Dr. Ferat Sahin, the director of CMCRL, Kumar started developing a system that changes robot behavior based on human-robot separation distance and actions in the shared workspace. In this context, a robot's behavior is deterministic and predictable based on rules.

Despite the promise shown by his approach, he found that human feedback did not always occur, which made it harder for his system to achieve satisfactory robot compliance. Robot compliance essentially means that the robot can manage the expectations of humans it is interacting with and effectively communicate with them.

"We believed that a better compliance of the system that gives the operator working with the robot a sense of control and predictiveness of robot behavior would increase his/her 'trust in automation," and thereby increase the productivity of the task," Kumar said. "However, we then started wondering, how do you quantify 'trust' or comfort level of a human operator working with the robot?"

Kumar and his colleagues believe that monitoring human physiological signals, which are indicators of mental and physical stress, as well as other emotions, during a task that involves human-robot interaction would be a step in the right direction. This belief is supported by previous research that found that such signals ( known as psychophysiological responses) are reliable indicators for changing robot behavior and motion.

Sensors and devices used in the prototype implementation. The researchers used a motion capture system to monitor human motion and a camera to record the experiment. They also tracked human-gaze using Pupil Labs and human physiological responses such as pupil dilation, PPG, GSR, EEG \& ECG recorded. Credit: Savur, Kumar & Sahin.

With this in mind, Celal Savur, a Ph.D. Student at RIT's CMCRL, carried out a study to explore the effects of robot's motion and behavior on human psychophysiological responses. The key goal of his study was to understand what changes in a robot's motion result in 'human discomfort' responses such as fear or stress.

"In order to do this, a framework for a system that represents and records the robot motion and human physiological state concurrently was needed," Savur explained. "Once the relationship between the robot motion and the human physiological state is identified, the human physiological response can be used as a feedback to directly control/update the robot's motion/behavior. Such systems are called 'physiological computing systems." Physiological computing is a subset field of affective computing often used in computer games, in which the games adapt in real time to the responses of the gamer, to achieve more interactive gaming experience."

In their study, Savur and Kumar set out to develop a framework that can monitor human psychophysiological responses as users complete tasks that involve human-robot collaboration. Their work lies at the intersection between computer science, robotics and psychophysiology, a branch of neuroscience that seeks to understand how a person's mental state and his/her physiological responses interact or affect one another.

The framework they developed falls under the category of 'physiological computing." This is a type of affective computing that incorporates real-time software adaption to the psychophysiological activity of the user. Essentially, the framework they proposed can be used to investigate how changes in the robot motion (e.g. speed and trajectory) affect the human operator in an industrial environment.

Speed and Separation Monitoring using on-robot Time--of--Flight laser--ranging sensor arrays

"Let us consider, for example, that there are two operators working with a robot, operator A and B," Kumar explained. "Operator A has worked longer and is comfortable working with the robot, as he/she can predict the robot behavior from experience. Operator B is new and is a bit skeptical of the robot motion. The robot behavior changes in terms of speed at which it moves and how much distance it maintains when it is near the operator based on the operator's physiological state and behavioral patterns. In this scenario, for a better human-robot interaction, the robot can move at higher speeds working near operator A, and move slowly working with operator B. As operator B gains more experience, the robot motion can adapt to it, thereby building trust and positively affecting the overall productivity."

The study carried out by the researchers in CMCRL had two main objectives. The first was to generate a database of human-robot collaboration tasks, recording human-robot interactions. They then wanted to use this database to investigate how human physiological responses can be used to affect a robot's motion, positively impacting the automation process. In other words, their aim was to build a physiological computing system for human-robot collaboration in industrial and manufacturing environments.

"Our framework is essentially a system that represents and records the robot's motion and human physiological state concurrently," Kumar said. "In order to record this information, the framework provides interfaces to the robot, sensors such as cameras, motion capture system and also to the biological/biometric data acquisition devices. As all these devices work on different sampling rates, this framework helps in synchronous data acquisition and representation of the human-robot collaboration."

In traditional social robotics experimental setups, human subjects are asked questions about their experiences and perceptions during or after an experiment. Using their feedback, researchers are then able to analyze and quantify the subjective data collected during the experiment.

A framework for monitoring human physiological response during human robot collaborative task

"Methods that interrupt subjects or have the subject recollect their experience, however, are not always able to maintain the integrity of the experiment or accurately represent the subjective data," Kumar explained. "Therefore, unlike traditional methods, this implementation of the framework enables the human subject or the principal investigator to generate event markers as the human robot collaboration (HRC) experiment / task is being performed."

Their framework can automatically generate event markers based on the human-robot representation in a virtual world, which is known as a digital twin. For instance, it can help to identify when the robot and human were closest to each other during the task, when the robot had to stop or was interrupted by a human action, the progress of the task performed by the human, events based on commands or control by the human operator, an event when the robot was working at maximum speed, and the beginning/end of a task.

"Our framework also provides a user interface for the researchers to replay and visualize their HRC experiments," Kumar said. "Moreover, it allows them to analyze and label the data collected. The continuous and synchronous collection of physiological data from various devices and interfacing them along with the robot control and interface in a single ecosystem, allows a complete representation of human and robot state. This can help understand the cause and effect between the human physiological state and the robot's movements."

With the advent of wearable devices and the Internet of Things (IoT), human will become easier to collect and thus be readily available. The framework developed by the research could thus prove extremely valuable, as it is designed to leverage this data to improve .

Developing a Digital Twin Setup for HRC System of Systems

"This framework enables a continuous data recording with built-in event generation and signal synchronization over distributed systems can maintain the integrity of the experiment (recreating the scenario of a task in an industrial setting) and accurately represent the subjective data," Kumar said.

Kumar and his colleagues believe that an agent capable of such physiological computing (i.e., which can detect human physiological responses and respond to these) could result in a closed human-in-the-loop system, where both human users and robots in an HRC setup are monitored and information is shared between them. This could result in better communication, which might ultimately foster greater trust in automation among the public, while also increasing productivity. In the future, the researchers plan to make the databases generated in their study available to other HRI researchers.

"Our next studies will focus on developing a complete user interface application of the physiological computing system for processing of recording signals, extracting information and applying machine-learning algorithm to provide feedback to the robot," Kumar said. "The final objective of this work is to generate a database that can be used to further the understanding of how human physiological responses can be inferred to result in adaptive motion behavior."

More information: A framework for monitoring human physiological response during human robot collaborative task. arXiv:1907.10782 [cs.MA]. arxiv.org/abs/1907.10782
(To be published in IEEE International Conference on Systems, Man, and Cybernetics, Oct 6th -9th 2019, Bari, Italy)

HRC-SoS: human robot collaboration experimentation platform as system of systems. arXiv:1905.01026 [cs.RO]. arxiv.org/abs/1905.01026

Speed and separation monitoring using on-robot Time—of—Flight laser—ranging sensor arrays." arXiv:1904.07379 [cs.RO]. arxiv.org/abs/1904.07379 (To be Published in IEEE 15th International Conference on Automation Science and Engineering (CASE), Aug 22nd -26th 2019, Vancouver, BC, Canada)

Psychophysiological methods to evaluate user's response in human robot interaction: A review and feasibility study. DOI: 10.3390/robotics2020092. https://www.mdpi.com/2218-6581/2/2/92

kumarshitij.com/blog/

© 2019 Science X Network