VRKitchen: An interactive virtual environment to train and test AI agents

A team of researchers at the University of California Los Angeles (UCLA), under the supervision of Prof. Song-Chun Zhu, has developed an interactive 3-D virtual environment for training and evaluating AI systems. Their platform, called VRKitchen, was recently presented in a paper pre-published on arXiv.

"Training robots in the real world is expensive, slow and hardly reproducible," Tianmin Shu, a Ph.D. student at UCLA who carried out the study, told TechXplore. "Because of this, in recent years, we have witnessed great progress in building physics-enabled 3-D virtual platforms (e.g. AI2-THOR, House3D) where embodied AI agents can walk around and find a target object."

Despite their many advantages, most existing virtual platforms for training AI systems do not enable complex interactions with physical objects, which are very common in real-world situations. In addition, many of these platforms do not allow humans to take part in the training of agents within the virtual environment.

"In this work, we aim to build a highly interactive and realistic virtual platform, allowing researchers to train and evaluate their AI agents in a large range of real world tasks involving complex object manipulations (e.g. cooking a meal)," Shu said. "A further goal of our research is to build VR interfaces that allow humans to teach AI agents by giving demonstrations in the same environment."

VRKitchen, the virtual environment created by Shu and his colleagues, is based on Unreal Engine 4, a suite of tools generally used by game developers to design games or simulations. Compared to existing virtual environments, VRKitchen has two key advantages.

First, in VRKitchen, an AI agent can perform a wide range of cooking tasks with rich object state changes and compositional goals. For instance, if an agent is learning to make pizza, the virtual environment simulates visual effects of the cheese being grated, tomatoes being cut and the ingredients being placed onto the dough.

"We really hoped that the ability to simulate these complex cooking tasks would make VRKitchen stand out from other platforms," Xiaofeng Gao, Ph.D. student at UCLA who carried out the study, told TechXplore.

A second advantage of VRKitchen is that the embodied agent navigating the virtual environment can be controlled by both AI algorithms and human users. This allows humans to give demonstrations to AI agents, so that the latter can acquire knowledge about how to complete a task via observation, rather than having to learn the task independently.

"We built three major modules in VRKitchen," Gao explained. "(1) the UE4 (Unreal Engine 4)-based physics engine and photorealistic rendering module with several humanoid agents, and a set of customizable kitchen scenes consisting of various furniture, kitchen appliances, food ingredients, containers, and tools for cooking activities; (2) a user interface module that allows users or algorithms to perform tasks by virtual reality device or Python API; (3) a Python-UE4 bridge, which transfers high-level commands to motor control signals and sends them to the agent."

The virtual environment developed by Shu, Gao and their colleagues could soon be used to train agents powered by a broad range of machine learning techniques on complex tasks that involve fine-grained object manipulation. Along with VRKitchen, the researchers presented new standardized evaluation benchmarks and data collection tools, which are designed to facilitate the use of their platform in research settings.

"Popular approaches in recent literature that work well in simpler virtual environments fail to achieve decent performance on the cooking tasks implemented in our platform, imposing a new challenge for AI research and highlighting the importance of building more realistic and complex virtual environments," "Shu said. "We now plan to open source our platform and host standardized challenges (VR Chef Challenge) for the AI community. We are also working on building AI agents that can efficiently acquire complex skills by exploring the virtual world themselves as well as by learning from human demonstrations."

© 2019 Science X Network